# Camera View Adjustment Prediction for Improving Image Composition

Yu-Chuan Su Raviteja Vemulapalli Ben Weiss Chun-Te Chu
Philip Andrew Mansfield Lior Shapira Colvin Pitts
Google Research

###### Abstract

Image composition plays an important role in the quality of a photo. However,
not every camera user possesses the knowledge and expertise required for
capturing well-composed photos. While post-capture cropping can improve the
composition sometimes, it does not work in many common scenarios in which the
photographer needs to adjust the camera view to capture the best shot. To
address this issue, we propose a deep learning-based approach that provides
suggestions to the photographer on how to adjust the camera view before
capturing. By optimizing the composition before a photo is captured, our
system helps photographers to capture better photos. As there is no publicly-
available data for this task, we create a view adjustment dataset by
repurposing existing image cropping datasets. Furthermore, we propose a two-
stage semi-supervised approach that utilizes both labeled and unlabeled images
for training a view adjustment model. Experiment results show that the
proposed semi-supervised approach outperforms the corresponding supervised
alternatives, and our user study results show that the suggested view
adjustment improves image composition $79\%$ of the time.

##  1 Introduction

Image composition has a significant effect on the perception of an image.
While a good composition can help make a great picture out of the dullest
subjects and plainest of environments, a bad composition can easily ruin a
photograph despite how interesting the subject may be. Unfortunately, a
typical camera user may lack the knowledge and expertise required to capture
images with great composition.

Figure 1:  Our goal is to improve the composition of the captured photo by
providing view adjustment suggestions when the user is composing the shot.

A commonly used technique for improving image composition is image cropping,
and several existing works study how to crop images automatically [38, 5, 35,
29, 4, 49, 23, 11, 31, 40, 10, 7, 12, 44, 6, 42, 47, 26, 39, 48]. However,
cropping works only in limited scenarios in which the best composition can be
achieved by removal of certain portions of the image. It is not suitable in
many common scenarios where the photographer needs to adjust the camera view
to get the best shot. When evaluated on our view adjustment dataset, the crops
predicted by the state-of-the-art GAIC cropping model [48] have an average IoU
of 0.61 with the groundtruth views, clearly indicating that cropping is not
enough. In comparison, the proposed view adjustment model achieves a much
higher IoU of 0.75.

While there are some existing rules for composing photos, each rule is valid
only for specific scenes, and requires detection of various low-level (leading
lines, triangles, etc.) and high-level semantic (face/person, foreground
objects) cues. Furthermore, it is non-trivial to determine which rule or
combination of rules is applicable for a given scene.

To address this problem, we introduce a system that provides camera view
adjustment suggestions to the photographer when they are composing the shot.
Given a view composed by the user, our goal is to suggest a candidate view
adjustment and its magnitude such that the photo captured after applying the
adjustment will have a better composition (see Fig. 1). Specifically, we
consider the following adjustments in this work: horizontal (left or right),
vertical (up or down), zoom (in or out), and rotation (clockwise or counter
clockwise) along the principal axis. The adjustment magnitude is represented
using a percentage of the image size for all adjustments except rotation, for
which we use radians. By adjusting the view prior to capture, we enable
generic modifications to image composition. Hence, our system can improve
image composition in scenarios where cropping fails (see Fig. 2). Note that
this work focuses on static scenes, or more specifically scenes where motion
is not the subject (e.g. portraits, nature and urban environment photography).
Suggesting view adjustments may not be suit dynamic (wildlife, sports,
actions) scenes where lightning-quick decisions are required from the
photographer.

To the best of our knowledge, there is no publicly-available data for
evaluating the performance of view adjustment prediction. While a common
practice is for human raters to annotate images with ground truth labels, this
is difficult for view adjustment because the results of adjustment are
generally not available. The raters need to infer how the adjustment affects
the composition, which may be difficult without professional photography
knowledge. Instead, we create a new view adjustment dataset from existing
image cropping datasets. The idea is to convert view adjustments into
operations on 2D image bounding boxes and use the best crop annotation as the
target view for adjustment. The proposed approach allows us to generate
samples and view adjustment labels from image cropping datasets automatically
without additional human labor.

Figure 2:  View adjustment enables more diverse modification to image
composition, and can improve composition in scenarios where image cropping
fails.

We acknowledge that our view adjustment dataset is not ideal in several
respects. It ignores perspective distortions while adjusting the camera view,
and the adjustment magnitude could be limited depending on the ratios between
the best crop and the uncropped image sizes. Despite these limitations, our
view adjustment dataset still provides a good starting point to evaluate
composition-aware view adjustment prediction models. Furthermore, we address
these limitations by also evaluating the view adjustment model on $360\degree$
images, which do not suffer from distortions or limited field-of-view (FOV).
Please refer to Sec. 4.3 for details.

Another limitation of our view adjustment dataset is that its amount and
diversity is inherently limited by the cropping datasets. Because state-of-
the-art machine learning models typically require a large and diverse dataset
to train well, our view adjustment dataset may not be sufficient for training
a good view adjustment model. In light of this problem, we propose a two-stage
training approach that makes use of additional unlabeled images. See Fig. 3.
Our empirical results show that the additional unlabeled data is important for
improving model performance.

We evaluate our approach on a view adjustment dataset consisting of 3,026
samples generated from 521 images from the FCDB [6] and GAICD [47] datasets.
Quantitative results show that the proposed semi-supervised approach clearly
outperforms the corresponding supervised alternatives, and user study shows
that the adjustments suggested by our model improve the composition $79\%$ of
the time.

Figure 3:  Our two-stage semi-supervsied approach leverages both labeled and
unlabeled data to learn the view adjustment model. In the first stage, we
learn a composition scoring model from both labeled data and unlabeled well-
composed images. The scoring model is used to generate pseudo view adjustment
labels for unlabeled images. These pseudo-labeled images along with real
labeled images are used to train the view adjustment model in the second
stage.

Our major contributions are as follows. First, we formulate the problem of
view adjustment prediction for improving image composition. Second, we
introduce a labeled dataset for evaluating view adjustment prediction models.
Finally, we propose a two-stage semi-supervised approach that leverages both
labeled and unlabeled data to train the view adjustment model. We show that
the proposed semi-supervised approach outperforms the corresponding supervised
alternative quantitatively and demonstrate the effectiveness of our model
through user study.

##  2 Related Work

#### Image cropping

Cropping is a widely used technique for changing image composition during
post-processing, and automatic image cropping algorithms aim to find the best
crop within an image. One common approach followed by existing methods is to
select the candidate crops using a scoring function, and the research focus
has been on designing a good scoring function for cropping. Existing works
exploit saliency [38, 5, 35, 29, 4], photography rules [49, 23, 11], or a data
driven approach to learn the scoring function [31, 40, 10, 7, 12, 44, 6, 42,
47, 26, 39, 48, 22]. While early methods learn the cropping model using
unsupervised data [7, 12] or data annotated for generic image aesthetic
quality [31, 40, 10], recent works show that a large scale annotated dataset
designed specifically for cropping is essential for learning the state-of-the-
art image cropping model [42, 47, 26, 39, 48]. Instead of following the
scoring function paradigm, some works try to predict the target crop directly
without generating and scoring candidate crops [19, 20, 25, 21, 24].

Our goal is not to produce the best image crop; it is to improve image
composition while the photographer is composing the image. Because view
adjustment is a more general operation than image cropping, it may improve
composition in scenarios where cropping is not suitable. Furthermore, a view
adjustment model needs to make a suggestion based on partial information,
while the target crop is fully visible to image cropping models. This
introduces unique challenges in view adjustment prediction for both data
collection and modeling.

#### Photography recommendation

Prior works on photography recommendation provide various types of
suggestions. Some of them study person location recommendation [50, 43, 28,
27, 41, 34]. They take a scenic image as input and suggest where a person
should stand within the image. Others study photography at popular landmarks
and suggest either a view enclosure in the input image or a geo-location for
capturing a better photo [16, 45, 51, 46, 32, 33]. These works utilize
abundant web photos captured near the target landmark in order to provide
landmark-specific suggestions. Our problem differs from the above in that we
provide view adjustment suggestions to the photographer. Furthermore, our
method is applicable to any photo instead of being restricted to specific
image content or geo-locations. Similar to our approach, Virtual Portraitist
[15] utilizes weakly-labeled “positives” and suggests how to adjust the head
pose to approximate positive samples, which can be considered as an inverse
camera view adjustment. However, it is designed specifically for learning head
poses in selfies and is not applicable for other types of photos. While some
works also provide view recommendation in arbitrary photos [8, 36], they
assume the availability of a wide-angle image and find the target view within
it. Therefore, they require specialized hardware and is not applicable to any
camera. Finally, some commercial systems provide photography recommendation in
the form of suggested viewfinder center [1, 3] but provide only a black box
system. To the best of our knowledge, this is the first work that formulates
and introduces an evaluation benchmark for the view adjustment prediction
problem.

##  3 Approach

In this section, we introduce our approach for learning view adjustment
prediction. We first formalize our problem and provide an overview of the two-
stage training approach. Next, we describe the two models and our pseudo-label
generation process in more detail.

###  3.1 Problem Formulation

Given an input image, the goal of view adjustment is to answer the following
questions: 1) whether there exists a candidate view adjustment that will
improve the composition of the image, 2) if the composition can be improved,
which candidate view adjustment can lead to the best composition, and 3) given
the candidate view adjustment, what is the appropriate magnitude for the
adjustment.

To create a dataset with the view adjustment labels for the above questions,
we propose generating the sample images and ground truth labels from image
cropping data. Given an image and the best crop within the image, we perturb
the image crop by one of the candidate view adjustments with a random
magnitude. We take the perturbed crop as the image sample and use the inverse
perturbation as the ground truth label for view adjustment. We also use the
best crop as the sample, where the label is that no adjustment is required.
The candidate adjustments considered in this work include horizontal (left or
right), vertical (up or down), zoom (in or out), and rotation (clockwise or
counter clockwise) about the principal axis. We focus on these eight
adjustments because they can be presented in the camera viewfinder and
executed by the photographer easily. Note that rotation about yaw and pitch
axes may be hard to distinguish from translation on a 2D display, and their
effects may be similar depending on the scene. Also, more complex adjustments
can be achieved by performing multiple basic adjustments in a sequence. We
control the perturbation magnitude such that 1) the perturbed crop has
sufficient overlap with the annotated crop, and 2) the perturbed crop is in
the original image. See supp. for data generation details.

While the proposed approach allows us to generate training samples for view
adjustment from image cropping datasets, the diversity of the generated data
is controlled by the source cropping datasets. To overcome this limitation, we
propose a two-stage training approach that makes use of additional unlabeled
images. In the first stage, we learn a composition scoring model that rates
the composition quality of a given image. We then use this scoring model to
generate pseudo view adjustment labels for unlabeled images. This is achieved
by simulating candidate view adjustments for a given image and selecting the
adjustment that leads to the best scoring image as the pseudo label. In the
second stage, we train a view adjustment model using both the labeled images
generated from cropping datasets and the additional pseudo-labeled images. See
Fig. 3. The two-stage training approach exploits unlabeled data to increase
the amount of training data and improve model performance in both stages.
Details are provided in the following sections. Also see supp. for the
training pipeline summary.

###  3.2 Composition Scoring Model

We implement the composition scoring model using a deep convolution neural
network (CNN), denoted $M_{c}$. The model takes an RGB image as input and
produces a score in the range $[0,1]$ as output. Since the purpose of this
model is to compare the results of different view adjustments, we train it
using a pairwise ranking loss. Let $(I_{p},I_{n})$ be two images of the same
scene where $I_{p}$ has better composition than $I_{n}$. The pairwise ranking
loss encourages the score of $I_{p}$ to be more than the score of $I_{n}$ by a
margin $\delta$:

$\max(0,\ \delta+M_{c}(I_{n})-M_{c}(I_{p})).$ (1)

To train this model, we use pairwise comparison data generated from both
labeled image cropping data and unlabeled well-composed images.

#### Labeled data

We utilize image cropping datasets in the following formats to generate image
pairs $(I_{p},I_{n})$:

  * $\bullet$

_Scored candidate crops_ —in this format, each image comes with a pre-defined
set of candidate crops along with their composition scores. In each training
iteration, we choose $N$ crops from a random image and generate
$\frac{N(N-1)}{2}$ pairs. $I_{p}$ and $I_{n}$ are determined for each pair by
comparing the provided scores. We use the pairwise ranking loss averaged over
these $\frac{N(N-1)}{2}$ pairs, denoted as $L_{sc}$, for training.

  * $\bullet$

_Best crop annotation_ —this format provides a best crop for each image. We
take the best crop as $I_{p}$ and generate $I_{n}$ by randomly perturbing the
bounding box of the best crop. We restrict the perturbation magnitude such
that $I_{n}$ remains within the original image. See supp. for the perturbation
details. In each training step, we generate $K$ pairs from $K$ images and use
the pairwise ranking loss averaged over these $K$ pairs, denoted as $L_{bc}$,
for training.

#### Unlabeled data

Modern CNNs require a large and diverse dataset for training. While recent
image cropping datasets provide a large number of candidate crops [42, 47,
48], they usually come from a relatively small number of images. To increase
the amount and diversity of the training data, we exploit unlabeled images
following prior works in learning image aesthetic [7, 12]. We first collect
well-composed images from a photography website where the photos are
contributed by experienced photographers111unsplash.com. Given a well-composed
image, we randomly perturb the image (by cropping, shifting, zooming-out, or
rotation) to generate a new image with inferior composition. We use the
original image as $I_{p}$ and the perturbed image as $I_{n}$ to form an image
pair. See supp. for perturbation details. Unlike the perturbation in the
labeled data, we allow the generated image to go beyond the original image.
Regions in $I_{n}$ that are not visible in $I_{p}$ are filled with zero
pixels.

Our method for using unlabeled data differs from prior works as we use
perturbations beyond cropping. This is important because our goal is to
provide more general view adjustment, and empirical results show that using
cropping alone leads to worse performance. In each training step, we generate
$P$ pairs from $P$ unlabeled images and use the ranking loss averaged over
these $P$ pairs, denoted as $L_{wc}$, for training. The total loss function
used to train the composition scoring model is given by
$L=L_{sc}+L_{bc}+L_{wc}$.

#### Data augmentations

Many of the pairs generated from unlabeled images will have zero-pixel borders
in $I_{n}$ but not in $I_{p}$. When trained on such pairs, the composition
scoring model may just learn to score images based on the presence or absence
of zero-pixel borders instead of focusing on the content. To avoid this, we
introduce three types of zero-pixel border augmentations:

  * $\bullet$

_Shift borders_ selects a certain percentage of columns on either the left or
the right side of the image, and a certain percentage of rows at either the
top or the bottom of the image. The pixel values in selected rows and columns
are replaced with zeros. This simulates the zero-pixel borders introduced by
shift perturbations.

  * $\bullet$

_Zoom-out borders_ selects a certain percentage of rows and columns on all
sides and replaces the pixel values in these rows and columns with zeros. This
simulates the zero-pixel borders introduced by zoom-out perturbation.

  * $\bullet$

_Rotation borders_ selects a random $\theta$ and applies $R_{-\theta}\circ
R_{\theta}$ to the image, where $R_{\theta}$ is the rotation operator. This
simulates the borders introduced by a rotation perturbation.

We randomly apply one of these augmentations to image $I_{p}$ in the pairs
generated from unlabeled images so that the model learns to focus on the image
content rather than zero-pixel borders. We also add these augmentations to
both $I_{p}$ and $I_{n}$ in the pairs generated from labeled data.

Our composition scoring model is motivated by existing cropping models [7, 42,
47]. The main difference is that existing cropping models are designed to
compare image crops that are fully visible to the model. In contrast, the goal
of our model is to compare the results of different view adjustments where
many of them are only partially visible. Also, we combine both labeled and
unlabeled data to train the model, while existing cropping models are trained
with only one of them.

Figure 4:  Pseudo label generation. If the composition score improvement is
below a given threshold, the label is set to _no adjustment is needed_.
Otherwise, we use the best scoring adjustment as the label.

###  3.3 Pseudo Label Generation

We generate pseudo view adjustment labels using the composition scoring model
through simulation. Given an image, we simulate the results of candidate view
adjustments. For each candidate adjustment, we generate the results for nine
magnitudes equally spaced between $[\frac{\pi}{36},\frac{\pi}{4}]$ for
rotation and $[5,45]$ for other adjustments (represented as the percentage of
the image size). This leads to $8{\times}9{=}72$ different view adjustment
results. We use the composition scoring model to rate these candidate views
and select the one with the highest score. If the best view leads to a
composition score higher than the original image by a margin of
$\Delta{=}0.2$, the adjustment is taken as the label. Otherwise, the label is
that no adjustment is required. Similar to the unlabeled image perturbation in
the composition scoring model, unknown regions that come into the image during
simulation are filled with zero pixels. Note that we ignore depths when
simulating the adjustment results, because depth information is generally
unavailable. See Fig. 4.

###  3.4 View Adjustment Model

Given the labeled samples from the view adjustment dataset and the images with
pseudo label generated by the composition scoring model, we learn a view
adjustment model as follows. The view adjustment model is implemented using a
CNN with three output heads:

  * $\bullet$

_Suggestion predictor_ predicts whether a view adjustment should be suggested.
_Suggestion predictor_ is a binary classification head trained with cross-
entropy loss.

  * $\bullet$

_Adjustment predictor_ predicts which candidate view adjustment should be
suggested when adjustment is needed. It is a multi-class classification head
trained with categorical cross-entropy loss.

  * $\bullet$

_Magnitude predictor_ predicts the adjustment magnitude given the suggested
adjustment. The _magnitude predictor_ consists of eight regressors that
regress the adjustment magnitude for each of the candidate adjustments and is
trained with $\ell_{1}$ loss.

During training, the gradients of the adjustment predictor are backpropagated
only for samples where a suggestion should be provided. Similarly, the
gradients of the magnitude predictor are only backpropagated for the
adjustment that should be suggested.

During inference, we first use the suggestion predictor to decide whether or
not the view needs to be adjusted. If the view needs to be adjusted, we use
the adjustment predictor to select the candidate adjustment. Finally, we use
the output of the magnitude predictor corresponding to the selected adjustment
as the suggested magnitude.

We also examined other model designs, e.g. combining the suggestion and
adjustment predictors as a single multi-class classifier, combining the
adjustment and magnitude predictors, etc. However, these alternative models do
not work as well as the proposed method. In particular, they fail
significantly in suggestion prediction, i.e. determining whether an adjustment
is needed.

##  4 Experiments

We evaluate our view adjustment model both objectively and subjectively. The
goal is to verify that 1) our two-stage training approach improves the view
adjustment prediction performance, and 2) the suggested view adjustments
improve the image composition.

#### Training datasets

As described in Sec. 3, the proposed approach exploits an array of datasets
for training.
_FCDB_ [6] is an image cropping dataset consisting of 1,265 training images
annotated with best crops.
_GAICD_ [47] is an image cropping dataset consisting of 1,036 training images.
Each image comes with 90 candidate crops along with their composition scores,
and we take the one with the highest score as the best crop.
_CPC_ [42] consists of 10,797 images. Each image comes with 24 candidate crops
and their composition scores.
_Unsplash_ [2] consists of 2M images shared by 200K photographers on
unsplash.com. Since many of these images are contributed by experienced
photographers, we assume that they are well-composed and use them as unlabeled
samples for training the composition scoring model.
_Open Images_ [18] consists of 9M images from Flickr. We generate pseudo
labels for a subset of 5.5M images222We use images containing human verified
labels..

For the composition scoring model, we use the annotated best crops in _FCDB_
and _GAICD_ (with $K{=}16$), the scored candidate crops in _CPC_ and _GAICD_
(with $N{=}16$), and the unlabeled images in _Unsplash_ (with $P{=}16$). For
the view adjustment model, we use labeled samples from _FCDB_ and _GAICD_ ,
and pseudo-labeled samples from _Open Images_. In each training iteration, we
use 64 labeled and 64 pseudo-labeled samples.

#### Comparative methods

Since there is no prior published work on view adjustment models, we use the
following variants of our approach for comparison. The goal is to demonstrate
the effectiveness of the proposed two-stage semi-supervised training approach.

  * $\bullet$

Supervised – The view adjustment model is trained using only the labeled data
from the view adjustment dataset. The unlabeled images from _Open Images_ are
not used.

  * $\bullet$

Aesthetic-scoring – Motivated by existing image cropping approaches [40, 10],
we use aesthetic scores to train the image scoring model. Specifically, we
train the scoring model to regresses the mean opinion scores of 250K images
from the _AVA_ [30] dataset, which is a widely-used image aesthetics dataset.

  * $\bullet$

Supervised-scoring – The composition scoring model is trained using only
labeled data from FCDB, GAICD and CPC. The unlabeled images from _Unsplash_
are not used.

The network architecture and training details are shared across all methods.

#### Implementation details

Both the view adjustment and composition scoring models use MobileNet [14]
architecture with input size $299{\times}299$. The last convolution layer is
followed by a spatial pyramid pooling layer ($1{\times}1$, $2{\times}2$ and
$5{\times}5$) [13] and two fully-connected layers with 1,024 units and ReLU
activation. The composition scoring model has an output layer with a single
node that uses the sigmoid activation function, and the view adjustment
prediction model has an output layer with three heads as described in Sec.
3.4.

All models are trained asynchronously on 5 P100 GPUs starting from ImageNet
[9] pretrained weights. We use the ADAM [17] optimizer with a learning rate of
$2e^{-5}$ and weight decay of $5e^{-4}$. The composition scoring models are
trained for 250K steps and the view adjustment models are trained for 480K
steps. Because the distribution of view adjustments in the wild is unknown, we
give equal importance to all adjustments and weight each sample by the inverse
frequency of the adjustment predictor label.

###  4.1 Evaluation Dataset and Metrics

A major contribution of this work is introducing a benchmark dataset for
evaluating view adjustment models. This section describes the evaluation
dataset and the proposed evaluation metrics.

The proposed view adjustment evaluation dataset is created from the test
splits of image cropping datasets _FCDB_ and _GAICD_ , following the procedure
described in Sec. 3.1. For each image, we generate an evaluation sample for
each candidate view adjustment by randomly sampling a magnitude in the range
$[5,45]$ for shift and zoom adjustment and $[\frac{\pi}{36},\frac{\pi}{4}]$
for rotation. See Table 1 for statistics. Note that the possible perturbations
for an image are determined by the size of its best crop. For example,
horizontal shift and zoom-out are not possible for a crop whose width is same
as the image width. Therefore, each candidate view adjustments has a different
number of evaluation samples.

Horizontal | Vertical | Zoom | Rotate | Total
---|---|---|---|---
258 (Left) | 370 (Up) | 268 (In) | 255 (Clockwise) | 3,076
277 (Right) | 350 (Down) | 521 (Out) | 256 (Counter) |
Table 1: Number of evaluations samples for each candidate view adjustment. |  |  | F1 score |
---|---|---|---|---
| AUC | TPR | Left | Right | Up | Down | Zoom-in | Zoom-out | Clockwise | Counter | IoU
Supervised | 0.570 | 0.366 | 0.126 | 0.074 | 0.0 | 0.0 | 0.022 | 0.364 | 0.082 | 0.061 | 0.730
Aesthetic-scoring | 0.536 | 0.352 | 0.052 | 0.115 | 0.0 | 0.0 | 0.102 | 0.302 | 0.061 | 0.088 | 0.717
Supervised-scoring | 0.536 | 0.356 | 0.078 | 0.082 | 0.164 | 0.234 | 0.007 | 0.350 | 0.118 | 0.117 | 0.737
Proposed | 0.608 | 0.436 | 0.221 | 0.221 | 0.390 | 0.341 | 0.015 | 0.378 | 0.124 | 0.110 | 0.750
Table 2: Performance of view adjustment models trained with different
strategies. The TPR, IoU and F1 scores are measured at 0.3 FPR for the
suggestion predictor.

We propose the following objective metrics for evaluating view adjustment
models. The goal is to evaluate 1) how accurately does a model trigger
suggestions, 2) whether the suggested adjustment is correct when the model
provides a suggestion, and 3) how well the suggested view approximates the
best view.

  * $\bullet$

_Area under receiver operating characteristics curve (AUC)_ measures the
performance of the suggestion predictor, i.e., measures how accurately a model
triggers suggestions.

  * $\bullet$

_True positive rate (TPR)_ also measures the performance of the suggestion
predictor. More specifically, we measure the TPR at $0.3$ False Positive Rate
(FPR). We choose a low FPR because suggesting an adjustment when it is not
needed has a higher cost (asking the user to do unnecessary work and making
the composition worse) compared to not suggesting an adjustment when the
composition can be improved. All other metrics are computed at the same $0.3$
FPR operating point.

  * $\bullet$

_F1-score_ for each candidate adjustment measures the accuracy of the
adjustment predictor and how accurate we are on the suggested adjustment.

  * $\bullet$

_Intersection over Union (IoU)_ between the predicted and ground truth views
measures how well the suggested view approximates the best view. It evaluates
the performance of the suggestion predictor, adjustment predictor, and
magnitude predictor jointly.

Besides the objective metrics, we also conduct subjective evaluation through a
user study.

Figure 5:  Confusion matrix. We normalize along each row, so each cell shows
the precision of adjustment prediction.

###  4.2 Objective Evaluation

Table 2 summarizes the performance of various methods on the view adjustment
evaluation dataset. Overall, the proposed semi-supervised approach achieves
the best performance when compared with supervised alternatives. Our model
clearly outperforms supervised approaches in determining whether a suggestion
should be provided. It achieves $43.6\%$ TPR when the FPR is $30\%$.
Similarly, our model achieves the best F1-score for all candidate adjustments
except for the _Zoom-in_ and _Counter_ clockwise rotation adjustment. The
absolute F1-score for _Zoom-in_ and _Rotation_ adjustments are low because of
low recall values. This may be explained by the adjustment magnitudes
distribution, where most zoom-in and rotation samples have small perturbation
magnitudes and are by nature more difficult for view adjustment prediction.
Finally, our method achieves the best IoU, which means that the suggested
views best approximate the ground truth views. We also compare with off-the-
shelf cropping models [42, 48] on our view adjustment dataset, but their
performance is significantly worse than our method. The IoU between predicted
crops and ground truth views is below 0.61. The results justify that cropping
is not sufficient for solving the view adjustment problem.

To further understand when our model fails, we show the confusion matrix in
Fig. 5. The results show that our model rarely makes the mistake of choosing
the wrong shift or rotation adjustment. Instead, most of the errors occur
between zoom-out and shifting. This is not surprising, because both zoom-out
and shifting can resolve the problem where the original view wrongly excludes
important content.

Comparing the performance of our model with the supervised alternatives, we
can clearly see that the two-stage approach helps to improve model
performance. The results also show that semi-supervised learning is important
not only for the view adjustment model but also for the composition scoring
model. If we train the composition scoring model using only supervised data,
the results are actually worse than the purely supervised view adjustment
model. This is potentially due to the poor quality of pseudo labels generated
by the supervised composition scoring model, which is trained on a limited
number of images. Finally, the performance of Aesthetic-scoring is worse than
other methods. This suggests that a generic aesthetic score is not sufficient
to differentiate the changes caused by camera view adjustment.

| After is better | Before is better | No difference
---|---|---|---
View Adjustment | $79.0\%$ | $12.6\%$ | $8.5\%$
Pano2Vid [37] | $78.9\%$ | $17.1\%$ | $4.1\%$
Table 3:  Subjective evaluation results on the view adjustment and the
Pano2Vid dataset. The raters are asked whether each image has better
composition before or after applying the suggested view adjustment.  Figure 6:
Qualitative examples. Each pair shows the original image (left) and the result
of the adjustment (right). Our model appears to provide suggestions based on
various factors, including objects, the ground plane, or even leading lines in
the image. The last two rows show examples where our model predicts the wrong
adjustment but still improves the image composition.

###  4.3 Subjective Evaluation

To demonstrate the effectiveness of the proposed model, we conduct a user
study that directly evaluates whether the suggested view adjustment improves
the composition of the original image. In the user study, we show the raters
the image both before and after applying the suggested view adjustment. The
raters are asked which image has the better composition or if they cannot
tell. The order of the two images is chosen randomly to avoid bias. We select
$130$ images from our evaluation dataset for the user study, and each image is
rated by 3 different raters. The results are in Table 4. It shows that our
model has a high success rate ($79.0\%$) when a suggestion is provided, and it
only wrongly suggests an adjustment about $12.6\%$ of the time. See Fig. 6 and
Fig. 7 for qualitative examples.

Beside the view adjustment dataset, we also conduct a user study on the
_Pano2Vid_ [37] dataset. _Pano2Vid_ is a $360\degree$ video dataset with
viewport annotations. The viewports capture a normal FOV video created from
the $360\degree$ video by raters and cover interesting content with
appropriate composition. We sample $200$ frames from the _Pano2Vid_ dataset
and apply our model on the user-selected viewport. The horizontal and vertical
adjustments become translations in spherical coordinates, and zoom-in and
zoom-out adjustments become changes in the FOV. We run the model iteratively
until the model stops suggesting any adjustment or reaches the maximum number
of steps (3 steps), and we ask raters whether the final view has a better
composition than the initial view.

The results on _Pano2Vid_ are in Table 4. Interestingly, the success rate of
the model remains high but has a slightly higher failure rate. The results
show that our model generalizes well despite the fact that it has never been
trained on $360\degree$ images. See Fig. 8 for examples. Note that we perform
multi-step adjustment on _Pano2Vid_ , so the results also imply that our model
can improve the composition iteratively even if it is trained only for single
step view adjustment. This is important in practice, because the best view may
not be reachable from the initial view using a single adjustment when applying
the system in the wild. On the other hand, we also observe that our model may
drift far from the initial view in some examples. In other words, it suggests
a final view that is unrelated to the initial view, which shows a limitation
of the single-step adjustment model.

(a) False negatives.

(b) False positives.

Figure 7:  Failure examples. False negative means that our model fails to
provide a suggestion when the composition can be improved. Each pair shows the
input image (left) and the ground truth (right). False positive means the
suggested adjustment degrades the composition according to our user study.
Each pair shows the input image (left) and our prediction (right). Our model
fails when the target adjustment is minor or the important content is not
visible in the initial view. Also, the bottom right example shows how a wrong
magnitude prediction degrades the composition.  Figure 8:  Qualitative
examples from $360\degree$ images. We perform multi-step view adjustment on
$360\degree$ images, and our model improves the composition iteratively.

##  5 Conclusion

We propose improving image composition through view adjustment prediction. Our
system suggests how to improve the composition while the user is still
composing the photo. We create a new benchmark dataset for evaluating the
performance of a view adjustment prediction and propose a two-stage training
approach that exploits both labeled and unlabeled data for learning the view
adjustment model. Experimental results show that the suggestion provided by
our model improves image composition $79\%$ of the time. Future plans are to
explore the iterative nature of view adjustment for more accurate and
consistent suggestions.

## References

  * [1] Camera51–a smarter camera.  https://play.google.com/store/apps/details?id=com.camera51.android.  Accessed: 2020-09-22.
  * [2] Unsplash dataset.  https://unsplash.com/data.
  * [3] What is shot suggestions on galaxy?  https://www.samsung.com/global/galaxy/what-is/shot-suggestions/.  Accessed: 2020-09-22.
  * [4] Jiansheng Chen, Gaocheng Bai, Shaoheng Liang, and Zhengqin Li.  Automatic image cropping: A computational complexity study.  In CVPR, 2016.
  * [5] Li-Qun Chen, Xing Xie, Xin Fan, Wei-Ying Ma, Hong-Jiang Zhang, and He-Qin Zhou.  A visual attention model for adapting images on small displays.  Multimedia systems, 9(4):353–364, 2003.
  * [6] Yi-Ling Chen, Tzu-Wei Huang, Kai-Han Chang, Yu-Chen Tsai, Hwann-Tzong Chen, and Bing-Yu Chen.  Quantitative analysis of automatic image cropping algorithms:a dataset and comparative study.  In WACV, 2017.
  * [7] Yi-Ling Chen, Jan Klopp, Min Sun, Shao-Yi Chien, and Kwan-Liu Ma.  Learning to compose with professional photographs on the web.  In ACM Multimedia, 2017.
  * [8] Bin Cheng, Bingbing Ni, Shuicheng Yan, and Qi Tian.  Learning to photograph.  In ACM Multimedia, 2010.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [10] Yubin Deng, Chen Change Loy, and Xiaoou Tang.  Aesthetic-driven image enhancement by adversarial learning.  In ACM Multimedia, 2018.
  * [11] Chen Fang, Zhe Lin, Radomir Mech, and Xiaohui Shen.  Automatic image cropping using visual composition, boundary simplicity and content preservation models.  In ACM Multimedia, 2014.
  * [12] Hui Fang and Meng Zhang.  Creatism: A deep-learning photographer capable of creating professional work.  arXiv preprint arXiv:1707.03491, 2017.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ECCV, 2014.
  * [14] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  CoRR, abs/1704.04861, 2017.
  * [15] Chuan-Shen Hu, Yi-Tsung Hsieh, Hsiao-Wei Lin, and Mei-Chen Yeh.  Virtual portraitist: An intelligent tool for taking well-posed selfies.  ACM Trans. Multimedia Comput. Commun. Appl., 15(1), Jan. 2019.
  * [16] Yen-Ta Huang, Kuan-Ting Chen, Liang-Chi Hsieh, Winston Hsu, and Ya-Fan Su.  Detecting the directions of viewing landmarks for recommendation by large-scale user-contributed photos.  In ACM Multimedia, 2012.
  * [17] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [18] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy.  Openimages: A public dataset for large-scale multi-label and multi-class image classification.  Dataset available from https://github.com/openimages, 2017.
  * [19] Debang Li, Huikai Wu, Junge Zhang, and Kaiqi Huang.  A2-rl: Aesthetics aware reinforcement learning for image cropping.  In CVPR, 2018.
  * [20] Debang Li, Huikai Wu, Junge Zhang, and Kaiqi Huang.  Fast a3rl: Aesthetics-aware adversarial reinforcement learning for image cropping.  IEEE Transactions on Image Processing, 28(10):5105–5120, 2019.
  * [21] Debang Li, Junge Zhang, and Kaiqi Huang.  Learning to learn cropping models for different aspect ratio requirements.  In CVPR, 2020.
  * [22] Debang Li, Junge Zhang, Kaiqi Huang, and Ming-Hsuan Yang.  Composing good shots by exploiting mutual relations.  In CVPR, 2020.
  * [23] Ligang Liu, Renjie Chen, Lior Wolf, and Daniel Cohen-Or.  Optimizing photo composition.  Computer Graphics Forum, 29(2):469–478, 2010.
  * [24] Peng Lu, Jiahui Liu, Xujun Peng, and Xiaojie Wang.  Weakly supervised real-time image cropping based on aesthetic distributions.  In ACM Multimedia, 2020.
  * [25] Peng Lu, Hao Zhang, Xujun Peng, and Xiaofu Jin.  An end-to-end neural network for image cropping by learning composition from aesthetic photos.  arXiv preprint arXiv:1907.01432, 2019.
  * [26] Weirui Lu, Xiaofen Xing, Bolun Cai, and Xiangmin Xu.  Listwise view ranking for image cropping.  IEEE Access, 7:91904–91911, 2019.
  * [27] Shuang Ma, Yangyu Fan, and Chang Wen Chen.  Finding your spot: A photography suggestion system for placing human in the scene.  In ICIP, 2014.
  * [28] Shuang Ma, Yangyu Fan, and Chang Wen Chen.  Pose maker: A pose recommendation system for person in the landscape photographing.  In ACM Multimedia, 2014.
  * [29] Luca Marchesotti, Claudio Cifarelli, and Gabriela Csurka.  A framework for visual saliency detection with applications to image thumbnailing.  In ICCV, 2009.
  * [30] Naila Murray, Luca Marchesotti, and Florent Perronnin.  AVA: A large-scale database for aesthetic visual analysis.  In CVPR, 2012.
  * [31] Masashi Nishiyama, Takahiro Okabe, Yoichi Sato, and Imari Sato.  Sensation-based photo cropping.  In ACM Multimedia, 2009.
  * [32] Yogesh Singh Rawat and Mohan S Kankanhalli.  Context-aware photography learning for smart mobile devices.  ACM Transactions on Multimedia Computing, Communications, and Applications, 12(1s):1–24, 2015.
  * [33] Yogesh Singh Rawat and Mohan S Kankanhalli.  Clicksmart: A context-aware viewpoint recommendation system for mobile photography.  IEEE Transactions on Circuits and Systems for Video Technology, 27(1):149–158, 2016.
  * [34] Y. S. Rawat, M. Song, and M. S. Kankanhalli.  A spring-electric graph model for socialized group photography.  IEEE Transactions on Multimedia, 20(3):754–766, 2018.
  * [35] Fred Stentiford.  Attention based auto image cropping.  In International Conference on Computer Vision Systems: Proceedings (2007), 2007.
  * [36] Hsiao-Hang Su, Tse-Wei Chen, Chieh-Chi Kao, Winston H. Hsu, and Shao-Yi Chien.  Preference-aware view recommendation system for scenic photos based on bag-of-aesthetics-preserving features.  IEEE Transactions on Multimedia, 14(3):833–843, 2012.
  * [37] Yu-Chuan Su, Dinesh Jayaraman, and Kristen Grauman.  Pano2vid: Automatic cinematography for watching 360$\degree$ videos.  In ACCV, 2016.
  * [38] Bongwon Suh, Haibin Ling, Benjamin B Bederson, and David W Jacobs.  Automatic thumbnail cropping and its effectiveness.  In UIST, 2003.
  * [39] Yi Tu, Li Niu, Weijie Zhao, Dawei Cheng, and Liqing Zhang.  Image cropping with composition and saliency aware aesthetic score map.  In AAAI, 2020.
  * [40] Wenguan Wang and Jianbing Shen.  Deep cropping via attention box prediction and aesthetics assessment.  In ICCV, 2017.
  * [41] Yinting Wang, Mingli Song, Dacheng Tao, Yong Rui, Jiajun Bu, Ah Chung Tsoi, Shaojie Zhuo, and Ping Tan.  Where2stand: A human position recommendation system for souvenir photography.  ACM Trans. Intell. Syst. Technol., 7(1), Oct. 2015.
  * [42] Zijun Wei, Jianming Zhang, Xiaohui Shen, Zhe Lin, Radomir Mech, Minh Hoai, and Dimitris Samaras.  Good view hunting: Learning photo composition from dense view pairs.  In CVPR, 2018.
  * [43] Pengfei Xu, Hongxun Yao, Rongrong Ji, Xian-Ming Liu, and Xiaoshuai Sun.  Where should i stand? learning based human position recommendation for mobile photographing.  Multimedia tools and applications, 69(1):3–29, 2014.
  * [44] Jianzhou Yan, Stephen Lin, Sing Bing Kang, and Xiaoou Tang.  Learning the change for automatic image cropping.  In CVPR, 2013.
  * [45] Wenyuan Yin, Tao Mei, and Chang Wen Chen.  Crowdsourced learning to photograph via mobile devices.  In ICME, 2012.
  * [46] Wenyuan Yin, Tao Mei, Chang Wen Chen, and Shipeng Li.  Socialized mobile photography: Learning to photograph with social context via mobile devices.  IEEE Transactions on Multimedia, 16(1):184–200, 2013.
  * [47] Hui Zeng, Lida Li, Zisheng Cao, and Lei Zhang.  Reliable and efficient image cropping: A grid anchor based approach.  In CVPR, 2019.
  * [48] Hui Zeng, Lida Li, Zisheng Cao, and Lei Zhang.  Grid anchor based image cropping: A new benchmark and an efficient model.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [49] Mingju Zhang, Lei Zhang, Yanfeng Sun, Lin Feng, and Weiying Ma.  Auto cropping for digital photographs.  In ICME, 2005.
  * [50] Yanhao Zhang, Xiaoshuai Sun, Hongxun Yao, Lei Qin, and Qingming Huang.  Aesthetic composition representation for portrait photographing recommendation.  In ICIP, 2012.
  * [51] Y. Zhang and R. Zimmermann.  Camera shooting location recommendations for landmarks in geo-space.  In International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems, 2013.

## Appendices

The supplementary materials consist of:

  1. A

Perturbation details for the composition scoring model

  2. B

Perturbation details for the view adjustment model

  3. C

Data augmentation details

  4. D

Training pipeline overview

  5. E

Experiment results analysis

  6. F

Additional qualitative examples

###  A Perturbation for Composition Scoring

In this section, we describe the perturbation details for generating the image
pairs $(I_{p},I_{n})$ for training the composition scoring model.

Let $c_{x}$, $c_{y}$, $w$ and $h$ denote the center $x$, $y$ coordinate, width
and height of the best crop bounding box normalized to $[0,1]$ for each image.
Given the perturbation ($o_{x}$, $o_{y}$, $o_{z}$, $o_{\alpha}$) where
$o_{x}$, $o_{y}$, $o_{z}$, and $o_{\alpha}$ are the magnitudes of horizontal
shift, vertical shift, zoom, and rotation respectively, the perturbed bounding
box is defined as

$\displaystyle c^{\prime}_{x}$ $\displaystyle=c_{x}+w*o_{x}$ (2)
$\displaystyle c^{\prime}_{y}$ $\displaystyle=c_{y}+h*o_{y}$ $\displaystyle
w^{\prime}$ $\displaystyle=w+w*o_{z}$ $\displaystyle h^{\prime}$
$\displaystyle=h+h*o_{z}$ $\displaystyle\alpha$ $\displaystyle=o_{\alpha}.$

Note that $\alpha$ is the orientation of the bounding box in the image
coordinate system and is defined as the counter clockwise angle between the
$y$-axis of the image and the bounding box. Therefore, the coordinates of the
bounding box corners in the image are given by $u=c+R_{\alpha}v$, where
$R_{\alpha}$ is the rotation matrix and $v=[\pm\frac{w}{2},\pm\frac{h}{2}]$.

Specifically, we apply four types of perturbation:

  * $\bullet$

Shifting randomly samples $o_{x}$ and $o_{y}$ in the range $[-0.4,0.4]$ and
applies $(o_{x},o_{y},0,0)$ to the bounding box.

  * $\bullet$

Zooming-out randomly samples $o_{z}$ in the range $[0,0.4]$ and applies
$(0,0,o_{z},0)$ to the bounding box.

  * $\bullet$

Cropping combines zooming-in and shifting. It first samples $o_{z}$ in the
range $[\sqrt{0.5},\sqrt{0.8}]$ and then samples $o_{x}$ and $o_{y}$ in the
range $[-o_{z}/2,o_{z}/2]$. The perturbation $(o_{x},o_{y},o_{z},0)$ is then
applied to the bounding box. In other words, it randomly sample a crop within
the original crop where the area is 0.5 to 0.8 times that of the original one
and the aspect ratio is the same.

  * $\bullet$

Rotation randomly samples $o_{\alpha}$ in the range
$[-\frac{\pi}{4},\frac{\pi}{4}]$ and applies $(0,0,0,o_{\alpha})$ to the
bounding box.

We apply the same perturbations for both the labeled crops and unlabeled
images. For unlabeled image, the bounding box is set to the entire image, i.e.
$c_{x}=c_{y}=0.5$, and $w=h=1.0$. Also, as described in the main paper, we
discard samples where the perturbed bounding box goes beyond the original
image for supervised samples.

###  B Perturbation for View Adjustment

This section describes how we generate samples for view adjustment prediction.
The generated samples are used for both training and evaluation.

Given an image crop represented as $(c_{x},c_{y},w,h)$, we apply the following
perturbations:

  * $\bullet$

Horizontal shift samples $o_{x}$ in either $[0.05,0.45]$ or $[-0.45,-0.05]$
and applies perturbation $(o_{x},0,0,0)$ to the bounding box.

  * $\bullet$

Vertical shift samples $o_{y}$ in either $[0.05,0.45]$ or $[-0.45,-0.05]$ and
applies perturbation $(0,o_{y},0,0)$ to the bounding box.

  * $\bullet$

Zoom samples $o_{z}$ in either $[-0.048,-0.310]$ or $[0.053,0.818]$ and
applies $(0,0,o_{z},0)$ to the bounding box.

  * $\bullet$

Rotation samples $o_{\alpha}$ in either $[-\frac{\pi}{4},-\frac{\pi}{36}]$ or
$[\frac{\pi}{36},\frac{\pi}{4}]$ and applies $(0,0,0,o_{\alpha})$ to the
bounding box.

Note that the ranges for shift and zoom perturbations are determined such that
the inverse perturbation, i.e., the ground truth view adjustment, falls in the
range of $[0.05,0.45]$. As described in the main paper, we discard samples
where the perturbed bounding box falls outside the image.

Result: Composition scoring model $M_{c}$

for  _$i\leftarrow 1,\text{MaxStep}$_ do

// Scored candidate samples

$I_{s}\,,crop_{s}\leftarrow$ $N$ images with scored crops

$crop_{p},crop_{n}\leftarrow$ create $\frac{N\times(N-1)}{2}$ pairs of crops

$I^{p}_{s}\leftarrow$ ExtractCrop($I_{s}$, $crop_{p}$)

$I^{n}_{s}\leftarrow$ ExtractCrop($I_{s}$, $crop_{n}$)

// Best crop samples

$I_{c}\,,crop_{c}\leftarrow$ $K$ images with best crop

$crop^{\prime}_{c}\leftarrow$ Perturb($crop_{c}$)

$I^{p}_{c}\leftarrow$ ExtractCrop($I_{c}$, $crop_{c}$)

$I^{n}_{c}\leftarrow$ ExtractCrop($I_{c}$, $crop^{\prime}_{c}$)

// Unsupervised samples

$I^{p}_{u}\leftarrow$ $P$ well-composed images

$I^{n}_{u}\leftarrow$ Perturb($I^{p}_{u}$)

// Data augmentation

$I^{i}_{j}\leftarrow$ Augment($I^{i}_{j}$)
$\forall(i,j)\in\\{p,n\\}\times\\{s,c\\}$

$I^{p}_{u}\leftarrow$ Augment($I^{p}_{u}$)

// Minimize joint loss

$y^{p}\leftarrow M_{c}(I^{p})$

$y^{n}\leftarrow M_{c}(I^{n})$

$M_{c}\leftarrow\operatorname*{arg\,min}_{M_{c}}\,\sum_{x}Loss(y^{p}_{x},y^{n}_{x})$

end for

Algorithm 1 Composition scoring model training

###  C Data Augmentation

This section describes the implementation details for the composition scoring
model data augmentation. For _shift borders_ augmentation, we randomly select
a percentage $s_{y}$ in $[0,40]$ and select $s_{x}\%$ of columns on either the
left or right side of the image and replace the pixel values with zero.
Similarly, we randomly select $s_{y}\%$ of rows from either the top or bottom
of the image and replace the pixel values. Note that $s_{x}$ and $s_{y}$ are
samples independently. For _zoom-out_ augmentation, we randomly select a
percentage $s_{z}$ in $[0,40]$ and select $0.5s_{z}\%$ of rows and columns on
both sides of the image. For _rotation_ augmentation, we randomly select an
angle $\theta\in[-\frac{\pi}{4},\frac{\pi}{4}]$ and apply two rotation
operator $R_{-\theta}\circ R_{\theta}$ to the image sequentially. Note that
while $R_{-\theta}\circ R_{\theta}=I$ mathematically, if we apply the two
operator sequentially on an image with finite size, they introduce pixels not
within the original image and therefore simulate the synthetic artifacts
introduced by rotation.

###  D Training Pipeline Overview

This section provides an overview of the training pipeline. See Algorithm 1
and Algorithm 2 for the training of the composition scoring and view
adjustment model respectively.

Result: View adjustment model $M_{v}$

for  _$i\leftarrow 1,\text{MaxStep}$_ do

// Pseudo-labeled samples

$I_{s}\leftarrow$ $K$ images

$y_{s}\leftarrow$ PseudoLabel($I_{s}$)

// Labeled samples

$I_{c}\,,crop_{c}\leftarrow$ $K$ images with best crop

$p_{c}\leftarrow$ samples random perturbations

$crop^{\prime}_{c}\leftarrow$ Perturb($crop_{c}$, $p_{c}$)

$I^{\prime}_{c}\leftarrow$ ExtractCrop($I_{c}$, $crop^{\prime}_{c}$)

$y_{c}\leftarrow p^{-1}_{c}$

// Minimize joint loss

$y^{\prime}_{s}\leftarrow M_{v}(I_{s})$

$y^{\prime}_{c}\leftarrow M_{v}(I^{\prime}_{c})$

$M_{v}\leftarrow\operatorname*{arg\,min}_{M_{v}}\,Loss(y_{s},y^{\prime}_{s})+Loss(y_{c},y^{\prime}_{c})$

end for

Algorithm 2 View adjustment model training

###  E Experiment Results

| After is better | Before is better | No difference
---|---|---|---
Horizontal | $80.0\%$ | $5.0\%$ | $15.0\%$
Vertical | $94.2\%$ | $0.0\%$ | $5.8\%$
Zoom | $74.7\%$ | $16.0\%$ | $9.3\%$
Rotation | $57.3\%$ | $41.3\%$ | $1.3\%$
Single-step | $81.6\%$ | $14.9\%$ | $3.5\%$
Multi-steps | $72.2\%$ | $22.2\%$ | $5.6\%$
Table 4:  Subjective evaluation results on the view adjustment (row1–row4) and
the Pano2Vid (row5–row6) dataset.

In this section, we expand the experiment results in the main paper by
providing more detailed analysis on the subjective evaluation results. In
particular, we analyze the user study results w.r.t. 1) different view
adjustments being suggested on the view adjustment dataset, and 2) the number
of adjustment being performed on the Pano2Vid dataset.

The results are in Table 4. Interestingly, the subjective evaluation results
have a strong dependency on the adjustment being suggested. Our method
achieves the best performance in vertical shift adjustment, where it improves
the composition more than $94\%$ of the time without any false positive. In
contrast, the rotation suggestion only improves the composition $57.3\%$ of
the time. This is primarily caused by the incorrect magnitude prediction,
because image composition is very sensitive to the orientation when the
horizon is visible. Therefore, even a small error in magnitude prediction may
significantly degrade the result.

Not surprisingly, our method has a better performance in single-step view
adjustment. In contrast, the success rate drops by $9.4\%$ when multiple
adjustments are required. The results show the need for considering the
iterative nature of view adjustment in future works.

###  F Qualitative Results

In this section, we show additional qualitative examples similar to Fig. 6 in
the main paper. See Fig. 9 and Fig. 10. The results show that our model works
in a wide range of scenarios, including both object centric and scenic photos.
Also, our model works for different aspect ratios, including photos in both
landscape and portrait orientations.

Figure 9:  Qualitative examples. Each pair shows the original image (left) and
our prediction (right).  Figure 10:  Qualitative examples (cont.). Each pair
shows the ground truth (left) and our prediction (right).