figurec

# Content-aware media retargeting based on deep importance map

Thi-Ngoc-Hanh Le  National Cheng-Kung University, Taiwan  Shih-Syun Lin
National Taiwan Ocean University, Taiwan  Weiming Dong  Chinese Academy of
Science and School of Artificial Intelligence, China.  Tong-Yee Lee  National
Cheng-Kung University, Taiwan

###### Abstract

We present a neural network to estimate the visual information of important
pixels in image and video, which is used in content-aware media retargeting
applications. Existing techniques are successful in proposing retargeting
methods. Yet, the serious distortion and the shrunk problem in the retargeted
results still need to be investigated due to the limitations in the methods
used to analyze visual attention. To accomplish this, we propose a network to
define the importance map, which is sufficient to describe the energy of the
significant regions in image/video. With this strategy, more ideal results are
obtained from our system. Besides, the objective evaluation presented in this
paper shows that our media retargeting system can achieve better and more
plausible results than those of other works. In addition, our proposed
importance map performs well in the enlarge operator and on the difficult-to-
resize images.

##  1 Introduction

Media retargeting has been an active and attractive research topic in both
computer vision and computer graphics in recent years due to the evolution of
heterogeneous devices for capturing images and videos as well as for
displaying them. Their displays also have different resolutions and aspect
ratios. Thus, methods for changing the sizes of images and videos are gaining
importance. A content-aware image/video retargeting method has been mentioned
as an algorithm which focuses on the aspect that important objects in the
image should be preserved when it is retargeted to other display resolutions.

The conventional content-aware image/video retargeting methods [2, 3, 4, 5, 1,
6] typically rely on the visual information of image/video to define the
importance in image/video which should be preserved after retargeting. The
objective of a retargeting method is to preserve either the importance of
image or the ratio of objects. These approaches have been successful in
maintaining the importance of image/video and hindering the distortion in the
retargeted output. However, they solely focus on designing a good retargeting
technique. The methods, which they use to identify importance in image/video,
are not sufficient to describe the visual information of the significant
regions. Such as the results in [2] are distorted seriously with the gradient
map. Or, Lin et al. [1, 6] generate shrunk results due to the lack of energy
on the important regions (as the examples shown in Fig.1). Some recent works
[7, 8, 9] attempt to adopt deep learning techniques to work on the problems in
this research field. However, the retargeting results are still need to
investigated. Besides, the challenge in building a large image retargeting
dataset asks for the retargeted images not only to maintain what people look
forward to but also to achieve aesthetic quality demands.

Figure 1: Visualization of the limitations in the previous results. The first
row, from left to right: original image, Lin et al. [1]’s result, our result.
The second row, from left to right: original image, Avidan and Shamir [2]’s
result, our result.

Our current work is motivated by seam carving and warping based method which
is studied in [2, 1, 6]. We propose a network to estimate the importance of
image/video frames. It is then employed in a comprehensive framework to
produce retargeting results. With our system, we tackle the aforementioned
drawbacks and boost the retargeting results more ideal. Our contributions can
be summarized in the following issues:

  * $\bullet$

We introduce a comprehensive system which is able to produce the better media
retargeting results for seam carving-based and warping-based method.

  * $\bullet$

We adopt a deep learning method to estimate the important pixels of a given
image/video which is used for image and video retargeting applications.

  * $\bullet$

More plausible retargeted images and videos are obtained from our proposed
system.

We organize the remainder of this paper as follows. Section 2 is a brief
review on the related studies. Section 3 presents the proposed network and
explains how our importance map tackles the problems in image and video
retargeting. In Section 4, the evaluation on the results is described and a
discussion on our results follows. Finally, section 5 presents the conclusion
and our future work.

##  2 Related work

In this section, previous studies, which are close to our current approach,
are reviewed. They consist of (1) the media retargeting techniques including
conventional approaches and deep learning-based approaches, and (2) the
schemes that are proposed to improve media retargeting results.

The existing approaches for Content Aware Image Retargeting (CAIR) are
probably categorized into discrete and continuous methods [10]. In both two
categories, the pipeline for retargeting firstly extracts the importance from
the input image/video frames then utilizes these information to drive the
input image/video frames to the target size by an resizing operator (as
demonstrated in Fig.2). Relying on this workflow, importance in the input
image is considered as a required process in a resizing system proposed by
[11, 12, 2, 4, 5, 1, 6]. These typical resizing methods utilize different
methods to determine the image importance in their own framework.

A naive approach was early used in resizing an image in terms of cropping. The
main challenge of cropping is finding an ideal cropping window. To overcome
this challenge, cropping techniques begin by calculating an importance map.
This map is obtained from a saliency method that is introduced in [13].

Figure 2: The basic workflow of the content-aware media retargeting frameworks
[10].

Seam carving [2] has been mentioned as the first method in content-aware image
retargeting. The main idea of this method is to remove seams that consist of
low-importance pixels to drive an image to its target size. To estimate the
weights of pixels, this approach utilizes the gradient-based energy map of the
input image and removes a seam where the energy is minimum. Inspired by this
concept, the studies in [14, 15, 16, 17] are lately introduced to improve the
fundamental concept. Most of these works perform well in landscape images or
pictures with large homogeneous areas. However, in the cases that images
consist of straight line, regular shapes or numerous objects, visual artifacts
can occur in [10]. Besides, all less-important regions have been removed leads
to serve artifacts.

While scaling resizes the entire image uniformly by the same scale factor,
warping is content-adaptive and determines scaling factors in a fine-grained
way [10]. Proposed in [18], this study tries to preserve important content in
an image. To measure the importance, an importance map is created consisting
of a saliency map computed by Ma and Zhang [19]. Yet, image features outside
the region of interest may suffer from significant distortions.

For the warping-based approach, Zhang et al. [20] proposed a method with the
attempts to ensure that important local regions undergo a geometric similarity
transformation as well as preserve image edge structure. To achieve this, an
importance map of the source image is first defined. Based on it, each
important region is assigned a value. Guo et al. [21] presented an image
retargeting method using a mesh image representation. A saliency map of the
source image is first produced by method proposed in Ma and Zhang [19]. It is
then associated with the source mesh. In this way, retargeting is transformed
into a parametrization problem of finding a target mesh with the desired
resolution. Jin et al. [3] introduced an approach for interactive content-
aware image resizing. The image saliency information of image is captured to
define visual information in the input image. A triangle mesh is then created.
The scales of all triangles are obtained by solving an optimization problem.
To preserve visually salient objects and structure lines in retargeting an
image, Lin et al. [1] proposed a novel scheme in term patch-based. In this
framework Lin et al. [1], input image is first segmented to partion image. A
saliency map method [22] is then employed. Each segmented patch is assigned an
average saliency value. However, this method can not work well for the images
contain with similar significant values [1].

In term of video retargeting, Wolf et al. [23] early introduced an efficient
algorithm for video retargeting. To detect the importance of each region in
frames, frames extracted from the input video are first analyzed. The proposed
analysis is fully automatic and based on local saliency, motion detection and
object detectors. A transformation is then applied with the respects that the
analysis shrinks less important regions more than importance ones. Motivated
by seam carving method, Rubinstein et al. [14] proposed to improve seam
carving method in video retargeting. Instead of removing 1D seams from 2D
images, they remove 2D seam manifolds from 3D space-time volumes. To reduce
the unpleasant distortion, Lin et al. [6] presented a novel continuous
approach in video retargeting. This approach proposed an object-preserving
warping scheme with object-based significance estimation. In this scheme,
visual salient objects in 3D space-time space are forced to undergo as-rigid-
as-possible warping, while low significance content are warped as close as
possible to linear scaling.

From above, even each system utilizes a different method to define the
importance in image/video frames, this process is considered as a prerequisite
stage in such retargeting framework. With the attempt to boost the retargeting
results better, the following methods improved the drawbacks in previous works
by different ways. To avoid artifacts in the conventional seam carving,
Achanta and Süsstrunk [24] introduced a saliency dectection method. Unlike
gradient maps, which may have to be recomputed several times during a seam
carving based retargeting operation, saliency maps in Achanta and Süsstrunk
[24] are computed once independent of the number of seams added or removed.
Kim et al. [25] developed a visual saliency measure to identify the important
image regions for obtaining the image importance map. They propose a robust
importance measure of self-ordinal resemblance (SOR) defined by computing
distances between ordinal signature of edge and color orientation histograms
obtained from center and surrounding regions. After calculating image
importance map, they employ the improved seam carving [14] for retargeting.
Wang et al. [26] presented salient edge and region aware image retargeting
(SERAR) method, by decomposing an image into a cartoon and a texture part to
provide the reliable salient edges from cartoon part. Based on the proposed
image importance map, they modified the seam carving to avoid distorting the
geometric structures.

Recently, with the advances in learning-based approaches, researchers have
attempted to use convolutional neural network to define the importance in
images. Song et al. [8] proposed an improved content-aware image resizing
method that uses deep learning. The proposed method is extended from seam
carving. Instead of using gradient based to compute the energy map, they
propose a method for creating a deep energy map using an encoder-decoder
convolution neural network. To improve retargeting results in both saliency
detection and retargeting, Ahmadi et al. [27] presented a new approach in
which the use of image context and semantic segmentation are examined.

In contrast, we propose a neural network scheme to define the importance map
in both image and video frames. This map is then examined by any retargeting
operator. Our experimental results and the evaluation presented in this paper
show our approach is effective in improving media retargeting results.

Figure 3: The pipeline of our proposed retargeting system. The black arrow
denotes the navigation procedure of the previous system. The red arrow is the
navigation procedure of our media retargeting system.

##  3 Methodology

The overall pipeline of our proposed media retargeting system is presented in
Fig.3. Given an image or video, output of our system is an image/video which
is retargeted to a new display resolution size. The proposed system consists
of two main stages. The first one, which is mentioned as a black box, is to
calculate the visual information in image/video through the proposed network.
Meanwhile, the remain stage is to produce retargeted image/video by the
retargeting operations. In the following, the specification of each process is
described.

###  3.1 Visual information estimation via the proposed network

The first fundamental step of a content-aware retargeting pipeline is the
estimation of importance of pixels in images/video frames [10]. Therefore,
analyzing the visual information to define the important pixels in
images/videos is a prerequisite in our proposed system. As mentioned in
related work section, the patch-based approach [1, 6] utilizes a context-aware
saliency method [23] to specify either nonhomogeneous regions or homogeneous
regions, and seam carving technique [2] adopts the gradient-based calculation
to define the removed/inserted seam. Yet, due to the diversity of media in the
real world, the aforementioned techniques, which are utilized to produce the
importance of pixels, do not perform well. This phenomenon results in
retargeted media are not plausible. Thus, preserving the shape of important
objects in the retargeted media is a necessary manner to be investigated.
Motivated by this, we propose a network to estimate pixel information with the
attempt to boost the retargeting media better. The proposed network consists
of two phases in the terms of feature extraction and layer-fusion. Feature
extraction is built upon a pre-trained network to extract features in an
image/video frames. Later, a computation in terms of layer-fusion is adopted
to estimate the important pixels in the given input.

Feature extraction
To extract features in images/video frames, the pre-trained VGG-16 network
[28] is adopted. This well-known network is trained with ImageNet dataset [29]
and has excellent performance in image classification, object detection, etc.
Besides, there are many of deep learning-based studies utilized VGG-16 network
to extract features in their applications. Thus, this pre-trained network is
suitable to adequately extract feature for our feature extraction manner.
VGG-16 network consists of 13 convolution layers, 5 max-pooling layers, and 3
fully-connected layers follow a stack of convolution layers, and a soft-max
layer goes at the end [28]. The fully-connected and soft-max layers are
typically used in image classification and object detection applications. In
our network, since we take advantage of VGG to extract image/video frame
features, we remove either fully-connected or soft-max layers and build our
network upon the convolution and pooling layers. These layers form five blocks
as we demonstrate in Fig.4. Layers in such a VGG network is applied by the
different number of filters $(K)$. Feature maps at each layer capture
different information from the input. As shown in Fig.4, except the first one,
each block composes of a max-pooling layer and convolution layers. We assume
that the input image size is $W$x$H$x$c$, where $W$, $H$, $c$ is the width,
height, and the number of channels, respectively. Due to the utilization of
max-pooling layers, feature maps at the lower layers are defined as
$(W/2^{i}$x$H/2^{i}$x$c),i=0\dots 3$.

Layer-fusion
To analyze the visual information of important pixels, we first propose to
fuse the feature maps from the layers in our network. Then a tensor obtained
from the first manner is used to predict visual information of the input
image. Visual attention map for media retargeting application needs to present
the contrast color values on both the edge of important objects and the
regions inside such these objects. Encoding directly from a single block may
not guarantee to obtain the expected feature. Thus, we introduce a strategy in
terms layer-fusion. The fused layers are extracted from the last convolution
layer in the $3^{th},4^{th},$ and $5^{th}$ block. As the demonstration in
Fig.4, these layers are named $conv3\\_3,conv4\\_3,$ and $conv5\\_3$,
respectively. In the following, we call these maps as, respectively,
$F_{3},F_{4}$, and $F_{5}$.

We solve the fusion through a concatenate operator. Since each pair of feature
maps, which are concatenated, are required to share the same spatial size, the
following describes our procedure to archive this requirement. Given two
adjacent feature maps $F_{i}$, $F_{j}$ $(i<j)$, the product of the two
$(F_{c})$ is defined as:

$F_{c}=f_{c}(\hat{F_{i}},\hat{F_{j}})$ (1)

, where $f_{c}(.)$ is the concatenate operator,

$\hat{F_{i}}=f_{2}(F_{i})$ (2)

, where $f_{2}(.)$ is the conv 3x3 - 64 filters,

$\hat{F_{j}}=f_{3}(F_{j})$ (3)

, where $f_{3}(.)$ is the UpSampling 2x2 window. Once the product of a
concatenation is obtained, it is fed into a conv 3x3 - 1 filter. Those of such
products form a tensor $(F_{T})$, which is expressed as:

$F_{T}=f_{1}(F_{c2})\otimes f_{1}(F_{c1})\otimes f_{1}(F_{5})$ (4)

, where $f_{1}(.)$ is the conv 3x3-1 filter, $F_{c2}$ is the product generated
by (1) with the input layers are $F_{3}$ and $F_{4}$, $F_{c1}$ is the product
generated by (1) with the input layers are $F_{4}$, and $F_{5}$. The detail of
this procedure is illustrated in Fig.5.

Figure 4: The proposed network. Figure 5: The illustration of the fusion
process.

Finally, a 1x1 convolution [30] is adopted for mapping the feature maps into a
ground truth through an activation function. The ground truth we imply in our
network is the masked image corresponding a color image. That is, the
important objects in an image are masked in white, whereas the background is
in black. In this final layer, we use a sigmoid activation function to
calculate the probability as an output that has value in the range of 0 and 1.
In the training process, all the parameters in our network are learned by
minimizing the loss function, which is computed by the errors between the
probability map and ground truth. Given a ground truth
$S_{g}(S_{g}\in{0,1}^{hxw})$ which is corresponding to the input image I
(HxWxc), stochastic gradient descent (SGD) is employed to minimize the loss of
training to predict visual information probability:

$L(S_{g},S_{p})=-y_{i}*log(\hat{y_{i}})+(1-y_{i})*log(1-\hat{y_{i}})$ (5)

, where $y_{i}\in S_{g}$ and $\hat{y_{i}}\in S_{p}$.

The effectiveness of the importance map produced by the proposed network in
retargeting results is shown in Fig.6. Information of image is visualized by
the corresponding heat map. If the important pixels are defined insufficiently
(Fig.6-b), retargeted image is shrunk (Fig.6a-1). Meanwhile, with a correct
importance map (Fig.6-c), better preservation of the shape of important
objects in the retargeted image is generated (Fig.6a-2). More experimental
results in retargeting image and video are shown and discussed in later
session.

###  3.2 Media retargeting

Once the importance map is produced, the second step of our retargeting
pipeline is employing retargeting operation based on the generated importance
map. In this paper, we examine two approaches to produce media retargeting
results. One of them represents for the continuous methods. We use the
modified version of the study in [1] for image resizing and [6] for video
retargeting. The other one, which has been mentioned as a typical technique in
discrete category, is seam carving technique in [2].

Image retargeting with seam carving method
To drive an image to a target size, Avidan and Shamir [2] relied on an energy
map to define a vertical/horizontal seam which is removed/inserted from/into
the input image. We describe this technique in reducing image size as the
below pseudocode:

Algorithm 1 Algorithm of seam carving

Require: An RGB image $I_{o},Irs$.

1:for $i=0$ to $Irs$ do

2: $E_{i}=E(I)$;

3: $s_{i}$ $\xleftarrow[]{}$ the minimum seam;

4: $i^{++}$;

5: $I_{i}$ $\xleftarrow[]{}$ Remove$(E_{i},s_{i})$;

6:end for

The iteration ($Irs$) is defined as:

$Irs=\begin{cases}|h-h^{\prime}|,&\text{if horizontal resizing}\\\
|w-w^{\prime}|,&\text{if vertical resizing}\end{cases}$ (6)

, where $(h,w)$ is the original size, $(h^{\prime},w^{\prime})$ is the target
size.
The energy map mentioned in [2] is the so-called gradient map. It is typically
calculated based on the equation:

$E(I)=\left|\frac{\partial}{\partial x}I\right|+\left|\frac{\partial}{\partial
y}I\right|$ (7)

, where $I$ is the given input image. Although the gradient energy function
has good efficiency for many images it may cause serious distortion due to the
sensitivity to noise and allocation of higher energy to edge pixels.
Therefore, this map itself is not sufficient to avoid significant distortion
occurs in the seam carving method [2].

Contrast to the approach in Avidan and Shamir [2], we adopt the proposed
network to produce the energy map. The energy map at each iteration can be
expressed as:

$E_{i}=g(I_{i})$ (8)

, where $g(.)$ denotes the model of the proposed network, $I_{i}$ is the
retargeted image at iteration $i^{th}$, $E_{i}$ is the energy map of $I_{i}$,
$i\in[1\dots Irs]$. At each iteration, based on the generated energy map,
seams are identified. A seam, which has the minimum energy, is defined and
removed from $I_{i}$. At each iteration, once the energy map is produced
through our model, seam carving [2] is borrowed for seam identification and
seam removal to produce final retargeting image.

Fig.7 illustrates the difference between our importance map and the gradient
map in seam removal. Since the important regions of image are not presented
with high energy in the gradient map, the removed seams cut through the
objects (as in Fig.7b). Meanwhile, the importance map produced from our
network controls efficiently to avoid distortion when removing seams.
Therefore, the retargeted result is significantly improved in our result (g)
compared to Avidan and Shamir [2]’s result (c).

Figure 6: Efficiency of importance map generated from our proposed network in
retargeting image. (a) Input image, (b) heat map of the saliency map obtained
in [22], (c) heat map of the importance map obtained in our network. (a-1)
image is retargeted with the importance map (b), (a-2) image is retargeted
with the importance map (c). Figure 7: Illustrates the difference between our
importance map and the gradient map in seam removal. (a) gradient map, (b) the
three seams are removed, (c) retargeted result with the energy map (a). (d)
visualization of pixel energy in the red rectangle in (a). The dark brown
represents for high energy pixel. From left to right: energy map, the removed
seams (red strokes), after removing seams. (e) our importance map, (f) the
three seams are removed, (g) retargeted result with the energy map (e).

Image and video retargeting with patch-based approach

The patch-based approach for image and video retargeting is introduced by Lin
et al. [1] and Lin et al. [6], respectively. This technique firstly take
advantage of a saliency detection method [22] to indicate the important region
in image/video frames. For short, we use the terms input to imply input image
and video frames in the following. The input is then segmented [31, 32] into
patches. The average of saliency value of each patch is computed. These values
are then used in the optimization equations which are raised to preserve the
shape of objects and the line structure after image/video is resized. This
approach typically has a good performance in many images and videos which
contain various attributes in the content. However, in the cases of the visual
values of distinct regions that are similar or the content of the input is
made up of sizable objects, the mentioned saliency detection method [22] fails
in assisting the patch-based method [1, 6] to generate the good retargeting
results. In the following, we present how we tackle this limitation through
our proposed network.

The given color input is segmented into $n$ patches
$P=\\{p_{1},\dots,p_{n}\\}$. Each patch is assigned an energy value
$(\omega)$, which is defined as:

$\omega_{k}=\frac{1}{m}\sum_{j=0}^{m}s_{j}$ (9)

, where $m$ denotes the total number of pixels in patch $p_{k}$, $s_{j}$ is
the visual information value at a pixel in patch $p_{k}$; $k\in[1\dots n]$.

In Lin et al. [1, 6], the energy values of patches, which is defined in Eq(9),
are used in warping. A high energy value is assigned to nonhomogeneous
regions, whereas a low energy value is assigned to homogeneous regions. The
saliency detection method [22] is used to detect the object regions,
especially the boundary of regions. However, the inner region, which locates
inside the important object, is not presented sufficiently to be recognized as
an important region.

Figure 8: (a-1) the heat map of visual information produced by [22]; (a-2) the
heat map of our importance map; (b), (b-1), (b-2) is a the visualization of a
quad in the green region in (a-1) and (a-2). And, the blue circle represents
for a vertex of the quad. (b) coordinate of quad vertex in original image.
(b-1) predicted coordinate of quad vertex in retargeted result using (a-1).
(b-2) predicted coordinate of quad vertex in retargeted result using (a-2).

To tackle this problem, we utilize our proposed network to generate an
importance map which is then used to calculate the energy value on each patch
through Eq(9) for both images and video frames. In Fig.8, the heat map of the
saliency map [22] (shown in (a-1)) and our importance map (shown in (a-2)) of
the input image in Fig.6-a are presented. We can see in the same region, which
is highlighted by the green square, it is recognized as unimportant region in
a-1 but it has a high energy in (a-2). Thus, when Lin et al. [1, 6] employ the
saliency map [22] in their retargeting system, vertices in the quad, which
belong to such regions, are scaled uniformly (as shown in (c-1)). This
phenomenon results in the important objects are shrunk after resizing (as in
Fig.6a-1). With the utilization of our network, the visual information of
considerable objects is sufficiently defined to describe the property of such
regions. Thus, vertices in the quads in these regions are deformed ununiformly
in the consistency with the significance of the corresponding region (as shown
in (c-2)). This aspect boosts the shape of the important objects that can be
retained close to the ratio of the original ones (as shown in Fig.6a-2).

Moreover, in terms of preserving the shapes of high-significance patches, Lin
et al. [1, 6] deal with this issue by defining the total patch transformation
energy with a weighting factor assigned to each energy term.

$\begin{split}D_{TF}(P)&=\sum_{k=1}^{np}(\alpha\times D_{ST}(patch_{k})\\\
&+(1-\alpha)\times D_{LT}(patch_{k}))\end{split}$ (10)

$\begin{split}D_{Sp}(M)&=(\alpha\times D_{SimT}(M)+(1-\alpha)\times
D_{LinT}(M))\\\ &+D_{Ori}(M)\end{split}$ (11)

The equation (10) and equation (11) is called the total patch transformation
energy in [1] and [6], respectively. In these equations, a large value of the
weighting factor $\alpha$ is assigned $(\alpha\in\\{0.7,0.8\\})$. In this
manner, the shape of the important objects can be preserved but they are still
shrunk. Assuming $\omega_{k}^{\prime}$ is the energy of patch $k$, which is
produced by: $\omega_{k}^{\prime}=\alpha\times\omega$. $\omega_{k}^{\prime}$
grows linearly with the energy of other patches in the same factor. Thus,
there is a lack of correlation between patches. In our current system, since
the importance map generated by our network is sufficient to describe the
visual information of the input, we propose to modify the total patch
transformation energy in [1, 6]. That is, we eliminate the weighting factor
$\alpha$ in our modified versions. We define as followed:

$E(P)=\sum_{k=1}^{n}(e_{t}(p_{k})+e_{s}(p_{k}))$ (12)

, where $e_{t},e_{s}$ is the energy term of rigid transformation and linear
scaling [1] of patch $p_{k}$, respectively; $k\in[1\dots n]$, $n$ is the total
number of patches in a segmented input.

$E(M)=e_{lim}(p_{k})+e_{lin}(p_{k})+e_{ori}(p_{k})$ (13)

, where $e_{lim},e_{lin},e_{ori}$ is the energy term of rigid transformation,
linear scaling, and grid orientation [6], respectively, in patch $p_{k}$;
$k\in[1\dots n]$; $n$ is the total number of patches in segmented frame. And,
all of the energy terms in Eq(12) and Eq(13) are adopted from our importance
map.

Figure 9: (a) Original image, Lin et al. [1]’s result without weighting factor
$\alpha$ in Eq(10) (b) and with Eq(10) (c), our result with Eq(12) (d). In
this example, the original image is reduced 50% of original width.

We describe the capability of Eq(12) and Eq(13) in preserving the shape of
important objects in comparison with Eq(10) and Eq(11) in Fig.9. We can see in
this figure if the authors in [1, 6] do not employ a weighting factor to
increase the energy of significant patches the shape of the important object
is stretched seriously (shown in (b)). When assigning the weighting factor to
each energy term (as in Eq(10)), the shape of object is just slightly improved
(shown in (c)). In contrast, with our importance map and Eq(12), our result
(shown in (d)) has a better performance in terms of preserving the shape and
the ratio of the important object.

After employing our importance map to calculate the energy of each patch in
image/frames, the studies [1, 6] are borrowed to produce retargeted results.
We regard to note that the modified versions of patch transformation energy,
which are presented in Eq(12) and Eq(13), are used in this process instead of
Eq(10) and Eq(11). Figures 10 and 11 show the effectiveness of our media
retargeting system in handling the limitation in both seam carving and patch-
based techniques through our better results. The other results, we present in
the next section.

Figure 10: (a) original images; retargeting image (a-1) with seam carving
operator by gradient map (b-1) and our importance map (c-1); retargeting image
(a-2) with patch-based method by saliency map (b-2) and our importance map
(c-2). Figure 11: Each pair of frames, which are extract from different frames
in the same video, is compared in each column. The first row is retargeted
frames obtained from Lin et al. [6]. The second row is the retargeted frames
generated by our system.

##  4 Experimental Results

Figure 12: The evaluation on the ARS score.

We implemented our system on the PC with Intel Core i7 CPU, 16GB RAM, and
Nvidia GTX1070 GPU. With an input image/frame in the size of 1024x768, our
proposed model takes less than one second to generate an importance map.
Meanwhile, the algorithm in [22] takes about 45 seconds to produce a saliency
map. In our overall system, we implement the proposed network with Python
version 3.6, the segmentation process and the retargeting operators are
computed with C++ programming language in Visual Studio 2015. In our proposed
network, we use MSRA 10K dataset [33], which is used for saliency detection,
as our training data. This dataset consists of 10000 images with the diversity
of the content structure of natural scenes. The dataset also contains manually
annotated ground-truth saliency.

###  4.1 Objective evaluation

To quantitate the quality of our retargeting results, we adopt the Aspect
Ratio Similarity (ARS) score. This measurement is introduced in [34] and has
been mentioned as an effective method to estimate the geometric transition for
the relationship images [8, 10]. Thus, this assessment is suitable to
adequately measure the aspect ratio changes between the original and
retargeted images.

In this computation, RetargetMe dataset [35] is used. There are a total of 37
sets of images with different attributes in image content. Each set consists
of an original image and the corresponding retargeted images which are
retargeted by eight retargeting methods [35]. In our objective evaluation, we
conduct this measurement on our image retargeting results in comparison with
four typical retargeting methods’ results. They are Multi-operators (MOP) [4],
Seam carving (SC) [2], Shift-map (SM) [5], and Warping [1]. All of the results
in this dataset are published by the authors, thus they are trustful enough to
use and for a fair comparison. The ARS score of each method is normalized in
the range from 0 to 1. That is, a method that has a higher ARS score implies
the retargeted result generated from that method is better.

###  4.2 Discussion

In terms of quantitating our results in comparison with the typical
retargeting methods on the benchmark dataset RetargetMe, we show the objective
evaluation in Fig.12. Based on the ARS scores in this figure, most of our
retargeting results have a good quality and better than the compared results.
There are 8 cases (21%), the scores are not the highest in the set, but they
are still in an acceptable level (greater than 0.8) and higher than Lin et al.
[1]’s results.

Figure 13: Images are retargeted by seam carving operator. From left to right:
original images, results in [2], our results.

Besides the RetargetMe dataset, we conduct more experiments on other data
(NRID [36] and DUTS-TR dataset111http://saliencydetection.net/duts). We show
respectively our retargeted images and videos in comparisons with the studies
Avidan and Shamir [2] in Fig.13, Lin et al. [1] in Fig.14, and Lin et al. [6]
in Fig.15. In term of seam carving operator, we examine on the original images
which contain different attributes in their content, as shown in Fig.13. The
results in [2] are distorted seriously with the utilization of the gradient
map. Whereas, our results preserve effectively most of the important regions
in the original images. For the warping based method, Figures 14 and 15
present our results in the comparisons with [1] and [6], respectively. As
shown in these two figures, the retargeted images and frames produced in [1,
6] are shrunk. However, our importance map boosts these results quite better.

Figure 14: Images are retargeted by patch-based method. From left to right:
original images, results in [1], our results. Figure 15: Examples of
retargeted videos. (a) Original frame, (b) Lin et al. [6]’s result, (c) our
result.

Apart from the above, we also compare our results with those that are
generated using deep learning. Fig.16 demonstrates the comparison of our
results on seam carving operator with Song et al. [8]. As shown in this
figure, with the same operator (seam carving), the serious distortion occurs
with the utilization of the gradient map in [2] and deep energy map in [8]. In
contrast, our results better preserve the content of the important regions
including texture, edge, line, etc. Fig.17 shows our results in the comparison
with Tan et al. [9]. The retargeted images, which are generated through the
network in [9], are significantly distorted. Meanwhile, ours have fully better
performance. Fig.18 is presented to compare our results with retargeted images
in [27]. The authors in [27] uses convolutional neural networks to generate
saliency map for image retargeting. From these results, their retargeted
images are distorted but ours are successful. The reason they fail in these
examples is that, as stated in [27], the method they use assigns equal scaling
factors to all the pixels in columns. Thus, by sudden changes in the scaling
factor from a column to the next column, the straight line is deflected (as in
the the car image). And, this reason also causes distortion in the less
salient columns (as in the right ear of the child and the player’s feet).
Another deep learning-based study we compare in this sector is the method
proposed in [37]. This research proposes a network to drive an image to a
square size. This comparison is depicted in Fig.19. The results in [37] are
not distorted but they are cropped on the left side of images. This phenomenon
leads to the squared photos fail in preserving image content in the cases that
the important content of the image are not located in the center area.
However, we handle well in these examples even they are retargeted to 50% of
the width.

Figure 16: From left to right: original image, Avidan and Shamir [2]’s
results, Song et al. [8]’s results, our results with seam carving operator.
Figure 17: From left to right: original images, Tan et al. [9]’s results, our
results. Figure 18: From left to right: original image, Ahmadi et al. [27]’s
result, our result. Figure 19: From left to right: original images, Song et
al. [37]’s results with 25% width reduction, our results with 25% width
reduction, our results with 50% width reduction.

To demonstrate more about the capability of our importance map in the
retargeting applications, we examine to enlarge images by our system (shown in
Fig.20). Moreover, Tang et al. [38] showed that the images with low
retargetability scores are reliable for assessing the new retargeting method.
Such images contain attributes like text, symmetry, texture, multiple objects.
Motivated by this, we conduct the experiment on the images which are made of
these attributes. Fig.23 shows the plausible results are generated by our
system in this manner. This implies that we not only boost the retargeting
result better but also handle well on the difficult-to-resize images. More
experimental results including videos are supplied in our supplementary
document which can be explored at our website:
http://graphics.csie.ncku.edu.tw/deep_saliency.

In the aforementioned part, we mostly compare our work with the related works
of different classes in retargeting. The question here is what if the proposed
importance map is substituted by another deep learning-based saliency map. We
argue this issue through the results in Fig.21. And, we examine the saliency
map obtained from [39] in this experiment. The BASNet, which is proposed in
[39], has a good performance in salient object detection and has been used in
many image processing applications. However, when it is employed in
retargeting application, it may not be adequate (as the results in images
c-1). The reason is that an importance map for retargeting application is
required to represent the energy for all the pixels in an image. The magnitude
of pixel energy describes how significant a pixel is. A good salient object
detection method focuses on efficiently detecting the recognized objects (see
images b-1). Thus, the pixels in other regions are skipped. This phenomenon
results in the retargeted images distorted with seam carving operator or
inconsistent deformation with warping method.

Figure 20: Enlarging images to 25% of the width by our system. In each pair of
images, the right image is enlarged from the left one. Figure 21: (a) original
image, (b-1) saliency map in [39], (c-1) retargeted image with seam carving
operator when using (b-1) as the energy map, (b-2) our importance map, (c-2)
retargeted image with seam carving operator when using (b-2) as the energy
map.

We finally show our examples of failure in Fig.22. Due to the limitation in
the dataset we use to train, the importance map obtained from our network is
not good enough to represent the information on the input. Consequently, it
may not perform well with seam carving operator. As shown in Fig.22, the
results are not good with the utilization of seam carving method. Yet, they
are still plausible in our system with warping operator [1]. This problem may
be also due to the original limitation of seam carving method. Our proposed
network can reduce the possibility of failure cases but can not solve its
original limitation, i.e., seam carving may include partials of important
areas in the carved seams.

Figure 22: Our examples of failure with seam carving operator. (a) Original
image, (b) Avidan and Shamir [2]’s result, (c) Our result with seam carving
operator, (d) our result with warping method. The examples in this figure are
retargeted to 50% of the width. Figure 23: More results on different image
attributes in our system. In each pair of images, the left image is original
image of the retargeted result on the left.

##  5 Conclusion

In this paper, we have presented a network to produce an importance map, which
is used in the image/video retargeting system to enhance the quality of the
image/video after retargeted. Once the importance map is obtained from the
proposed network, a retargeting operator can be adopted to generate retargeted
image/video. Two methods are examined in this study including seam carving,
modified versions of warping for image and video retargeting. We demonstrate
the effectiveness of our framework by an objective evaluation and several
ideal retargeted results. Experimental results prove that the proposed scheme
performs well in comparisons with those of other works. In future work, we
hope to improve the dataset to alleviate the limitation in this study.
Besides, we plan to investigate new image retargeting and video retargeting
methods using a deep learning-based approach.

## References

  * Lin et al. [2012] Shih-Syun Lin, I-Cheng Yeh, Chao-Hung Lin, and Tong-Yee Lee.  Patch-based image warping for content-aware retargeting.  _IEEE transactions on multimedia_ , 15(2):359–368, 2012.
  * Avidan and Shamir [2007] Shai Avidan and Ariel Shamir.  Seam carving for content-aware image resizing.  In _ACM SIGGRAPH 2007 papers_ , pages 10–es. 2007.
  * Jin et al. [2010] Yong Jin, Ligang Liu, and Qingbiao Wu.  Nonhomogeneous scaling optimization for realtime image resizing.  _The Visual Computer_ , 26(6-8):769–778, 2010\.
  * Rubinstein et al. [2009] Michael Rubinstein, Ariel Shamir, and Shai Avidan.  Multi-operator media retargeting.  _ACM Transactions on graphics (TOG)_ , 28(3):1–11, 2009.
  * Pritch et al. [2009] Yael Pritch, Eitam Kav-Venaki, and Shmuel Peleg.  Shift-map image editing.  In _2009 IEEE 12th International Conference on Computer Vision_ , pages 151–158. IEEE, 2009.
  * Lin et al. [2013a] Shih-Syun Lin, Chao-Hung Lin, I-Cheng Yeh, Shu-Huai Chang, Chih-Kuo Yeh, and Tong-Yee Lee.  Content-aware video retargeting using object-preserving warping.  _IEEE transactions on visualization and computer graphics_ , 19(10):1677–1686, 2013a.
  * Cho et al. [2017] Donghyeon Cho, Jinsun Park, Tae-Hyun Oh, Yu-Wing Tai, and In So Kweon.  Weakly-and self-supervised learning for content-aware deep image retargeting.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4558–4567, 2017.
  * Song et al. [2018a] Eungyeol Song, Minkyu Lee, and Sangyoun Lee.  Carvingnet: content-guided seam carving using deep convolution neural network.  _IEEE Access_ , 7:284–292, 2018a.
  * Tan et al. [2019] Weimin Tan, Bo Yan, Chuming Lin, and Xuejing Niu.  Cycle-ir: Deep cyclic image retargeting.  _IEEE Transactions on Multimedia_ , 2019.
  * Kiess et al. [2018] Johannes Kiess, Stephan Kopf, Benjamin Guthier, and Wolfgang Effelsberg.  A survey on content-aware image and video retargeting.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 14(3):1–28, 2018.
  * Suh et al. [2003] Bongwon Suh, Haibin Ling, Benjamin B Bederson, and David W Jacobs.  Automatic thumbnail cropping and its effectiveness.  In _Proceedings of the 16th annual ACM symposium on User interface software and technology_ , pages 95–104, 2003.
  * Kopf et al. [2009] Stephan Kopf, Benjamin Guthier, Hendrik Lemelson, and Wolfgang Effelsberg.  Adaptation of web pages and images for mobile applications.  In _Multimedia on Mobile Devices 2009_ , volume 7256, page 72560C. International Society for Optics and Photonics, 2009.
  * Itti et al. [1998] Laurent Itti, Christof Koch, and Ernst Niebur.  A model of saliency-based visual attention for rapid scene analysis.  _IEEE Transactions on pattern analysis and machine intelligence_ , 20(11):1254–1259, 1998.
  * Rubinstein et al. [2008] Michael Rubinstein, Ariel Shamir, and Shai Avidan.  Improved seam carving for video retargeting.  _ACM transactions on graphics (TOG)_ , 27(3):1–9, 2008.
  * Han et al. [2010] Dongfeng Han, Milan Sonka, John Bayouth, and Xiaodong Wu.  Optimal multiple-seams search for image resizing with smoothness and shape prior.  _The Visual Computer_ , 26(6-8):749–759, 2010\.
  * Ito [2016] Izumi Ito.  Gradient-based global features for seam carving.  _EURASIP Journal on Image and Video Processing_ , 2016(1):1–9, 2016.
  * Yin et al. [2015] Ting Yin, Gaobo Yang, Leida Li, Dengyong Zhang, and Xingming Sun.  Detecting seam carving based image resizing using local binary patterns.  _computers & security_, 55:130–141, 2015.
  * Liu and Gleicher [2005] Feng Liu and Michael Gleicher.  Automatic image retargeting with fisheye-view warping.  In _Proceedings of the 18th annual ACM symposium on User interface software and technology_ , pages 153–162, 2005.
  * Ma and Zhang [2003] Yu-Fei Ma and Hong-Jiang Zhang.  Contrast-based image attention analysis by using fuzzy growing.  In _Proceedings of the eleventh ACM international conference on Multimedia_ , pages 374–381, 2003.
  * Zhang et al. [2009] Guo-Xin Zhang, Ming-Ming Cheng, Shi-Min Hu, and Ralph R Martin.  A shape-preserving approach to image resizing.  In _Computer Graphics Forum_ , volume 28, pages 1897–1906. Wiley Online Library, 2009.
  * Guo et al. [2009] Yanwen Guo, Feng Liu, Jian Shi, Zhi-Hua Zhou, and Michael Gleicher.  Image retargeting using mesh parametrization.  _IEEE Transactions on Multimedia_ , 11(5):856–867, 2009.
  * Goferman et al. [2011] Stas Goferman, Lihi Zelnik-Manor, and Ayellet Tal.  Context-aware saliency detection.  _IEEE transactions on pattern analysis and machine intelligence_ , 34(10):1915–1926, 2011.
  * Wolf et al. [2007] Lior Wolf, Moshe Guttmann, and Daniel Cohen-Or.  Non-homogeneous content-driven video-retargeting.  In _2007 IEEE 11th International Conference on Computer Vision_ , pages 1–6. IEEE, 2007.
  * Achanta and Süsstrunk [2009] Radhakrishna Achanta and Sabine Süsstrunk.  Saliency detection for content-aware image resizing.  In _2009 16th IEEE international conference on image processing (ICIP)_ , pages 1005–1008. IEEE, 2009.
  * Kim et al. [2011] Wonjun Kim, Chanho Jung, and Changick Kim.  Spatiotemporal saliency detection and its applications in static and dynamic scenes.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 21(4):446–456, 2011.
  * Wang et al. [2014] Weiwei Wang, Dong Zhai, Tao Li, and Xiangchu Feng.  Salient edge and region aware image retargeting.  _Signal Processing: Image Communication_ , 29(10):1223–1231, 2014.
  * Ahmadi et al. [2019] Mahdi Ahmadi, Nader Karimi, and Shadrokh Samavi.  Context-aware saliency detection for image retargeting using convolutional neural networks.  _arXiv preprint arXiv:1910.08071_ , 2019.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Lin et al. [2013b] Min Lin, Qiang Chen, and Shuicheng Yan.  Network in network.  _arXiv preprint arXiv:1312.4400_ , 2013b.
  * Felzenszwalb and Huttenlocher [2004] Pedro F Felzenszwalb and Daniel P Huttenlocher.  Efficient graph-based image segmentation.  _International journal of computer vision_ , 59(2):167–181, 2004.
  * Grundmann et al. [2010] Matthias Grundmann, Vivek Kwatra, Mei Han, and Irfan Essa.  Efficient hierarchical graph-based video segmentation.  In _2010 ieee computer society conference on computer vision and pattern recognition_ , pages 2141–2148. IEEE, 2010.
  * Liu et al. [2010] Tie Liu, Zejian Yuan, Jian Sun, Jingdong Wang, Nanning Zheng, Xiaoou Tang, and Heung-Yeung Shum.  Learning to detect a salient object.  _IEEE Transactions on Pattern analysis and machine intelligence_ , 33(2):353–367, 2010.
  * Zhang et al. [2016] Yabin Zhang, Weisi Lin, Xinfeng Zhang, Yuming Fang, and Leida Li.  Aspect ratio similarity (ars) for image retargeting quality assessment.  In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1080–1084. IEEE, 2016.
  * Rubinstein et al. [2010] Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir.  A comparative study of image retargeting.  In _ACM SIGGRAPH Asia 2010 papers_ , pages 1–10. 2010.
  * Hsu et al. [2014] Chih-Chung Hsu, Chia-Wen Lin, Yuming Fang, and Weisi Lin.  Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss.  _IEEE Journal of Selected Topics in Signal Processing_ , 8(3):377–389, 2014.
  * Song et al. [2018b] Yu Song, Fan Tang, Weiming Dong, Xiaopeng Zhang, Oliver Deussen, and Tong-Yee Lee.  Photo squarization by deep multi-operator retargeting.  In _Proceedings of the 26th ACM international conference on Multimedia_ , pages 1047–1055, 2018b.
  * Tang et al. [2019] Fan Tang, Weiming Dong, Yiping Meng, Chongyang Ma, Fuzhang Wu, Xinrui Li, and Tong-Yee Lee.  Image retargetability.  _IEEE Transactions on Multimedia_ , 22(3):641–654, 2019.
  * Qin et al. [2019] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand.  Basnet: Boundary-aware salient object detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7479–7489, 2019.