Further author information: (Send correspondence to J.K.K)
J.K.K.: E-mail<EMAIL_ADDRESS>

# MagAO-X Phase II Upgrades: Implementation and First On-Sky Results of a New
Post-AO 1000 Actuator Deformable Mirror

Jay K. Kueny  Steward Observatory, University of Arizona, Tucson, 933 N Cherry
Ave, Tucson, AZ 85721, USA  James C. Wyant College of Optical Sciences,
University of Arizona, 1630 E. University Blvd., Tucson, AZ 85721, USA
National Science Foundation Graduate Research Fellow  Kyle Van Gorkom  Steward
Observatory, University of Arizona, Tucson, 933 N Cherry Ave, Tucson, AZ
85721, USA  Maggie Kautz  Steward Observatory, University of Arizona, Tucson,
933 N Cherry Ave, Tucson, AZ 85721, USA  James C. Wyant College of Optical
Sciences, University of Arizona, 1630 E. University Blvd., Tucson, AZ 85721,
USA  National Science Foundation Graduate Research Fellow  Sebastiaan Haffert
Steward Observatory, University of Arizona, Tucson, 933 N Cherry Ave, Tucson,
AZ 85721, USA  Subaru Telescope, National Observatory of Japan, Hilo, HI
Jared R. Males  Steward Observatory, University of Arizona, Tucson, 933 N
Cherry Ave, Tucson, AZ 85721, USA  Alex Hedglen  Northrop Grumman in Rolling
Meadows, IL  Laird Close  Steward Observatory, University of Arizona, Tucson,
933 N Cherry Ave, Tucson, AZ 85721, USA  Eden McEwen  Steward Observatory,
University of Arizona, Tucson, 933 N Cherry Ave, Tucson, AZ 85721, USA  James
C. Wyant College of Optical Sciences, University of Arizona, 1630 E.
University Blvd., Tucson, AZ 85721, USA  National Science Foundation Graduate
Research Fellow  Jialin Li  Steward Observatory, University of Arizona,
Tucson, 933 N Cherry Ave, Tucson, AZ 85721, USA  National Science Foundation
Graduate Research Fellow  Joseph D. Long  Center for Computational
Astrophysics, Flatiron Institute, NY  Warren Foster  Steward Observatory,
University of Arizona, Tucson, 933 N Cherry Ave, Tucson, AZ 85721, USA  Logan
Pearce  Steward Observatory, University of Arizona, Tucson, 933 N Cherry Ave,
Tucson, AZ 85721, USA  Avalon McLeod  Draper Laboratory in Cambridge, MA  Jhen
Lumbres  Northrop Grumman in Pasadena, CA  Lauren Schatz  Starfire Optical
Range, Kirtland Air Force Base in Albuquerque, NM  Olivier Guyon  Subaru
Telescope, National Observatory of Japan, Hilo, HI  Joshua Liberman  Steward
Observatory, University of Arizona, Tucson, 933 N Cherry Ave, Tucson, AZ
85721, USA  James C. Wyant College of Optical Sciences, University of Arizona,
1630 E. University Blvd., Tucson, AZ 85721, USA

###### Abstract

MagAO-X is the extreme coronagraphic adaptive optics (AO) instrument for the
6.5-meter Magellan Clay telescope and is currently undergoing a comprehensive
batch of upgrades. One innovation that the instrument features is a deformable
mirror (DM) dedicated for non-common path aberration correction (NCPC) within
the coronagraph arm. We recently upgraded the 97 actuator NCPC DM with a 1000
actuator Boston Micromachines Kilo-DM which serves to (1) correct non-common
path aberrations which hamper performance at small inner-working angles, (2)
facilitate focal-plane wavefront control algorithms (e.g., electric field
conjugation) and (3) enable 10 kHz correction speeds (up from 2 kHz) to assist
post-AO, real-time low-order wavefront control. We present details on the
characterization and installation of this new DM on MagAO-X as part of our
efforts to improve deep contrast performance for imaging circumstellar objects
in reflected light. Pre-installation procedures included use of a Twyman-Green
interferometer to build an interaction matrix for commanding the DM surface,
in closed-loop, to a flat state for seamless integration into the instrument.
With this new NCPC DM now installed, we report on-sky results from the MagAO-X
observing run in March—May 2024 for the Focus Diversity Phase Retrieval and
implicit Electric Field Conjugation algorithms for quasistatic speckle removal
and in-situ Strehl ratio optimization, respectively.

###### keywords:

adaptive optics, deformable mirrors, high-contrast imaging

##  1 INTRODUCTION

MagAO-X [1, 2] is the “extreme” coronagraphic adaptive optics (AO) instrument
for the 6.5-meter Magellan Clay telescope (Figure 1) which is optimized to
image at visible wavelengths. The instrument includes an array of deformable
mirrors (DM) including: a high-stroke 97 actuator woofer ALPAO DM97, a 2040
actuator tweeter DM (Boston Micromachines; BMC), and an additional DM for non-
common path correction (NCPC) in the coronagraph.

Figure 1:  A rendering of the 6.5 m Magellan-Clay telescope with MagAO-X
installed on the Nasmyth platform highlighted and enlarged.

The woofer-tweeter DM architecture[3] is driven by a $\sim 3$ kHz transmissive
pyramid wavefront sensor. The $50\times 50$ 2040-actuator tweeter provides
high order wavefront correction with low-order modes offloaded to a $11\times
11$ 97-actuator woofer DM.

One unique aspect of MagAO-X is its additional DM for NCPC and Low-Order
Wavefront Sensing (LOWFS) within the coronagraph arm which we recently swapped
with a 1000 actuator BMC Kilo-DM as part of the MagAO-X ”Phase II”
upgrades[4]. This new NCPC DM surface required characterization to confirm
device quality and for commanding the reflective face sheet to a flat state
for efficient installation.

The main scientific motivation for the abovementioned Phase II upgrades is to
significantly enhance MagAO-X’s ability to image reflected starlight at close
inner-working angles (IWAs). Currently, one of the main limitations for this
mode of direct imaging is noise due to non-common path aberrations (NCPAs)
which are instrumental in origin and originate downstream of the wavefront
sensor (WFS) arm, so the AO system is unable to correct for them in typical
configurations. Furthermore, the “quasi-static” speckles due to NCPAs are
difficult to remove through conventional post-processing methods (e.g.,
angular differential imaging[5]) because they evolve too slowly to be averaged
and too quickly to be removed using a single speckle pattern realization for a
typical science observation. After characterization and installation of the
Kilo-DM, we used it to address these concerns. Specifically, this upgrade led
to faster LOWFS correction speeds and facilitated the Focus-Diversity Phase
Retrieval (FDPR) and implicit Electric Field Conjugation (iEFC) algorithms on-
sky for quasi-static speckle removal and in-situ Strehl ratio optimization,
respectively.

##  2 DM Characterization

We replaced MagAO-X’s NCPC DM as part of the Phase II upgrades. Namely, we
swapped the old ALPAO DM97 with a new BMC Kilo-DM which features a $32\times
32$ 952 microelectromechanical system (MEMS) actuator grid rated for $3.5\mu$m
of peak-to-valley (PV) surface stroke. Before the DMs could be exchanged, the
Kilo-DM needed to be characterized to avoid aberrating MagAO-X’s point-spread
function (PSF) and maintaining accurate wavefront control.

###  2.1 Procedures

Figure 2:  The BMC Kilo-DM (left) set up on the testbed and (right) installed
in MagAO-X after characterization.

We set up and aligned the DM on a test bench such that the surface could be
measured in real time with a 4D Technology PhaseCam (Twyman-Green)
interferometer (Figure 2) utilizing a HeNe laser source ($\lambda=632.8$ nm).
We used the BMC-supplied API to command the DM and the Python API provided by
4D Technology for interferometer operation, post-processing, and writing the
measurements to disk. Because the diameter of the stock source beam was not
large enough to measure the full test surface in a single exposure, we made
use of additional lenses to expand the beam size. These expanding optics
introduced low-order aberrations which were subtracted in post-processing
through the use of a reference made from a 4” flat mirror ($\lambda/20$ PV;
Figure 3).

We built an interaction matrix to command the DM surface using the same
general procedures established during past characterization of MagAO-X’s
tweeter and woofer DMs[6]. Namely, we first commanded a global DM surface
command to a voltage bias of 70% to maximize single actuator stroke then
measured single-actuator IFs by recording positive and negative DM pokes with
amplitude $0.16$ $\mu$m ($\lambda/4$). We then computed the difference image
to remove the surface form at its bias position; see Van Gorkom et al. [6] for
a representative IF for a MEMS DM. To expedite measurements of these IFs, we
built a Python-based pipeline to automate DM and interferometer operations
supported by a simple file monitoring class. On both the DM and 4D computer,
the monitor process passively watched for changes to an empty file before
allowing a DM command or interferometer capture to take place. For either a
successful DM command or interferometer measurement, the pipeline then updated
the empty file on the conjugate computer and the process continued, in sync,
until all measurements were written to disk.

Figure 3:  Optical setup we utilized for measuring DM influence functions for
driving the surface flat in closed-loop.

After capturing the IFs, we flattened and stacked the images to make the
response matrix. Put mathematically,

$\vec{s}=F\vec{a}$ (1)

where $\vec{s}$ is a vector portraying the form of the DM surface, $F$ is the
response matrix, and $\vec{a}$ is a vector of actuator commands. Furthermore,
the backbone for the closed-loop flattening operation is projection of the
measured DM surface onto the interaction matrix, which is given by the pseudo-
inverse of $F$, to obtain an approximation in DM space:

$\vec{a}=F^{\dagger}\vec{s}$ (2)

where $F^{\dagger}$ denotes the pseudo-inverted response matrix. To summarize,
we assume that we can approximate an arbitrary surface shape $\vec{s}$ given a
sufficient span of orthogonal IF measurements $F$. Since we measured the IFs
of single-actuator pokes, no IF can be reconstructed from any combination of
IFs from the other actuators thus this condition is satisfied.

Minimizing the surface rms about the mean, we used the abovementioned
interaction matrix $F^{\dagger}$ to drive the surface to a flat state, in
closed-loop, using the interferometer measurements (i.e., the WFS) to obtain
$\vec{s}$. We found solutions to two types of flats: a “relaxed” flat where
tip/tilt and defocus aberrations were fit and removed in post-processing and
thus were ignored when evaluating the fit metric and an “absolute” flat where
only tip/tilt was ignored (Figure 4).

Figure 4:  The best-fit DM command maps after the closed-loop flattening
procedure. The colorbars are in units of microns. Left: The command map after
12 iterations pertaining to the ”relaxed” flat where tip/tilt and defocus was
removed prior to calculating the surface rms about the mean. Right: The
command map after 11 iterations for the ”absolute” flat where only tip/tilt
was removed in post-processing.

The relaxed flat converged in 12 iterations and the absolute flat converged in
11. We chose the relaxed flat to maximize the available stroke across all
actuators[6] and opted to instead compensate for the remaining defocus by
translating MagAO-X’s science and LOWFS cameras as well as the focal plane
mask along the $z$-axis. The DM surface rms at the bias position was $\sim 47$
nm and was reduced to $\sim 10$ nm after the flattening procedure (Figure 5).

Figure 5:  DM surface pre- and post-flattening. Left: The bias position with
defocus removed exhibited roughly 47 nanometers surface rms. Right: The
surface rms reduced to about 10 nanometers, post-flattening. The spikes along
the edges of the surface map correspond to edge effects leaking through the
imperfect software mask and are $\sim 50$ nm above the surface mean.

###  2.2 Installation and Optimization

Figure 6:  To assemble a basis for use on MagAO-X’s new NCPC DM, we (1, top
left) define a generous aperture of size 32 by 26 actuators, (2, top right)
isolate an outer single actuator-wide ring to ensure that there are no sharp
surface features at the edge of the aperture, (3, bottom left) perform
nearest-neighbor interpolation within this outer ring, and (4, bottom right)
apply Gaussian smoothing. All values exterior to the original aperture from
(1) are replaced with the interpolated + smoothed values. An astigmatism
component is shown for example purposes.

After obtaining the best-fit command map for a relaxed flat, we installed the
new NCPC DM in MagAO-X in late 2023 (see the right panel in Figure 2 to see
the new DM installed in the instrument). We then aligned it in the lab using
the internal telescope simulator and a commanded binary pattern marking the
shape and position of the expected beam footprint on the NCPC DM. However, we
observed some remaining static aberrations even with the best-fit flat
command. To scrub out these residual aberrations, and for general utility, we
constructed an orthogonal basis for use over the area of beam footprint using
Zernike modes up to Z51. In practice, this basis is useful for manually
dialing out or inducing specific aberrations at the detector focal plane or
for use with, e.g., the “eye-doctor”[7, 6] algorithm for automated instrument
Strehl ratio optimization. Figure 6 details the process we used to construct
such a basis. We note that due to the highly elliptical illuminated area on
the DM, the question of how best to control the actuators outside of this
region is still being actively investigated.

##  3 On-Sky Performance

MagAO-X went on-sky with the new NCPC DM during the observing block in
March—May 2024, spearheading enhancements to MagAO-X’s unique ability to do
coronagraph-stage, real-time wavefront control. We had ample opportunity to
test the upgraded configuration during engineering time and science
observations; the results of these tests are briefly discussed in the next few
subsections.

Figure 7:  A render of MagAO-X tracing the beam paths as they traverse the
upper and lower optical benches. The blue beam starts on the upper bench and
shows the path taken by the light from the telescope. The magenta beam
illustrates the beam to the wavefront sensor. The yellow beam shows the light
reflected off the coronagraph for LOWFS.

###  3.1 NCPC and LOWFS

NCPAs originate within the instrument due the disconnect between the WFS and
science arms. Since NCPAs are not sensed by the high-order WFS, they become a
limiting factor for the contrast at the imaging plane. One way to fight this
phenomenon is adding additional WFSs further along in the optical train.

MagAO-X makes use of light rejected by the reflective coating of the Lyot
focal plane mask for LOWFS (yellow beam in Figure 7) for NCPA correction [8].
In addition to a considerable increase in the number of modes LOWFS can
correct for, the new Kilo-DM also increased the maximum closed-loop correction
speeds (from 2kHz to 10 kHz), increasing robustness to instrumental jitter. We
plan to perform more comprehensive tests with 10kHz correction speeds as well
as measuring and calibrating mode linearity with LOWFS during the next MagAO-X
observing block in Fall 2024.

###  3.2 Mitigating quasi-static speckles with iEFC

Figure 8:  iEFC results on the star $\beta$ Corvi (V = 2.6 mag). The star is
located at the top of both images and is blocked with a knife-edge
coronagraph. Top: iEFC disabled and quasistatic speckles are readily seen.
Bottom: Marked decrease in quasistatic speckles with iEFC enabled. The bright
dot remaining at the IWA is an injected companion. The prominent speckles that
are present in both images at the $\sim 4$, 6, and 8 o’clock positions are
induced by the DM surface.

Quasistatic speckles are a source of noise that stems from NCPAs and are
debilitating for AO imaging performance at close IWAs[9]. During the last
MagAO-X observing run in 2024A, we demonstrated focal plane wavefront sensing,
on-sky, using the iEFC algorithm[10] to mitigate this noise source. Briefly,
this algorithm combines Electric Field Conjugation[11] with an empirical
calibration step via the NCPC DM (i.e., no optical model is needed to
reconstruct the electric field). With knowledge of the morphology of the
electric field, the NCPC DM can then cancel by instilling the opposite
electric field. Our on-sky iEFC efforts produced a rectangular dark hole of
size $11\times 13$ $\lambda/D$ which is a region of high-contrast almost
completely devoid of quasistatic speckles (Figure 8). These results will be
described in detail in a later publication (Haffert et al., in prep). One
avenue for future work is studying the effects of DM registration to the
detector plane.

###  3.3 In-situ Strehl ratio optimization with FDPR

Figure 9:  Closed-loop FDPR results on the star $\beta$ Corvi (V = 2.6 mag)
imaged through a narrowband $CH4$ filter (875 nm). The initial Strehl ratio
was 0.71. After 10 iterations of closed-loop FDPR, the Strehl ratio increased
to 0.85. The PSF core is shown inset in the bottom right corner.

The FDPR algorithm[12, 13, 6] is one way to measure the spatial distributions
of phase and amplitude at the pupil plane in an AO system. At a high level,
FDPR attempts to minimize the error between PSF models and images at multiple
planes of defocus to avoid solution degeneracies.

We report successful on-sky use of closed-loop FDPR leading to drastic
improvements in Strehl ratio as part of an initial calibration step before
observing a given science target. To supply the algorithm with the necessary
components, we use the NCPC DM to apply precise amounts of defocus to the PSF
image which is recorded on one of MagAO-X’s science cameras. After the
requested amount of iterations is completed on a bright ($V\lesssim 5$ mag)
star, we use the NCPC DM command map that minimizes differences between the
PSF images and models as the new flat command for any subsequent science
observations in a nearby portion of the sky. Figure 9 shows the degree of
improvement for the Strehl ratio after 10 iterations of closed-loop FDPR while
imaging through a narrow $CH4$ filter (875 nm) on the bright star $\beta$
Corvis. We note, however, that FDPR does instantaneous band-limited
measurements for the Strehl ratio thus neglecting vibrations; the true Strehl
ratio is likely lower by about 0.1 to 0.2.

Currently, for on-sky use, the defocused PSF images are first averaged for
$\sim 5$ seconds to temper the effects of atmospheric turbulence before
feeding it into the algorithm. There would be value in further study as to
determine optimal FDPR parameters based on the current observing conditions
(e.g., how do we best average the PSF images over turbulence?).

##  4 Summary and Future Work

We upgraded MagAO-X’s NCPC DM to a BMC Kilo-DM as part of a comprehensive
suite of ongoing upgrades[4] to be completed in the next couple of years. We
detailed successful characterization of the device prior to installation
within the instrument. We also shared our on-sky results using the new NCPC DM
during the MagAO-X observing block in 2024A. The key takeaways from our on-sky
tests with this new DM are listed as follows:

  * •

MagAO-X’s unique ability to do coronagraph-stage wavefront control has been
greatly improved.

  * •

We demonstrated quasi-static speckle removal in a $11\times 13$ $\lambda/D$
control radius ”dark hole” without any loss of stroke from the 2040-actuator
tweeter DM.

  * •

The FDPR algorithm is an effective way to drastically improve Strehl ratio as
part of an initial calibration step before observing a given science target.

  * •

LOWFS on MagAO-X is much improved with faster correction speeds, increasing
robustness to instrumental jitter.

###### Acknowledgements.

We are thankful for support by the NSF MRI Award no. 1625441 (MagAO-X). This
work and the rest of the MagAO-X Phase II upgrades are made possible by the
generous support by the Heising-Simons Foundation.

## References

  * [1] Males, J. R., Close, L. M., Miller, K., Schatz, L., Doelman, D., Lumbres, J., Snik, F., Rodack, A., Knight, J., Van Gorkom, K., Long, J. D., Hedglen, A., Kautz, M., Jovanovic, N., Morzinski, K., Guyon, O., Douglas, E., Follette, K. B., Lozi, J., Bohlman, C., Durney, O., Gasho, V., Hinz, P., Ireland, M., Jean, M., Keller, C., Kenworthy, M., Mazin, B., Noenickx, J., Alfred, D., Perez, K., Sanchez, A., Sauve, C., Weinberger, A., and Conrad, A., “MagAO-X: project status and first laboratory results,” in [Adaptive Optics Systems VI ], Close, L. M., Schreiber, L., and Schmidt, D., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10703, 1070309 (July 2018).
  * [2] Close, L. M., Males, J. R., Durney, O., Sauve, C., Kautz, M., Hedglen, A., Schatz, L., Lumbres, J., Miller, K., Van Gorkom, K., Jean, M., and Gasho, V., “Optical and mechanical design of the extreme AO coronagraphic instrument MagAO-X,” in [Adaptive Optics Systems VI ], Close, L. M., Schreiber, L., and Schmidt, D., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 10703, 107034Y (July 2018).
  * [3] Brennan, T. J. and Rhoadarmer, T. A., “Performance of a woofer-tweeter deformable mirror control architecture for high-bandwidth high-spatial resolution adaptive optics,” in [Advanced Wavefront Control: Methods, Devices, and Applications IV ], Giles, M. K., Gonglewski, J. D., and Carreras, R. A., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 6306, 63060B (Aug. 2006).
  * [4] Males, J. R., Close, L. M., Haffert, S., Long, J. D., Hedglen, A. D., Pearce, L., Weinberger, A. J., Guyon, O., Knight, J. M., McLeod, A., Kautz, M., Van Gorkom, K., Lumbres, J., Schatz, L., Rodack, A., Gasho, V., Kueny, J., and Foster, W., “MagAO-X: current status and plans for Phase II,” in [Adaptive Optics Systems VIII ], Schreiber, L., Schmidt, D., and Vernet, E., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 12185, 1218509 (Aug. 2022).
  * [5] Marois, C., Lafrenière, D., Doyon, R., Macintosh, B., and Nadeau, D., “Angular Differential Imaging: A Powerful High-Contrast Imaging Technique,” The Astrophysical Journal 641, 556–564 (Apr. 2006).
  * [6] Van Gorkom, K., Males, J. R., Close, L. M., Lumbres, J., Hedglen, A., Long, J. D., Haffert, S. Y., Guyon, O., Kautz, M., Schatz, L., Miller, K., Rodack, A. T., Knight, J. M., and Morzinski, K. M., “Characterizing deformable mirrors for the MagAO-X instrument,” Journal of Astronomical Telescopes, Instruments, and Systems 7, 039001 (July 2021).
  * [7] Bailey, V. P., Hinz, P. M., Puglisi, A. T., Esposito, S., Vaitheeswaran, V., Skemer, A. J., Defrère, D., Vaz, A., and Leisenring, J. M., “Large binocular telescope interferometer adaptive optics: on-sky performance and lessons learned,” in [Adaptive Optics Systems IV ], Marchetti, E., Close, L. M., and Vran, J.-P., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 9148, 914803 (July 2014).
  * [8] McLeod, A. L., Low Order Wavefront Sensing and Control on MagAO-X, Master’s thesis, University of Arizona (2023).  https://repository.arizona.edu/handle/10150/669618.
  * [9] Hinkley, S., Oppenheimer, B. R., Soummer, R., Sivaramakrishnan, A., Roberts, Lewis C., J., Kuhn, J., Makidon, R. B., Perrin, M. D., Lloyd, J. P., Kratter, K., and Brenner, D., “Temporal Evolution of Coronagraphic Dynamic Range and Constraints on Companions to Vega,” The Astrophysical Journal 654, 633–640 (Jan. 2007).
  * [10] Haffert, S. Y., Males, J. R., Van Gorkom, K., Close, L. M., Long, J. D., Hedglen, A. D., Ahn, K., Guyon, O., Schatz, L., Kautz, M., Lumbres, J., Rodack, A., Knight, J. M., Sun, H., Fogarty, K., and Miller, K., “Advanced wavefront sensing and control demonstration with MagAO-X,” in [Adaptive Optics Systems VIII ], Schreiber, L., Schmidt, D., and Vernet, E., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 12185, 1218581 (Aug. 2022).
  * [11] Give’on, A., Kern, B., Shaklan, S., Moody, D. C., and Pueyo, L., “Broadband wavefront correction algorithm for high-contrast imaging systems,” in [Astronomical Adaptive Optics Systems and Applications III ], Tyson, R. K. and Lloyd-Hart, M., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 6691, 66910A (Sept. 2007).
  * [12] Thurman, S. T., DeRosa, R. T., and Fienup, J. R., “Amplitude metrics for field retrieval with hard-edged and uniformly illuminated apertures,” Journal of the Optical Society of America A 26, 700 (Mar. 2009).
  * [13] Fienup, J. R., “Phase retrieval algorithms: a personal tour [Invited],” Applied Optics 52, 45 (Jan. 2013).


# Retargeting Visual Data with Deformation Fields

Tim Elsner
Visual Computing Institute
RWTH University
<EMAIL_ADDRESS>Julia Berger
Visual Computing Institute
RWTH University
<EMAIL_ADDRESS>Tong Wu
Institute of Computing Technology
Chinese Academy of Sciences
<EMAIL_ADDRESS>Victor Czech
Visual Computing Institute
RWTH University
<EMAIL_ADDRESS>Lin Gao
Institute of Computing Technology
Chinese Academy of Sciences
Leif Kobbelt
Visual Computing Institute
RWTH University

###### Abstract

Seam carving is an image editing method that enable content-aware resizing,
including operations like removing objects. However, the seam-finding strategy
based on dynamic programming or graph-cut limits its applications to broader
visual data formats and degrees of freedom for editing. Our observation is
that describing the editing and retargeting of images more generally by a
displacement field yields a generalisation of content-aware deformations. We
propose to learn a deformation with a neural network that keeps the output
plausible while trying to deform it only in places with low information
content. This technique applies to different kinds of visual data, including
images, 3D scenes given as neural radiance fields, or even polygon meshes.
Experiments conducted on different visual data show that our method achieves
better content-aware retargeting compared to previous methods.

Figure 1: Different objectives applied to different visual domains with our
approach: We demonstrate retarget images, NeRFs, and meshes. Image from [44].

##  1 Introduction

Media retargeting is a way to edit images, videos, 3D objects, or even entire
3D scenes by a global deformation such that relevant content, details, and
features are properly preserved. The overall goal is to change the aspect
ratio of the scene’s bounding box so that it fits an allocated space, _e.g_.
fitting an image to a display with prescribed format. Retargeting is based on
the idea of identifying regions with little detail and to accumulate the
necessary distortion induced by the deformation in these regions. Small
editing operations are possible _e.g_. through concentrating the distortion
there, specifying a region containing a particular object to have this object
removed.
While most existing retargeting methods typically use some form of seam
carving, _i.e_. the deletion of discrete pixels [2, 43, 8, 9], we formulate
the problem via a continuous deformation field that maps from the retargeted
image or scene back to the undistorted input such that, _e.g_., colours or
normals can be queried. This allows us to handle discrete media (images) and
continuous media (3D objects and scenes) equally.
In this paper, we introduce a more general approach for retargeting various
forms of visual media, including the first approach that allows retargeting of
neural radiance fields [29] (NeRFs). We also demonstrate our approach for
images and polygon meshes, as can be seen in Fig. 1. Our contributions are as
follows:

  1. (a)

We introduce the use of a neural deformation field compressing or stretching
low information content regions to achieve a smaller or larger output or to
follow other editing objectives.

  2. (b)

We demonstrate the domain agnostic nature of our formulation by applying it to
images, 3D meshes, and 3D scenes in the form of NeRFs.

  3. (c)

We show that the high flexibility of our approach from optimising a global
deformation field produces better outputs than iteratively computing seams.

To achieve this, we regularise the neural deformation fields to follow general
sanity guidelines, while attempt to minimize distortion in places with
presumed high information content. We evaluate our results on both qualitative
and quantitative levels. To showcase our approach on interesting scenes
captured as NeRFs and inspired by the dataset of Richter _et al_. [41], we
also provide a small synthetic NeRF retargeting dataset captured in the video
game GTA V [34]. Our method can be applied in a matter of seconds, requires
only minor modifications for vastly different applications and domains, and is
straightforward to implement. We will provide the code of our method
demonstrated on images on GitHub111Prior to the final version, our code,
including our NeRF retargeting dataset and example videos showing 3D scenes in
motion, can be found in the supplemental material.

##  2 Related Work

As our approach is inspired by seam carving, it produces results from a single
example without any learned prior, and was designed to allow seam carving-like
deformations for learned 3D representations. We hence discuss these three
branches of related work.

#### Seam Carving

Image/video retargeting is an important tool in computer vision and computer
graphics. Early works [25] first annotate or detect the region of interest
(ROI), which is uniformly scaled to the target size afterwards. While contents
out of the ROI are deformed by fisheye warping, Liu and Gleicher [26] later
extend it to the video domain by introducing a motion salience map. Avidan and
Shamir [2] propose seam carving, a content-aware image resizing method. It
first defines an energy map by summing the gradient norms along the horizontal
and vertical directions. The cost of a seam is defined by adding up the energy
values of pixels on it. The seam with minimal energy can be found by dynamic
programming and will be removed in the resizing process. Rubinstein _et al_.
[42] extend the seam carving idea to the video domain and transform the
dynamic programming problem to a graph cut problem. Follow-up works [43, 8, 9]
incorporate homogeneous cropping and scaling with seam carving and excavate
more high-level information like symmetry [51], depth [4], and object
similarity [10] to assist seam carving. With the advances in deep learning, a
few methods [30, 32, 33] made attempts to detect and localise the seam carving
forgery. Song _et al_. [46] propose Carving-Net to replace the previous
handcrafted energy map with a neural network while the seam finding algorithm
is still dynamic programming. Instead of using networks to improve seam
selection, our approach replaces the discrete seams and dynamic programming
itself with a neural network, globally optimising a continuous deformation
field.

#### Single Example Generation

The goal of single example generation is to discover similar local patterns
from a single input and synthesise novel samples. Texture synthesis on 2D
images and 3D shapes has been widely studied in computer vision and computer
graphics by non-neural methods [12, 49, 11, 23]. There are also methods
focusing on synthesising geometric textures for 3D shapes [24, 5, 18], which
transfer local geometry details from an example shape to other surfaces.
Recently, neural-based methods have been proposed to generate 2D textures [14,
54, 45] and transfer them to 3D shapes [16, 17], resulting in improved
results. With recent advances in implicit representation [7, 35, 28] and
neural radiance fields [29], Wu _et al_. propose to learn implicit 3D shapes
from a shape with [53] or without texture [52]. To enhance the realism of
synthesised shapes, Huang _et al_. [20] reconstructs the 3D shape with a NeRF
by optimising texture features on a reconstructed mesh. These features can be
applied to arbitrary surfaces to render the target surface with synthesised
textures. These methods often only work for a certain kind of data format,
while our method proposes a domain agnostic approach to synthesise visual data
from a single example.

#### Deformations for Learned 3D Representations

Neural radiance fields (NeRF), as a newly proposed 3D shape representation,
has been widely used in scene reconstruction and novel view synthesis.
However, the original NeRF only works for static scenes thus unable to deal
with deformation in dynamic scenes or generate motions for a static scene. To
overcome this, the deformation field is introduced to NeRF. NeRF-Editing [57]
is the first to explore deformation in NeRF. It first reconstructs the
explicit mesh of a static scene with NeuS [48] and deforms the explicit mesh
with the ARAP algorithm [47]. Then sample points in volume rendering are
deformed along with the mesh via barycentric coordinate interpolation,
resulting in the change of rendered images. Follow-up NeRF deformation methods
[55, 39, 13] utilise a similar pipeline but use different geometry proxies. To
reconstruct dynamic scenes, Albert _et al_. [40] first explore the possibility
by adding an extra time-conditioned deformation field to the static NeRF. The
time-conditioned deformation field transforms sample points in the observation
space into the canonical space, where their colours and opacities are queried
and then rendered into images. This idea stimulated a series of dynamic NeRF
reconstruction methods [36, 37, 6, 27, 56, 50]. While our approach also uses a
deformation field, we inject general plausibility constraints into it to keep
the results plausible, enabling the idea behind seam carving on different
types of visual data.

Figure 2: Our proposed pipeline for retargeting visual data: For a given input
(left), we train two simple networks that learn the energy and cumulative
energy along the deformation axis of the input (centre-left). We then
initialise a network that stretches samples to the desired position (centre-
right), then optimise this deforms to distribute the distortion to low
information content regions (right). Image from [44]. Figure 3: The part of
our pipeline to apply it to neural radiance fields: We first obtain a point
cloud, then estimate the energy values. We then learn a continuous energy
function and a cumulative energy function that we use to optimise our
deformation network. We visualise the resulting deformation field on the
bottom right.

|  |
---|---|---
Input | Seams to carve | Seam carved

|
---|---
Our folds | Ours

Figure 4: Example of applying retargeting to $75$ percent width. Seam carving
fails to use seams that are non continuous or with a too steep angle, not
properly using the empty space in the shelf to close the gaps before shrinking
the bookshelves content.

Figure 5: Applying our retargeting to NeRFs. Top row: Input, stretched, retargeted with our method. Bottom row: Input, folds on the input suggested by our approach, retargeted with our method. |  |
---|---|---
Input | Initial / uniform | Optimised
Figure 6: Applying our retargeting to 3D meshes.

##  3 Deformation Fields for Retargeting Visual Data

Editing strategies for visual data require avoiding modifications on
meaningful contents, which create visible artefacts. An approach that focuses
on retargeting the input to match a target size can be guided by the following
set of rules:

  1. (a)

Content Aware Objective: Prevent the accidental creation of new or the removal
of existing regions with high information content (controlling where
deformation happens).

  2. (b)

Sanity Objective: Avoid introducing deformations that yield implausible
results (controlling how deformation happens).

Note that for the following sections, we only consider retargeting visual data
to a smaller size (shrinking). This formulation can be extended naturally to
provide editing (see Sec. 3.5) or to retarget to a larger size (expansion, see
Sec. 8).

#### Discrete Solution

Seam carving [2] performs editing by manipulating (removing or doubling) one
seam of pixels at a time, _i.e_. a path with the width of one single pixel
passing through the image. Through dynamic programming, seam carving chooses
this path based on the lowest energy of the pixels, with high energy
indicating information-rich areas. In a simple case, this energy is the colour
gradient. This yields a discrete solution to the retargeting problem that is
both content aware and sane. While this naturally extends to voxelised 3D
scenes, it would become costly at high resolutions while suffering from
voxelisation artefacts.

#### Continuous Solution

Our approach follows the same guidelines, but instead of discrete, pixel-wise
paths through the image, we learn a deformation function for continuous
inputs. By working directly in the continuous domains (_e.g_. not requiring
voxelisation of a 3D scene), our formulation is less prone to artefacts, while
allowing for more degrees of freedom by accommodating non-continuous
compression and expansion (see Fig. 4). We consistently utilise the same core
formulation, regardless of the underlying domain and application, _e.g_.
moving an object in an image or retargeting a 3D scene. For clarity, we will
refer to our continuous seams as folds.

###  3.1 General Formulation

For input visual data $I$, we train a neural deformation field $D$, a simple
MLP. The deformation field provides an offset to each input coordinate, _i.e_.
it learns an $I$-specific deformation field to look up the content of $I$ at a
different place (see Fig. 2). For any point $p\in P$ of the domain of $I$, the
deformation $D(p)$ yields a scalar offset in a deformation direction $v$ for
$p$. We define $P$ to contain all possible input positions for $I$, _e.g_.
$P=[0,1]^{2}$ for a square image. Adding this scaling of $v$ to $p$ we obtain
a deformed point $p^{\prime}$:

$p^{\prime}=p+vD(p)$ (1)

This deformation field is always tuned to the visual data $I(p)$, _e.g_. a
previously trained NeRF or an image. We only optimise the deformation field
that gives an offset for the mapping from the edited output data to the input
data. To fulfil the objectives stated in Sec. 3.1, we introduce two sets of
losses:
Content Aware Objectives We apply regularisation to guide the deformation
process in order to adjust the learnt deformation to the content of the data.
To discourage noticeable changes in regions with high information content, we
penalise the product between deformation magnitude and information content,
expressed as an energy function $E$. The change in the deformation field
between two points also change the distance between them in the output,
suggesting a potentially noticeable deformation at this point. We penalise
these distortions in areas with high information content. We define $E(p)$ to
be a measure of content information at $p$, expressed as, _e.g_., the gradient
in the visual data:

$E(p)=||\nabla I(p)||_{2}$ (2)

We therefore define our general energy term that measures content deformation
as follows:

$L_{C}=\int\limits_{p\in P}\left[E(p+v\cdot D(p))\cdot||\nabla
D(p)||_{1}\right]\ dp$ (3)

Thus, in Eq. 3, we effectively minimise the product of information content at
the deformed point as $E(p+v\cdot D(p))$ and the change in the deformation
itself as $||\nabla D(p)||_{1}$, penalising folds in the deformation at
regions with high information content. Using the $L^{1}$-norm for the
distortion promotes piecewise constant, possibly sparse, deformations. We can
then split up the change in deformation to obtain:

$\displaystyle L_{C}=\int\limits_{p\in P}\biggl{[}$ $\displaystyle E(p+v\cdot
D(p))$ (4) $\displaystyle\cdot\left(\left|\frac{\partial}{\partial
v}D(p)\right|+\left|\frac{\partial}{\partial
v^{\perp}}D(p)\right|\right)\biggr{]}\ dp$

This equation is then split into two components, $L_{e}$ penalising stretching
or compression and $L_{s}$ punishing shearing (deformation change in a
direction that is not our deformation direction). For a now discrete sample
set $P$ from our domain, these terms are expressed as

$\displaystyle L_{e}=$ $\displaystyle\sum_{p\in P}\left[E\left(p+v\cdot
D(p)\right)\cdot\left|\frac{\partial}{\partial v}D(p)\right|\right]\ dp,$ (5)
$\displaystyle L_{s}=$ $\displaystyle\sum_{p\in P}\left[E\left(p+v\cdot
D(p)\right)\cdot\left|\frac{\partial}{\partial v^{\perp}}D(p)\right|\right]\
dp.$

$L_{e}$ directs deformation to low information content areas, while $L_{s}$
discourages shearing for high information content regions. Note that while
$L_{s}$ would also penalise non-straight deformation seams, we tune our losses
such that this only affects the output substantially if an area is sheared
(_e.g_. for the balloons in Fig. 2, the deformation going around the balloons
introduces shearing only right at the seam in an area with low information
content).
In case of image deformation, neighbouring pixels with low energy in the input
may translate to pixels widely apart in the output, potentially bypassing a
substantial amount of (important) content. To overcome this, we want to
penalise the amount of energy $\hat{E}$ between two points $p,q$ in the target
domain, rather than the energy at individual positions:

$\hat{E}(p,q)=\int\limits_{p^{\prime}\in[p,q]}E(p^{\prime})\ dp^{\prime}$ (6)

Now, we re-define $L_{e}$ to be the product of the energy between points and
their deformation change,

$\displaystyle L_{e}=\sum_{p\in P}$
$\displaystyle\hat{E}\left(p+vD(p),p_{\varepsilon}+vD(p_{\varepsilon})\right)$
(7)
$\displaystyle\cdot\frac{\left|D(p)-D(p_{\varepsilon})\right|}{\varepsilon}$

where $p_{\varepsilon}:=p+\varepsilon$ is a slightly offset $p$ for small
$\varepsilon$. This formulation guides the deformation to avoid introducing
deformation to high information content regions or fold over them, while
$L_{s}$ discourages non-local deformations that would shear data with high
information content.

Sanity objectives To further prevent undesired outputs, we introduce two
additional losses. Assume we shrink the input by a factor $\alpha\in[0,1]$:

  1. 1.

The boundaries of the affected region should not change, _e.g_. an image
should not be simply cropped. Based on $\alpha$, the deformation field should
map the start/end of the deformation axis in the output to the start/end of
the input. For $P_{0}$, $P_{1}$ containing all points at the start/end of the
deformation axis of the output, we express this requirement as follows:

$L_{b}=\int\limits_{p\in P_{0}}|D(p)|\ dp+\int\limits_{p\in
P_{1}}|D(p)-(1-\alpha)|\ dp$ (8)

  2. 2.

We additionally demand monotonicity in the deformation field, expressed as

$L_{m}=\int\limits_{p\in P}\max\left(0,-\frac{\partial}{\partial
v}D(p)\right)\ dp$ (9)

indicating that any change in the deformation direction should be positive,
_i.e_. the deformation $D(p)$ should be monotonic and only stagnate or
increase exclusively along $v$. This prevents the repetition of the same image
portion (”jumping back and forth” through our deformation), which might result
in bad optima.

We then obtain the total loss $L$ with weights $\lambda$ as

$L=\lambda_{e}L_{e}+\lambda_{s}L_{s}+\lambda_{b}L_{b}+\lambda_{m}L_{m}.$ (10)

To promote convergence and avoid undesirable local optima, we always
initialise our deformation function with a uniform stretch to the target size.
In the following, we demonstrate the application of this general formulation
to different domains, adapting the energy formulation. All implementation and
architecture details are given in Sec. 6 and the exact loss terms in Sec. 7.

###  3.2 Optimising Deformation Fields for Images

To ensure consistency with our continuous approaches, we first train a
continuous image representation MLP $I$ to serve as our input data, mapping
pixel coordinates to colours.
We train a neural network to first learn a energy field $E$, then a cumulative
energy field $\Sigma$ learning to reproduce $\hat{E}$ with an MLP. The
difference in cumulative energy $\Sigma$ between two points $p,q$ on the same
axis $v$ is then the absolute difference in information content between them:

$\hat{E}(p,q)\approx|\Sigma(q)-\Sigma(p)|$ (11)

For all objectives, we simply use discrete approximations of the loss terms
introduced in Sec. 3.1.
The whole pipeline is visualised in Fig. 2, and extra results can be found in
Tab. 4.

###  3.3 Optimising Deformation Fields for Neural Radiance Fields

A naive extension of the energy term from 2D to 3D as the product of density
and change in colour, as given by the NeRF has two major issues: change of
colour and density may not be well-defined for single points (_e.g_. consider
mip-NeRF [3] only using volumes), while a large number of samples are in empty
space, hence are unimportant. To address this, we propose in the following a
strategy to optimise for points on the surface instead, as visualised in Fig.
3:

#### 1\. Extracting a Sparse Set of Representatives

Using the RGB and depth images of applying the NeRF from the training dataset
positions, we extract the surface point cloud that, by definition, contains
the points with the most significant role in the deformation. We then compute
the energy value for each point corresponding to a pixel as a sum of colour
change in the RGB image and depth change in the depth image. As this leads to
highly noisy energy values, we apply smoothing by taking the minimum energy
for each point in a neighbourhood radius of 5 points.

#### 2\. Defining an Energy Function

Since colour change alone is not a sufficient energy formulation, our defined
energy combines colour change in the rendered image and change in the depth
image. Hence, each point contributing to the rendered image is equipped with
its corresponding energy value. To obtain a continuous energy function we can
evaluate for every point in the domain, we train an auxiliary energy network
$E_{net}$. The network maps all 3D points to their energy values, and clamps
the energy to zero for all points further than twice the average neighbour
distance. The resulting energy field has zero energy in empty space and inside
objects, and non-zero energy values for points near the surface. Following the
deformation axis, we additionally learn a cumulative energy field, expressed
as a network $E_{cnet}$. The network is trained by shooting rays in the
direction of deformation, and accumulating energy values from $E_{net}$ from
uniformly sampled points. Having the cumulative energy field, we can
implicitly compute the energy for a segment between two points along $v$.

#### 3\. Inverting the Deformation

We define an inverse deformation function to determine those points that lie
on the surface after deformation. In contrast to images, we only perform
optimisation on a sparse set of points, namely surface points. As we optimise
a deformation from the target space to the source space, we need the position
of surface points in both spaces. By training an MLP $U$ to reverse the
deformation, _i.e_. minimising $(U(D(x))=x)^{2}$, we can effortlessly extract
surface points, and hence their energy values, in the target space. This
relation is also visualised in Fig. 3. We always update $U$ once after
training one iteration of $D$.

We therefore adapt our general approach from Sec. 3.1 for NeRFs as follows:

  * •

We train a continuous energy function $E$ from sparse samples on the NeRF
surface, then train a cumulative energy function $\Sigma$ from that.

  * •

We use an inverse deformation network $U$ to find points in the target space
that map to surface points in the source space.

  * •

We apply our regularisation terms on a mix of surface and random points to
optimise our loss function.

As we use a discrete set of points, we apply the losses from Sec. 3.1 to Eq.
10 for a discrete interval, _i.e_. mean over a random subset of sample points.
Example images from deformed scenes can be found in Fig. 5, while a larger
number of results can be found in Sec. 10.

###  3.4 Optimising Deformation Fields for Meshes

To show the versatility of our approach, we also it to meshes, only redefining
the energy term and using surface samples and vertex positions to optimise.
For objects like chairs, we used curvature, and for the example in Fig. 6, we
simply set the energy for the floor to $0$ and to $1$ for everything else.

Input |  |
---|---|---
Edited |  |
Figure 7: We demonstrate different editing operations by simply adding and
adapting the loss formulation, leading to different editing operations (always
input on top, output on the bottom, green indicating a given input mask).
Left: Removing an Object, Right: Moving an object. Images from [44].

###  3.5 Other Applications

While retargeting is our main contribution, just as for seam carving [2], our
framework opens up multiple different editing opportunities that even extend
seam carving capabilities. While all details can be found in Sec. 9, the
following exemplary operations were implemented through minor changes in the
loss formulation, are applicable to any domain, and could form the basis for
further editing methods or user-driven applications. Examples can be found in
Fig. 7.
Removal To remove a part of a scene, we remove the monotonicity loss, use the
same size for input and output, and add another loss term that penalises every
deformation target that leads to the region, that should be removed. Hence,
points originally mapping to the target region will be distorted to
neighbouring regions.
Moving an Object Although classic seam carving cannot trivially move an
object, since seams are assumed to cross the whole image, our formulation has
no such restriction. We remove the monotonicity term, then enforce the
deformation to point at the target feature from the desired new place of the
object and punish it occurring anywhere else to avoid just repeating the
object.

##  4 Evaluation

As retargeting quality is difficult to measure and lacks similar methods for
_e.g_. meshes, as the backbone for our evaluation, we compare our results to
seam carving [2] on images using the RetargetMe [44] benchmark for image
retargeting. In 3D, lacking an alternative, we compare with the video-seam-
carved animation frames of the same scene. To allow a fairer comparison with
video seam carving, we only pan down the camera instead of rotating around the
object (_e.g_. spinning the camera around an object would be disadvantageous
for video seam carving). We always produce a short, $~{}3$ second clip at
30fps.

###  4.1 Comparisons

#### Quantitative Comparison

$x$ FID($\downarrow$) | 50% | 60% | 70% | 80% | 90% | mean | 110% | 120% | 130% | 140% | 150% | mean
---|---|---|---|---|---|---|---|---|---|---|---|---
Seam carving | 85.75 | 66.41 | 51.67 | 35.20 | 23.80 | 52.57 | 20.86 | 30.01 | 37.38 | 43.45 | 46.37 | 35.61
Ours | 79.39 | 58.06 | 43.69 | 30.53 | 21.71 | 46.68 | 11.17 | 19.42 | 29.19 | 37.87 | 49.74 | 29.47
$y$ FID($\downarrow$) | 50% | 60% | 70% | 80% | 90% | mean | 110% | 120% | 130% | 140% | 150% | mean
Seam carving | 116.70 | 84.33 | 60.59 | 40.86 | 24.92 | 65.48 | 23.04 | 33.58 | 40.82 | 48.49 | 52.59 | 39.70
Ours | 89.44 | 67.77 | 49.62 | 35.59 | 25.42 | 53.57 | 13.55 | 21.02 | 27.55 | 38.20 | 46.40 | 29.00
Table 1: Comparison of FID scores on the RetargetMe [44] dataset, for
retargeting x (top) and y (bottom) axis to a smaller (left) and bigger (right)
size.

In accordance with recent publications in the generative model and image
editing domain [15, 38, 21], we compare our results and seam carving results
with the original data using FID [19] in Tab. 1 for images. While we observe
the results to align with our perception, we also validate this on a
qualitative level.

#### Qualitative Comparison

Figure 8:  Selected results showing scenes where our approach benefits from higher degrees of flexibility (always input, seam carving, ours), all images from [44].  Retargeting width | Ours | SC | Draw
---|---|---|---
50 $\%$ | 48.48 % | 20.87 % | 30.67 %
150 $\%$ | 33.8 % | 36.27 % | 29.92 %
Retargeting height | Ours | SC | Draw
50 $\%$ | 49.5 % | 24.25 % | 26.25 %
150 $\%$ | 45.84 % | 20.0 % | 34.15 %
Total |  44.57 % | 25.1 % | 30.32 %
Table 2: User study comparing our approach to seam carving (SC) for 2D images, for different sizes and axis. $19$ participants were asked to indicated which solution they prefer for random examples from the RetargetMe dataset[44], casting about $1200$ votes in total. | X Axis to 50 % | X Axis to 150 %
---|---|---
Scene | Ours | VSC | Draw | Ours | VSC | Draw
Beach | 17 | 3 | 1 | 20 | 1 | 0
Farm | 21 | 0 | 0 | 21 | 0 | 0
Harb. | 21 | 0 | 0 | 21 | 0 | 0
| Y Axis to 50 % | Y Axis to 150 %
Scene | Ours | VSC | Draw | Ours | VSC | Draw
Beach | 19 | 1 | 1 | 19 | 1 | 1
Farm | 16 | 3 | 2 | 21 | 0 | 0
Harb. | 21 | 0 | 0 | 21 | 0 | 0
Table 3: User study comparing retargeting a scene, then recording it (ours),
to recording it and then retargetet it (video seam carving [42], VSC), using
our provided dataset. $21$ participants were asked to indicated which solution
they prefer, casting $252$ votes in total.

As image quality is a subjective criterion and may not align with our
quantitative results, we validate our quantitative results by having humans
classify which retargeted version they prefer. For this, we evaluate in a
simple user study where participants were asked to classify which retargeted
version of a randomly chosen image for a randomly chosen deformation axis they
preferred, always choosing between our and the seam carved[2] version. We used
the $80$ images of the RetargetMe [44] for 2D, producing $360$ variations per
approach.
For retargeting quality of 3D scenes given as NeRFs, we compare ours results
with what we consider to be the approach nearest of kin, video seam
carving[42]. We use a $50\%$ and a $150\%$ version for each axis of our
provided GTA V NeRF retargeting dataset.
The results can be found in Tab. 2 (2D images) and Tab. 3 (3D scenes).

#### Visual Ablation

|
---|---
without $L_{s}$ | without $L_{e}$
|
without $L_{m}$ | without $L_{b}$
Figure 9: Visual ablation for retargeting to $80$ percent width on an
image[1], from top left, clockwise: Wobbly contours (no $L_{s}$), only uniform
stretch (no $L_{e}$), swallowed features (no $L_{m}$), no boundary (no
$L_{b}$).

We provide a simple visual ablation that shows how our method requires all of
its loss terms in Fig. 9. We observed similar behaviour for 3D NeRFs and
meshes.

#### Methodological Comparison

Compared to classic seam carving, our approach provides a superset of all
possible solutions from discrete seam carving, as _e.g_. any removed seam can
simply be expressed as a jump in the output of our deformation network. Hence,
we extend the space of possible solutions. Our approach also tends to be less
”jaggy” compared to individually removed seams, as neural networks fall into
optima that are continuous rather than changing with high frequency. While
removal of individual seams one after the other also prevents shearing by
construction, we do not punish shearing that occurs in unimportant areas,
gaining flexibility. Especially compared to video seam carving, our approach
excelled in 3D, as we are able to produce geometry aware results.

###  4.2 Discussion and Limitations

Successors to seam carving focused on adding clever regularisations [51, 4,
10] or better energy terms [46]. While we only relied on colour gradient for
our comparison to the baseline, showing our advantage without any improved
energy formulation, and focused on redefining and generalising the approach
itself rather than exploring _e.g_. energy terms we can incorporate in our
approach by design. However, when _e.g_. applying our approach to images,
where the assumption of important parts having high colour gradient breaks,
will produce flawed outputs, just as seam carving does. This could be
addressed by using a similar loss function like the discriminator from SinGAN
[45] or using a more complex measure for information content, as also
partially explored with a face detector in [2]. Further, we do not learn a
general deformation network but have to newly optimise a neural field for each
individual input; However, this also means we do not require any training data
other than our input image. Lastly, regarding computation time, our approach
is slightly slower than seam carving for images, which we attribute to a more
global solution, but excelled in 3D, as _e.g_. video seam carving on a 600 by
400 pixel image can takes multiple hours, while our approach produces a
retargeted 3D scene within 15 minutes.

##  5 Conclusion

We present an approach that generalises the principle underlying seam carving
to create image variants by manipulating its low content areas. By training
neural deformation fields that avoid applying distortion to important parts of
an input, our approach is applicable to different editing objectives, centred
around visual data retargeting.
Our formulation is largely domain invariant and only needs minor alterations
to work on different types of visual data, as we demonstrated through its
application to neural radiance fields and even meshes, which is novel for
approaches from the seam carving family that do not operate in continuous
domains. Compared to the existing idea of seam carving, our approach offers
increased flexibility by using continuous deformation fields instead of
discrete decisions for seams, while bringing compatibility _e.g_. for the use
of more complex energy formulations in the future.

## References

  * at English Wikipedia [2007] Newton2 at English Wikipedia.  Broadway tower, 2007.
  * Avidan and Shamir [2007] Shai Avidan and Ariel Shamir.  Seam carving for content-aware image resizing.  _ACM Trans. Graph._ , 26(3):10, 2007.
  * Barron et al. [2021] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5855–5864, 2021.
  * Basha et al. [2013] Tali Dekel Basha, Yael Moses, and Shai Avidan.  Stereo seam carving a geometrically consistent approach.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 35(10):2513–2525, 2013.
  * Berkiten et al. [2017] Sema Berkiten, Maciej Halber, Justin Solomon, Chongyang Ma, Hao Li, and Szymon Rusinkiewicz.  Learning detail transfer based on geometric features.  _Comput. Graph. Forum_ , 36(2):361–373, 2017\.
  * Cai et al. [2022] Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang.  Neural surface reconstruction of dynamic scenes with monocular rgb-d camera.  In _Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Chen, Zhiqin and Zhang, Hao [2019] Chen, Zhiqin and Zhang, Hao.  Learning implicit fields for generative shape modeling.  In _CVPR_ , pages 5939–5948, 2019.
  * Dong et al. [2009] Weiming Dong, Ning Zhou, Jean-Claude Paul, and Xiaopeng Zhang.  Optimized image resizing using seam carving and scaling.  _ACM Trans. Graph._ , 28(5):125, 2009.
  * Dong et al. [2012] Weiming Dong, Guan-Bo Bao, Xiaopeng Zhang, and Jean-Claude Paul.  Fast multi-operator image resizing and evaluation.  _J. Comput. Sci. Technol._ , 27(1):121–134, 2012\.
  * Dong et al. [2014] Weiming Dong, Ning Zhou, Tong-Yee Lee, Fuzhang Wu, Yan Kong, and Xiaopeng Zhang.  Summarization-based image resizing by intelligent object carving.  _IEEE Trans. Vis. Comput. Graph._ , 20(1):1, 2014.
  * Efros and Freeman [2001] Alexei A. Efros and William T. Freeman.  Image quilting for texture synthesis and transfer.  In _Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH_ , pages 341–346. ACM, 2001\.
  * Efros and Leung [1999] Alexei A. Efros and Thomas K. Leung.  Texture synthesis by non-parametric sampling.  In _ICCV_ , pages 1033–1038, 1999.
  * Garbin et al. [2022] Stephan J Garbin, Marek Kowalski, Virginia Estellers, Stanislaw Szymanowicz, Shideh Rezaeifar, Jingjing Shen, Matthew Johnson, and Julien Valentin.  Voltemorph: Realtime, controllable and generalisable animation of volumetric representations.  _arXiv:2208.00949_ , 2022.
  * Gatys et al. [2015] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  Texture synthesis using convolutional neural networks.  In _Neural Information Processing Systems_ , pages 262–270, 2015\.
  * Gu et al. [2023] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Josh Susskind, and Navdeep Jaitly.  Matryoshka diffusion models.  _arXiv preprint arXiv:2310.15111_ , 2023.
  * Henzler et al. [2019] Philipp Henzler, Niloy J Mitra, , and Tobias Ritschel.  Learning a neural 3d texture space from 2d exemplars.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Henzler et al. [2021] Philipp Henzler, Valentin Deschaintre, Niloy J. Mitra, and Tobias Ritschel.  Generative modelling of BRDF textures from flash images.  _ACM Trans. Graph._ , 40(6):284:1–284:13, 2021\.
  * Hertz et al. [2020] Amir Hertz, Rana Hanocka, Raja Giryes, and Daniel Cohen-Or.  Deep geometric texture synthesis.  _ACM Trans. Graph._ , 39(4):108:1–108:11, 2020\.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Huang et al. [2023] Yihua Huang, Yan-Pei Cao, Yu-Kun Lai, Ying Shan, and Lin Gao.  Nerf-texture: Texture synthesis with neural radiance fields.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 43:1–43:10, 2023.
  * Kawar et al. [2023] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6007–6017, 2023.
  * Kingma and Ba [2017] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * Kopf et al. [2007] Johannes Kopf, Chi-Wing Fu, Daniel Cohen-Or, Oliver Deussen, Dani Lischinski, and Tien-Tsin Wong.  Solid texture synthesis from 2d exemplars.  _ACM Transactions on Graphics_ , 26(3):2:1–2:9, 2007.
  * Lai et al. [2005] Yu-Kun Lai, Shi-Min Hu, Xianfeng Gu, and Ralph R. Martin.  Geometric texture synthesis and transfer via geometry images.  In _Proceedings of the Tenth ACM Symposium on Solid and Physical Modeling_ , pages 15–26. ACM, 2005.
  * Liu and Gleicher [2005] Feng Liu and Michael Gleicher.  Automatic image retargeting with fisheye-view warping.  In _Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology_ , pages 153–162. ACM, 2005.
  * Liu and Gleicher [2006] Feng Liu and Michael Gleicher.  Video retargeting: automating pan and scan.  In _Proceedings of the 14th ACM International Conference on Multimedia_ , pages 241–250. ACM, 2006.
  * Luiten et al. [2024] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan.  Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.  In _3DV_ , 2024.
  * Mescheder et al. [2019] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3D reconstruction in function space.  In _CVPR_ , pages 4460–4470, 2019.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Moreira et al. [2022] Thierry Pinheiro Moreira, Marcos Cleison S. Santana, Leandro A. Passos, João Paulo Papa, and Kelton Augusto Pontara da Costa.  An end-to-end approach for seam carving detection using deep neural networks.  In _Pattern Recognition and Image Analysis_ , pages 447–457, 2022\.
  * Müller et al. [2022] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Nam et al. [2021] Seung-Hun Nam, Wonhyuk Ahn, In-Jae Yu, Myung-Joon Kwon, Minseok Son, and Heung-Kyu Lee.  Deep convolutional neural network for identifying seam-carving forgery.  _IEEE Trans. Circuits Syst. Video Technol._ , 31(8):3308–3326, 2021.
  * Nataraj et al. [2021] Lakshmanan Nataraj, Chandrakanth Gudavalli, Tajuddin Manhar Mohammed, Shivkumar Chandrasekaran, and B. S. Manjunath.  Seam carving detection and localization using two-stage deep neural networks.  _CoRR_ , abs/2109.01764, 2021.
  * North [2015] Rockstar North.  Grand theft auto v.  Steam, 2015.
  * Park et al. [2019] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning continuous signed distance functions for shape representation.  In _CVPR_ , pages 165–174, 2019.
  * Park et al. [2021a] Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Steven M. Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In _ICCV_ , pages 5845–5854, 2021a.
  * Park et al. [2021b] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B. Goldman, Ricardo Martin-Brualla, and Steven M. Seitz.  Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields.  _ACM Trans. Graph._ , 40(6):238:1–238:12, 2021b.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4195–4205, 2023.
  * Peng et al. [2022] Yicong Peng, Yichao Yan, Shenqi Liu, Yuhao Cheng, Shanyan Guan, Bowen Pan, Guangtao Zhai, and Xiaokang Yang.  Cagenerf: Cage-based neural radiance fields for generalized 3d deformation and animation.  In _Advances in Neural Information Processing Systems_ , 2022.
  * Pumarola et al. [2021] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-NeRF: Neural Radiance Fields for Dynamic Scenes.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * Richter et al. [2022] Stephan R Richter, Hassan Abu AlHaija, and Vladlen Koltun.  Enhancing photorealism enhancement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(2):1700–1715, 2022.
  * Rubinstein et al. [2008] Michael Rubinstein, Ariel Shamir, and Shai Avidan.  Improved seam carving for video retargeting.  _ACM Trans. Graph._ , 27(3):16, 2008.
  * Rubinstein et al. [2009] Michael Rubinstein, Ariel Shamir, and Shai Avidan.  Multi-operator media retargeting.  _ACM Trans. Graph._ , 28(3):23, 2009.
  * Rubinstein et al. [2010] Michael Rubinstein, Diego Gutierrez, Olga Sorkine, and Ariel Shamir.  A comparative study of image retargeting.  _ACM Transactions on Graphics (Proc. SIGGRAPH ASIA)_ , 29(6):160:1–160:10, 2010.
  * Shaham et al. [2019] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.  Singan: Learning a generative model from a single natural image.  In _ICCV_ , pages 4569–4579, 2019.
  * Song et al. [2019] Eungyeol Song, Minkyu Lee, and Sangyoun Lee.  Carvingnet: Content-guided seam carving using deep convolution neural network.  _IEEE Access_ , 7:284–292, 2019.
  * Sorkine-Hornung and Alexa [2007] Olga Sorkine-Hornung and Marc Alexa.  As-rigid-as-possible surface modeling.  In _Symposium on Geometry Processing_ , 2007.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  In _Advances in Neural Information Processing Systems_ , pages 27171–27183, 2021.
  * Wei and Levoy [2000] Li-Yi Wei and Marc Levoy.  Fast texture synthesis using tree-structured vector quantization.  In _Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH_ , pages 479–488. ACM, 2000\.
  * Wu et al. [2023a] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang.  4d gaussian splatting for real-time dynamic scene rendering.  _arXiv preprint arXiv:2310.08528_ , 2023a.
  * Wu et al. [2010] Huisi Wu, Yu-Shuen Wang, Kun-Chuan Feng, Tien-Tsin Wong, Tong-Yee Lee, and Pheng-Ann Heng.  Resizing by symmetry-summarization.  _ACM Trans. Graph._ , 29(6):159, 2010.
  * Wu and Zheng [2022] Rundi Wu and Changxi Zheng.  Learning to generate 3d shapes from a single example.  _ACM Transactions on Graphics (TOG)_ , 41(6), 2022.
  * Wu et al. [2023b] Rundi Wu, Ruoshi Liu, Carl Vondrick, and Changxi Zheng.  Sin3dm: Learning a diffusion model from a single 3d textured shape.  _CoRR_ , abs/2305.15399, 2023b.
  * Xian et al. [2018] Wenqi Xian, Patsorn Sangkloy, Varun Agrawal, Amit Raj, Jingwan Lu, Chen Fang, Fisher Yu, and James Hays.  Texturegan: Controlling deep image synthesis with texture patches.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Xu and Harada [2022] Tianhan Xu and Tatsuya Harada.  Deforming radiance fields with cages.  In _ECCV_ , pages 159–175, 2022.
  * Yang et al. [2023] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin.  Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction.  _arXiv preprint arXiv:2309.13101_ , 2023.
  * Yuan et al. [2022] Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao.  Nerf-editing: Geometry editing of neural radiance fields.  In _CVPR_ , pages 18332–18343, 2022.

Supplementary Material

##  6 Architecture

Figure 10: The simple architecture we apply for learning the neural image
field, the deformation field(s), energy network and cumulative energy network.
We use positional encoding (yellow), then apply linear layers (cyan) with a
residual connection, LeakyReLu (black arrow), and an output depending on
context (red, with sigmoid for images, otherwise LeakyReLu).

We use simple components for our architecture, with a basic MLP with residual
connection being the core of all our used networks. We use the Adam
optimiser[22] in all instances, without weight or learning rate decay, and the
default learning rate of $0.001$ if not specified otherwise.

###  6.1 Architecture for Images

For the neural field holding the images as described in Sec. 3.2, we apply the
network depicted in Fig. 10, a simple MLP with 192 channels, a residual
connection, and positional encoding that outputs $3$ scalars for the RGB
value. For the deformation and the cumulative gradient network, we use the
same network with 64 channels and a single output scalar value. We use a
learning rate of $0.0001$ for expanding the image. We train the network
learning the image itself and the network learning the cumulative gradient for
$250$ epochs with $100$ iterations each, while we initialise the deformation
network for $50$ epochs with a uniform transformation that we then train on
our loss defined in Sec. 7. Our slim architecture allows optimisation of all
of the image data in one single batch.

###  6.2 Architecture for Neural Radiance Fields

We use a pre-trained, customised iNGP [31] as our learned 3D scene (both
vanilla [29] and mip-NeRF [3] worked just as well in our early tries), then
use the same setup as for images. We train initial deformation, energy, and
cumulative energy for 5000 iterations each, always using $10000$ samples. We
optimise the deformation field for $10000$ random points from the surface and
$10000$ random points from the boundary for 50 epochs with 100 iterations
each, and always choose the network parameters from the epoch with the best
average loss.

##  7 Exact Loss Formulations

###  7.1 Loss Formulation for Images

Due to out light weight network we use for our deformation field, we can apply
our loss terms directly to all points $p$ in the output domain (_i.e_. the
shrunken image), training one single batch for an image.

For $L_{e}$, we define:

$\displaystyle
L_{e}=\frac{|E_{cnet}(p)-E_{cnet}(p_{\epsilon})|\cdot|D(p)-D(p_{\epsilon})|}{\epsilon}$
(12)

where $E_{cnet}$ is the MLP for cumulative energy and $p_{\epsilon}$ are all
pixels $p$ moved by one pixel in direction $v$. For shearing, for energy net
$E$ and $p_{\epsilon}^{\perp}$ being all points $p$ moved by one pixel in the
direction orthogonal to $v$, we define:

$\displaystyle L_{s}$
$\displaystyle=\frac{E(p)\cdot|D(p)-D(p_{\epsilon}^{\perp})|}{\epsilon}$ (13)

For the boundary, we build our loss around the pixels on the one end of the
image, $p_{l}$, and the other, $p_{r}$, regularising their deformation to be
$0$ at the one, and $1.0-\alpha$ at the other end:

$L_{b}=mean(|D(p_{l})|)+mean(|D(p_{l})-(1-\alpha)|)$ (14)

For monotonicity in images, we use:

$L_{m}=\frac{mean(max(0,D(p)-D(p_{\epsilon})))}{\epsilon}$ (15)

As weights for our loss terms, we use
$\lambda_{e}=10000,~{}\lambda_{s}=250~{}\lambda_{b}=\lambda_{m}=10000$. For
the cumulative energy net, we accumulate energy over all pixels, then train
the MLP to minimise mean squared error between input (2D position) and output
(cumulative energy for that position) for 10000 iterations, always for all
pixels.

###  7.2 Loss Formulation for Neural Radiance Fields

We first define a mixture of surface points and uniform random points
$p^{\prime}$ in source space we run our optimisation on, then finding those
points $p$ in target space that are deformed to become $p^{\prime}$:

$p=U(p^{\prime})$ (16)

For $L_{e}$, we define:

$\displaystyle
L_{e}=\frac{mean(|E_{cnet}(p)-E_{cnet}(p_{\epsilon})|\cdot|D(p)-D(p_{\epsilon})|)}{\epsilon}$
(17)

where $E_{cnet}$ is the MLP for cumulative energy and $p_{\epsilon}$ are our
points $p$ moved by an offset in direction $v$. For shearing, for energy net
$E$ and $p_{\epsilon^{1}}^{\perp}$, $p_{\epsilon^{2}}^{\perp}$ being all
points $p$ moved by an offset in the directions orthogonal to $v$, we define:

$\displaystyle L_{s}$
$\displaystyle=\frac{E(p)\cdot|D(p)-D(p_{\epsilon^{1}}^{\perp})|}{\epsilon}$
(18)
$\displaystyle+\frac{E(p)\cdot|D(p)-D(p_{\epsilon^{2}}^{\perp})|}{\epsilon}$

For the boundary, we build our loss around the pixels on the one end of the
image, $p_{l}$, and the other, $p_{r}$, regularising their deformation to be
$0$ / $1.0-\alpha$:

$L_{b}=mean(|D(p_{l})|+mean(|D(p_{l})-(1-\alpha)|)$ (19)

For monotonicity in images, we use:

$L_{m}=\frac{mean(max(0,D(p)-D(p_{\epsilon})))}{\epsilon}$ (20)

As weights for our loss terms, we use
$\lambda_{e}=10,~{}\lambda_{s}=0.1~{}\lambda_{b}=100,=\lambda_{m}=1$. After
every iteration, we also update $U$ with all points $p$, _i.e_. minimise
$(D(U(p))-p)^{2}$ for one step. For the cumulative energy net, we accumulate
energy by shooting random rays along the deformation direction with $100$
uniformly distributed samples. We gather and accumulate energy from $E_{net}$,
then train the cumulative energy MLP to minimise mean squared error between
input (3D position) and output (cumulative energy for that position on that
ray) for $10000$ iterations, always for $10000$ points.

##  8 Expansion

Expansion of visual data follows the same idea as for seam carving[2]: Instead
of compressing (in seam carving: removing) low energy parts, we stretch (in
seam carving: double) them. Examples can be seen in Sec. 10. To expand data,
_e.g_. to increase its width to $150$ percent, we can use the exact same
losses as for retargeting to a smaller size, but with one additional loss term
and a minor modification:
We require a limit on the change in the deformation to avoid only duplicating
a single seam, meaning to punish any change in the deformation that is too
large, hence would repeat the same part of the image over and over again. Seam
carving prevents this for image expansion by only doubling a seam once. As a
new loss term, this gives us:

$L_{cap}=mean(max(0,\frac{(p-p_{\epsilon})}{\epsilon}-1)))$ (21)

Preventing to more than double an input. In addition, we modify $L_{e}$ to not
punish any information content skipped between two points (as we no longer
skip over anything), but instead simply punish the energy directly, _i.e_.
replacing $\hat{E}$ from Eq. 6 with $E$ applied to only one point.

##  9 Editing

For the editing procedures describe in Sec. 3.5, we can perform the same
concept for every domain. For this, we always retain the same image size.

#### Removal

To remove an object, we punish every point that maps to the content that we
want to remove, _i.e_. take all points of the input, deform them, and look up
if the deformed point lies within the marked area. For continuity, we do this
by training another network $net_{mask}$ to learn a mask, _i.e_. $0$ for
regions outside, $1$ for regions inside the area to remove. We then simply add
$mean(net_{mask}(p))$ for all points $p$ to the loss function, punishing any
deformation that maps inside the target area. To allow both sides of the
removed content to fill in the space, we disable the monotonicity loss
$L_{m}$. The remaining loss terms then keep the rest of the image in place.

#### Moving an object

To move an object, we first remove the monotonicity loss $L_{m}$ to allow more
complex deformations. As this disables protection against repetition, we then
actively punish our target object occuring anywhere else, _i.e_. using a
network $net_{mask}$ containing a binary mask of the object to move, then add
$net_{mask}(p)$ for all points $p$ in the output to the loss function except
for the target coordinates of the object. We then add a term to the loss that
enforces the exact offset value that would place the target at the right
location for all parts of it. Lastly, to avoid any other unwanted edits and
make the optimisation more stable, we add a last loss term scaled by $0.01$
that punishes the absolute mean of deformation, _i.e_. punishes applying any
deformation at all. While this avoids unnecessary deformations, it is
neglectable compared to the additional loss caused by only the part relevant
for the deformation.
To move an object along multiple axis, we decompose the movement to the target
position into two axis, then apply our approach for each direction.

##  10 Additional Results

(see following pages)

Ours | Seam carving | Input | Seam carving | Ours
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
Ours | Seam carving | Input | Seam carving | Ours
|  |  |  |
|  |  |  |
|  |  |  |
Ours | Seam carving | Input | Seam carving | Ours
|  |  |  |
|  |  |  |
|  |  |  |
Ours | Seam carving | Input | Seam carving | Ours
|  |  |  |
Table 4: More examples of our deformation approach on images, all from the RetargetMe dataset [44]. Ours to 50 % | Input scene | Ours to 150 %
---|---|---
|  |
|  |
|  |
Ours to 50 % | Input scene | Ours to 150 %
|  |
|  |
|  |
Table 5: More examples of our deformation approach on 3D NeRF scenes.