# Parametric Reshaping of Portraits in Videos

Xiangjun Tang  State Key Lab of CAD&CG, Zhejiang University<EMAIL_ADDRESS>,
Wenxin Sun  State Key Lab of CAD&CG, Zhejiang University<EMAIL_ADDRESS>,
Yong-Liang Yang  University of Bath<EMAIL_ADDRESS>and  Xiaogang Jin
State Key Lab of CAD&CG, Zhejiang University<EMAIL_ADDRESS>
(2021)

###### Abstract.

Sharing short personalized videos to various social media networks has become
quite popular in recent years. This raises the need for digital retouching of
portraits in videos. However, applying portrait image editing directly on
portrait video frames cannot generate smooth and stable video sequences. To
this end, we present a robust and easy-to-use parametric method to reshape the
portrait in a video to produce smooth retouched results. Given an input
portrait video, our method consists of two main stages: stabilized face
reconstruction, and continuous video reshaping. In the first stage, we start
by estimating face rigid pose transformations across video frames. Then we
jointly optimize multiple frames to reconstruct an accurate face identity,
followed by recovering face expressions over the entire video. In the second
stage, we first reshape the reconstructed 3D face using a parametric reshaping
model reflecting the weight change of the face, and then utilize the reshaped
3D face to guide the warping of video frames. We develop a novel signed
distance function based dense mapping method for the warping between face
contours before and after reshaping, resulting in stable warped video frames
with minimum distortions. In addition, we use the 3D structure of the face to
correct the dense mapping to achieve temporal consistency. We generate the
final result by minimizing the background distortion through optimizing a
content-aware warping mesh. Extensive experiments show that our method is able
to create visually pleasing results by adjusting a simple reshaping parameter,
which facilitates portrait video editing for social media and visual effects.

video portrait editing, face reconstruction, face reshaping

††journalyear: 2021††copyright: acmcopyright††conference: Proceedings of the
29th ACM International Conference on Multimedia; October 20–24, 2021; Virtual
Event, China††booktitle: Proceedings of the 29th ACM International Conference
on Multimedia (MM ’21), October 20–24, 2021, Virtual Event, China††price:
15.00††doi: 10.1145/3474085.3475334††isbn: 978-1-4503-8651-7/21/10††ccs:
Computing methodologies Mesh geometry models††ccs: Computing methodologies
Image processing

Figure 1. Our parametric reshaping method allows users to reshape the portrait
in a video footage easily by simply adjusting a reshaping parameter. Given an
input portrait video sequence (second row), our approach can reshape the
portrait in the video with weight-change such that the face appears thinner
(the first row) or rounder (the third row), respectively.

##  1\. Introduction

Due to the recent development of social networks and personalized media, more
and more people have become active to share their own photos and videos to
others using mobile phones. Portrait editing techniques are often utilized to
create special portrait effects through face stretch, exaggeration,
beautification, etc., such that the resultant portraits are more appealing.
Research attention has also been focused on different face retouching methods
to edit face colors, textures, styles, and even shapes for portrait images
(Leyvand et al., 2008) (Shih et al., 2014) (Zhao et al., 2018) (Xiao et al.,
2020). Compared with portrait images, portrait video editing has been much
less explored. Although video can be directly processed by applying image
editing to each individual video frame, this can easily lead to various
artifacts due to the lack of editing consistency and temporal coherence across
neighboring video frames.

Our aim is to generate high-quality portrait video reshaping results (see
Figure 1) by editing the overall shape of the portrait faces according to
natural face deformation in real world. This can be used for applications such
as shapely face generation for beatification, and face exaggeration for visual
effects. Compared with portrait image reshaping (Zhao et al., 2018)(Xiao et
al., 2020), we also employ a 3D face reconstruction based approach to guide 2D
portrait editing with faithful face deformation in 3D. However, for plausible
video reshaping, a key difference in our setting is that our new problem
requires the consistency and coherency of the reshaped portraits over the
entire video. This new requirement poses two challenges in practice, i.e., how
to achieve consistency and coherency not only on the reconstructed 3D faces,
but also on the reshaped video frames.

In this paper, we present a parametric reshaping method that can generate
quality reshaped portrait videos. Given an input video, our method contains
two main stages to address the above two challenges. The first stage mainly
focuses on consistent and coherent face reconstruction. Based on a state-of-
the-art face parametric model, we employ a multi-phase optimization that
robustly estimates pose transformations, face identity parameters, and face
expression parameters, respectively. Our method particularly concerns the face
contour’s stability in face tracking to avoid artifacts after reshaping. To
estimate a consistent face identity across all frames, we jointly optimize the
representative frames rather than all frames to reduce computational cost. The
energy terms in the optimization are carefully chosen through extensive
ablation study by taking both spatial-temporal coherence and computational
efficiency into consideration. The second stage attempts to generate reshaped
video frames without visual artifacts. Guided by reshaped faces in 3D, we
employ a content-aware image warping method to deform each frame. To avoid
warping artifacts caused by face occlusions after reshaping, we use a SDF to
construct the dense 2D mapping of the face contour before and after reshaping.
In addition, we use the 3D structure of the face to correct the dense mapping.
The dense 2D mapping is then transferred to a sparse set of grid points to
warp video frames.

We evaluate our work on a variety of videos with different portrait
characteristics including gender, face color, hair style, pose variations,
etc. The results in the paper and supplemental video both demonstrate the
effectiveness and robustness of our work. Extensive comparisons with different
design choices also verify the superiority of our method.

The major contributions of this work are summarized as the following:

  * •

We present the first robust parametric reshaping approach for high-quality
reshaped portrait video generation.

  * •

We propose an efficient and stable 3D face reconstruction method using a
multi-phase optimization strategy with a refined dense flow energy.

  * •

We propose an effective correspondence estimation method based on signed
distance function and 3D information to deform a portrait video without
artifacts.

##  2\.  Related work

In this section, we discuss previous approaches that are relevant to our
method. As there is no prior work on parametric reshaping of portraits in
videos, we organize this section according to the two stages of our method by
reviewing related works on video-based reconstruction and video deformation,
respectively.

###  2.1. Video based face reconstruction

Morphable Model. Monocular 3D face reconstruction is an ill-posed optimization
problem, which requires priors of face identity and expression. Blanz and
Vetter (1999) proposed the 3D Morphable Model (3DMM) using principal component
analysis (PCA) on 3D scans, which can be used as identity priors for
reconstruction. Blendshapes Model (Lewis et al., 2014) offered expression
priors using faces with the same topology but different expressions. Over the
past years, a variety of works used both linear models and their extensions
for face reconstruction (Egger et al., 2020). Surrey Face Model proposed by
Huber et al. (2016) is a multi-resolution 3DMM that contains different mesh
resolution levels. Booth et al. (2017) extended 3DMM by combining a
statistical model of face shape with an “in-the-wild” texture model to reduce
illumination parameters in optimization. Rapid progress has been made for
improving reconstruction speed, accuracy, and ease of use even in
unconstrained conditions (Zollhöfer et al., 2018). However, the type and
amount of training data constrained the performance of linear 3DMM. Tran and
Liu (2018) proposed to learn a nonlinear 3DMM from face images rather than 3D
face scans to produce a more extensive database. Tran et al. (2019) further
improved the nonlinear 3DMM in both learning objective and network
architecture to achieve high-fidelity face reconstruction results. Based on a
set of 4,000 high resolution facial scans, Li et al. (2020) proposed a deep-
learning based morphable face model.

Video based reconstruction. Although portrait video contains abundant frames,
the joint optimization of face pose, identity, and expression is still
challenging. Simply adding more constraints to the optimization is hard to
achieve satisfactory results. Thies et al. (2016) used model-based non-rigid
bundle adjustment over keyframes with different head poses. Cao et al. (2018)
proposed the on-the-fly method for face tracking using a dynamic rigidity
prior learned from realistic datasets. This method can achieve plausible
results when the landmarks are stable and mostly visible. However, the
reconstruction of the current frame depends on the result of the previous
frame. If the landmarks are not accurate enough or very different between
frames, it is still challenging to achieve accurate and continuous results.

Figure 2. Given an image sequence, we first estimate the rigid pose of the
face for each frame, which is crucial for computing a consistent identity
across all frames. Then, we leverage an optimization method by using the most
representative frames jointly to estimate the whole sequence’s consistent
identity parameters. Next, we estimate each frame’s expression parameters.
Given a reshaping parameter, a linear regression method is employed to
generate the reshaped 3D face model sequence by combining the identity and the
expression parameters. After that, we employ an SDF based method to construct
the 2D dense mapping of the face contour before and after reshaping, and
perform a content-aware warping optimization to deform the video in order to
get our final result.

###  2.2. Portrait video deformation

For reducing image distortion after face editing, content-aware image warping
becomes a powerful tool in a wide range of editing applications (Kaufmann et
al., 2013) (Shih et al., 2019) (Xiao et al., 2020). However, it is challenging
to generate continuous and stable deformations across all video frames because
the mappings from source to target of consecutive frames are usually
inconsistent. Chen et al. (2013) noticed the importance of a consistent
blending boundary and presented a video blending approach to merge the
gradient of source and target videos. Thies et al. (2015) transferred
expressions by mapping the expression parameters from the source to the target
without changing other parameters, such that the target identity, rigid head
motion, and scene lighting can be preserved. Moreover, portrait video
attributes such as head pose, facial expression, scene illumination can be
manipulated by GAN-based methods (Tewari et al., 2020) (Nagano et al., 2019)
(Zeng et al., 2020) (Chen et al., 2020).

##  3\. Overview

This work proposes a novel method that parametrically reshapes a portrait
video, making the resultant portrait sequence realistic and stable. This
requires robust extraction of portrait shapes from the video and consistent
deformation of video frames to reshape the portraits, which are all addressed
by our method.

Figure 2 shows the pipeline of our method. Given a portrait image sequence,
our method consists of two main stages. In the first stage (Section 4), we
utilize a video-based face reconstruction approach to faithfully reconstruct
high-quality face identity with stable poses and expressions. We first
estimate the head pose in each frame, which is crucial to estimate a
consistent identity across frames. Then we find the $k$ frames that best
represent the face identity and jointly optimize a consistent face identity.
Finally we estimate the face expression of each frame to achieve the whole 3D
face sequence. In the second stage (Section 5), we generate reshaped faces in
3D based on the reconstruction results, then leverage deformed faces in 3D to
guide the reshaped portrait video generation in 2D. We first reshape the
reconstructed 3D neutral face model and combine it with facial expression and
pose for each video frame. We then employ an SDF based method to construct a
2D dense mapping of the face contour before and after reshaping. A content-
aware warping optimization is used to deform portrait frames according to
reshaped faces in 3D, resulting in the final reshaped portrait video.

##  4\. Video based face reconstruction

In this section, we first describe the parametric face model and the
objectives that our optimization is based on. Then we elaborate how we
optimize face pose, identity, and expression step by step while taking into
account robustness and efficiency.

###  4.1. Parametric Face Model and Objectives

The parametric face model can be represented via a linear combination of
identity basis vectors and expression basis vectors:

(1)
$\mathbb{s}=\mathbb{a}_{s}+\sum_{i=1}^{m_{s}}\alpha_{i}\mathbb{b}_{i}^{s}+\sum_{i=1}^{m_{e}}\beta_{i}\mathbb{b}_{i}^{e},$

where $\mathbb{a}_{s}\in R^{3n}$ is the mean face identity ($n$ is the number
of vertices of the face), $\mathbb{b}_{i}^{s}\in R^{3n}$ is one of the $m_{s}$
identity basis vectors, and $\mathbb{b}_{i}^{e}\in R^{3n}$ is one of the
$m_{e}$ expression basis vectors.
$\bm{\alpha}=[\alpha_{1},...,\alpha_{m_{s}}]$,
$\bm{\beta}=[\beta_{1},...,\beta_{m_{e}}]$ are the identity coefficients and
expression coefficients, respectively.

The parametric face model proposed by Huber et al. (2016) consists of
$m_{s}=63$ facial identity coefficients and $m_{e}=6$ expression coefficients.
Although this model does not express ample expressions, it is capable of
representing common portrait faces.

We denote the projection operator by $\Pi:R^{3}\to R^{2}$ , which maps the
$k$-th mesh vertex to image coordinate $\mathbb{p}_{i}\in R^{2}$ as:

(2) $\mathbb{p}_{i}=\Pi(\mathbb{r},\mathbb{t},\bm{\alpha},\bm{\beta})_{k},$

where $\mathbb{r}\in R^{3}$ and $\mathbb{t}\in R^{3}$ are the rotation and
translation parameters of the face model, respectively. Now we present our
objective energy terms based on the aforementioned parameters which will be
used in the latter optimization.

Landmark energy. We denote a set of 2D facial landmark indices as
$\mathcal{L}=\\{L_{1},...,L_{NL}\\}$ ($NL$ is the number of landmarks). The
first objective aims to match all 3D landmarks to their corresponding 2D
landmarks after projection (assume 2D and 3D landmark indices are the same for
simplicity):

(3)
$E_{land}=||\Pi(\mathbb{r},\mathbb{t},\bm{\alpha},\bm{\beta})_{i}-\mathbb{p}_{i}||^{2}_{\forall
i\in\mathcal{L}}.$

Contour energy. The part of face away from the camera is likely to have offset
landmarks in 2D due to the incomplete face region induced by occlusion (see an
example in Figure 8(a) near the eye). Therefore we also define a contour
energy to match face boundary between 3D and 2D. Note that the boundary of the
projected 3D face model will change when the pose changes, and thus it is
impossible to obtain the accurate boundary. However, as our method estimates
face rigid pose first (see Section 4.2), we can utilize the boundary of the
approximate 3D face model obtained from the rigid pose estimation step to
constrain the alignment.

Then the contour energy can be expressed as follows:

(4)
$E_{contour}=||\Pi(\mathbb{r},\mathbb{t},\bm{\alpha},\bm{\beta})_{k}-\mathbb{p}_{i}||^{2}_{\forall
i\in\mathcal{L}_{b}},$

where $\mathcal{L}_{b}$ is the set of indices of contour landmarks, and $k$ is
the corresponding vertex’s index on the 3D face boundary.

Alignment energy. By now, we can define the alignment energy as the linear
combination of landmark energy and contour energy:

(5) $E_{align}=E_{land}+\sigma E_{contour}.$

Empirically, we set $\sigma=0.5$.

Prior energy. A regularization term is also defined to constrain the
reconstructed face to be regular in face space:

(6) $E_{prior}=w_{prior}(||\bm{\alpha}||_{2}+||\bm{\beta}||_{2}),$

where $w_{prior}$ is the weight of the regularization. We empirically set
$w_{prior}$ as 0.4.

Temporal coherence energy. Typically the temporal coherence energy reduces the
parameter difference between previous frame and current frame. The expression
coherence term can be defined as:

(7) $E_{temp}^{expr}=||\bm{\beta}-\bm{\beta}^{\prime}||^{2}.$

Similarly, the pose coherence term can be defined as:

(8)
$E_{temp}^{pose}=||\mathbb{r}-\mathbb{r}^{\prime}||^{2}+\gamma||\mathbb{t}-\mathbb{t}^{\prime}||^{2},$

where $\gamma$ is the parameter to balance the effect of translation and
rotation. In our experiment, we set $\gamma$ as the reciprocal of face length.
Then the overall temporal coherence energy is defined as:

(9) $E_{temp}=E_{temp}^{pose}+\sigma E_{temp}^{expr},$

where we set $\sigma=2$ in our experiments.

Dense flow energy. Inspired by Cao et al. (2018), we use optical flow to
construct dense correspondences to overcome landmark detection errors. With
pose parameter $\mathbb{t}^{\prime},\mathbb{r}^{\prime}$ and expression
coefficients $\bm{\beta}^{\prime}$ from the previous frame, the dense flow
energy is defined as:

(10)
$E_{optic}=||\Pi(\mathbb{r},\mathbb{t},\bm{\alpha},\bm{\beta})_{i}-\Pi(\mathbb{r}^{\prime},\mathbb{t}^{\prime},\bm{\alpha},\bm{\beta}^{\prime})_{i}-\mathbb{U}_{i}||^{2}_{\forall{i\in\mathcal{L}_{b}}},$

where $\mathbb{U}_{i}$ is the motion vector. As expression change is not our
main concern, we define the energy term only on face contour $\mathcal{L}_{b}$
rather than the full face, which also reduces computational cost without
affecting visual quality. However, artifacts occur when the optical flow map
is inaccurate on face contour due to occlusions caused by other objects (see
Figure 3 for a failure case). The change of optical flow on the face contour
is mainly induced by camera movement and face pose transformations. Besides,
the occluded regions may have different values on the flow map. Employing a
low-frequency filter to eliminate the outliers can preserve the correct part.

(a) Input portait image

(b) Using optical flow only

(c) Our result

Figure 3. Inaccurate optical flow map due to occluded face (a) may lead to
failed results (b) by simply applying Eqn. 10. (c) shows our correct result.

###  4.2. Stabilized Face Tracking

Reconstructing face shape on the fly is convenient for real-time applications.
However, it is hard to ensure that the identity reconstructed from the first
frame is the same as that from the last frame. In contrast, joint optimization
for all frames can achieve a consistent face identity but is too costly. As
such, we first estimate the face pose for each frame, and then choose the
consecutive $k$ frames which best represent the face identity coefficients.
After that, we apply joint optimization for these $k$ frames to achieve the
accurate face identity. Finally we estimate the expression coefficients of
each frame to obtain the final reconstruction result. The advantage of
approximating face pose first is that face poses provide stable projected face
contours and face regions in 2D, which enables us to employ the contour energy
and revised dense flow energy defined in Section 4.1 for identity and
expression reconstruction.

####  4.2.1. Rigid Pose Estimation

Based on the objectives defined in Section 4.1, the pose estimation energy is
defined as:

(11)
$E_{pose}=\lambda_{land}E_{land}+\lambda_{temp}^{pose}E_{temp}^{pose}+\lambda_{optic}E_{optic},$

where we empirically set $\lambda_{land}=0.6$, $\lambda_{temp}^{pose}=0.9$,
and $\lambda_{optic}=0.5$. At this stage, we fix the identity parameters and
expression parameters. Our extensive experiments show that such a strategy
enables us to achieve more stable final results as the energy terms involving
all variables may conflict with each other during the optimization.

####  4.2.2. Identity Estimation

It is hard to estimate the full shape of a face if only partial face is
visible in the image. After pose estimation, we choose the consecutive $k$
$(k\leq 10)$ frames whose faces are facing to the camera, and apply bundle
optimization (Triggs et al., 1999) to these frames.

Based on the above and the objectives defined in Section 4.1, the identity
estimation energy is defined as:

(12) $\displaystyle E_{identity}=$
$\displaystyle\sum_{i}^{k+i}[\lambda_{align}(E_{align})_{i}+\lambda_{optic}(E_{optic})_{i}+$
$\displaystyle\lambda_{temp}(E_{temp})_{i}+(E_{prior})_{i}].$

To estimate the face identity as accurate as possible, we intentionally
increase the weight of the landmark energy term $E_{align}$. In our
experiments, we set $\lambda_{align}=0.7$, $\lambda_{optic}=0.1$ and
$\lambda_{temp}=0.2$.

####  4.2.3. Expression Estimation

After shape estimation, we solve for face expressions by fixing shape
coefficients. Similar to $E_{shape}$, we can define the expression energy as
follows:

(13) $\displaystyle E_{expr}$
$\displaystyle=\lambda_{align}(E_{align})_{i}+\lambda_{optic}(E_{optic})_{i}+$
$\displaystyle\lambda_{temp}(E_{temp})_{i}+(E_{prior})_{i},$

where we empirically set $\lambda_{align}=0.9$, $\lambda_{optic}=0.5$ and
$\lambda_{temp}=0.5$. However, we apply it for all video frames.

##  5\. Reshaping

In this section, we reshape the 3D faces reconstructed in the last section and
use them to guide the generation of the reshaped portrait video. More
specifically, given a reshaping parameter, we first use a linear regression
model to generate the reshaped faces in each frame (Section 5.1). Then we
employ image warping to deform each frame respecting the corresponding
reshaped face (Section 5.2).

###  5.1. 3D Face Reshaping

Xiao et al. (2020) proposed a reshaping model to generate reshaped portrait
images based on an estimated adjusting parameter. We extend this method from
reshaping one monocular image to reshaping the whole image sequence. The
reshaping model deforms the whole face by utilizing a scalar parameter
$\delta$ as input. We denote the reshape operator as $f(\bm{X};\delta)$.

For a sequence of reconstructed 3D faces
$\bm{X}_{i}(\bm{\alpha},\bm{\beta}^{(i)})$, where $\bm{\beta}^{(i)}$ is the
expression coefficients vector of the $i$-th frame, the reshaped 3D face model
$\bm{X}^{*}_{i}$ is defined as the linear combination of the neutral reshaped
face model and the expression coefficients $\bm{\beta}^{(i)}$:

(14)
$\bm{X}_{i}^{*}=f(\bm{X}(\bm{\alpha},\bm{\beta}_{0});\delta)+\sum_{k=1}^{m_{e}}\beta^{(i)}_{k}\mathbb{b}_{k}^{e},$

where $\bm{\beta}_{0}$ is the expression coefficients vector of the neutral
face.

###  5.2. Consistent Video Deformation

Using the reshaped 3D faces as guidance, we warp each portrait image to
generate the face with the target shape in 2D while avoiding visible
artifacts. We first place a uniform grid $M_{u}=\\{\mathbb{u}_{i}\\}$ over the
image as a warping proxy (see Figure 5 (a)), where $\mathbb{u}_{i}$ denotes
the 2D coordinates of the grid point on the image. Then we propose a new
method to find a set of control points $\\{\mathbb{u}_{c}\\}\in M_{u}$, which
drive the image warping induced by the face deformation. Finally, we employ a
least-square optimization to warp all the other grid points by minimizing the
distortion of the entire grid.

Control points selection. It is straightforward to select control points
directly on the 3D face model. Jain et al. (2010) selected face mesh vertices
and used their 2D projections as control points. However, some control points
(mesh vertices) may be occluded (or may emerge) after face deformation and
projection, which can significantly affect the warped face shape on the image
and cause severe artifacts. Xiao et al. (2020) utilized the closest grid
points to the 2D face contour as control points. While their method works well
for a single portrait, it ignores the occluded control points especially when
enlarging a face. Therefore, such a method cannot ensure that the control
points are still at the face contour after reshaping, which leads to
noticeable inconsistent and incoherent video frames after warping.

As the original video is already continuous with the right content, after face
reshaping, the control points need to be stabilized at the location that
expresses the same semantic information. Hence the problem of selecting
control points amounts to find a mapping between face shape and control points
before and after reshaping.

SDF based selection. We propose a SDF based method to establish consistent
mapping of the image contour points before and after reshaping, and then
transfer the mapping to control points. The benefit of stabilized face
reconstruction is that the face contour in 3D is always complete, and its 2D
projection gradually changes, which can be leveraged to achieve consistent and
coherent control points selection. Note that unlike in 3D, semantically
accurate face contour mapping in 2D does not always exist due to possible
occlusions after reshaping and projection (e.g., face boundary partly blocked
by nose but not as before), thus we employ an approximate mapping in practice
as follows.

We first choose all face contour pixels of the original frame and map them to
the reshaped face in 2D. We find that dense mapping of all contour pixels can
further reduce inconsistency compared to sparse mapping of a subset of pixels
(see comparison in Figure 10 and discussion in Section 6). We use the
reconstructed 3D face model to guide the dense mapping. For each contour
pixel, we first unproject it to the reconstructed 3D face model, then reshape
the face model and project it back to the image to get its mapped pixel. This
process cannot achieve consistent contour of 3D model after reshaping (see
Figure 4 (b)). Therefore, we revise the incorrect mapping (by checking if the
mapped pixel lies on the reshaped face contour) based on the SDF of the
original 2D face contour (see pink contour in Figure 4 (c)). More
specifically, for such a contour pixel, we move it along the gradient of the
SDF until it meets the mapped pixel on the reshaped face contour in 2D
(boundary of the projected 3D face after reshaping). This method still has
problems in extreme cases where the mapping incorrectly maps nose points to
cheeks points (see the boxed-out area both in Figure 4 (a) and (c)). For such
cases, we remove the incorrect mappings according to the 3D structure
information of each points (see Figure 4 (d)).

(a) Before reshaping

(b) After reshaping

(c) Without 3D information

(d) Our mapping

Figure 4. The white points in (a) are the unprojected 3D contour points before
reshaping. The pink points in (b) are the 3D vertices corresponding to white
points in (a) after reshaping, which causes inconsistent mapping. Figure (b)
shows two kinds of inconsistency caused by occlusions after reshaping: 1)
between nose and cheek, and 2) between cheeks. The pink points in (c)
represent the corresponding contour points after reshaping without considering
occlusion, which has established wrong mappings compared with (a). (d) is the
result of our mapping method where the white contour is the same with (a) and
the pink contour is the reshaped 3D model contour. Note that the wrong mapping
points are successfully removed here.

(a) Original grid

(b) Grid after MLS

(c) Grid after optimization

Figure 5. Grid deformation through MLS and optimization. (a) The original
uniform grid $M_{u}$. (b) The grid after MLS based reshaping. (c) The
optimized grid while fixing the control points.

Warping. Based on the dense contour mapping, we select grid points
$\\{\mathbb{u}_{c}\\}$ which are closest to the face contour. Then we employ
moving least squares (MLS) deformation (Schaefer et al., 2006) to find the
target position of $\\{\mathbb{u}_{c}\\}$ guided by the dense mapping. After
that, we employ a least-square optimization to other grid points while fixing
the control points to minimize the overall distortion (see Figure 5). We adds
the linear bending term $E_{l}$ and regularization term $E_{r}$ proposed by
Shih et al. (2019):

(15) $E_{l}=\sum_{i}\sum_{j\in
N(i)}||(\mathbb{v}_{i}-\mathbb{v}_{j})\times\mathbb{e}_{ij}||^{2}_{2},$ (16)
$E_{r}=\sum_{i}\sum_{j\in N(i)}||\mathbb{v}_{i}-\mathbb{v}_{j}||^{2}_{2},$

where $\mathbb{v_{i}}$ is the $i$-th key point of grid $M_{u}$, $N(i)$ is the
set of 4-way adjacent points of key point $\mathbb{v}_{i}$, and $\bm{e}_{ij}$
is the unit vector along the direction $\mathbb{v}_{i}-\mathbb{v}_{j}$.

We only select the area surrounding the face as the target area for
optimization. In order to make the optimized area blend perfectly with the
non-optimized area, we fix the boundary grid points instead of adding a grid
border term as in (Shih et al., 2019) to adjust the distortion in the
boundary. The final energy function is:

(17) $E=w_{l}E_{l}+w_{r}E_{r}.$

We empirically set the weights as $w_{l}=1$, $w_{r}=0.8$.

##  6\. Evaluation

In this section, we extensively evaluate the proposed method, including
testing on various examples, comparing with image reshaping based method,
validating our method design choices with baselines, and presenting the
performance and implementation details.

###  6.1. Results

We test our method on a variety of portrait videos with different gender,
hairstyle, skin color, etc. Figure 1 shows some sampled video frames generated
by our method. It can be seen that our method can successfully reshape
portraits without introducing visual artifacts. By changing the parameter
$\delta$, we can continuously adjust the portrait shape (either thinner or
rounder) for potential face retouching and exaggeration applications. More
examples and full video sequences can be found in the supplemental video.

(a) Initial frame

(b) Xiao et al. (2020)

(c) Our method

Figure 6. Comparison with the baseline method. Given a frame from a portrait
video (a), Xiao et al. (2020)’s reshaping method produces artifacts (b) near
the tip of the nose because the nose occludes the side face, while our method
can still generate satisfactory results (c) with the same reshaping parameter.

###  6.2. Comparison

As there is no prior work on parametric reshaping of portraits in videos, we
take the parametric reshaping method of portrait images (Xiao et al., 2020) as
the baseline for comparison. Given a portrait video footage, for a frame shown
in Figure 6(a), as the nose occludes the side face of the portrait, the method
of Xiao et al. (2020) will produce obvious artifacts (see Figure 6(b)) near
the tip of the nose because of their sparse mapping, which is successfully
solved using our dense mapping method (see Figure 6(c)) with the same
reshaping parameter. Moreover, their method cannot generate smooth and
coherent reshaping portrait video results. Please refer to the accompanying
video for side-by-side comparison. The results show that our approach can
robustly produce coherent reshaped portrait videos while the image-based
method can easily lead to noticeable flickering artifacts.

Figure 7.  The results of using the naive dense flow energy defined on face
contours (top row). Results by using our dense flow energy (bottom row). The
green line connects the chin bottom of the first and third images. By
referring to the green line, we can see that the face models in the second-row
have jittering effects (as shown by the gap between the green line and the
chin bottom).

###  6.3. Ablation studies

We also verify the design choices of our method by validating its two main
stages, i.e., face reconstruction and video reshaping, respectively. We show
the effectiveness of our design choices by comparing them with other
baselines.

(a) Input portrait image

(b) Without constraint

(c) Our method

Figure 8. Comparison between the method without constraining face contours (b)
and ours (c). White points in (a) are detected as landmarks having offset to
the left near the eye. The method without contour constraints results in a
worse alignment of face contour (see gap in-between 2D and 3D).

####  6.3.1. Face reconstruction

We first evaluate the effectiveness of our new dense flow energy. We compare
our result to the result without using dense flow energy, and the result using
dense flow energy but defined on the entire face.

In most cases, using dense flow energy helps to produce visually pleasing
results. The stability difference between using dense flow energy or not is
better to be seen in video sequence. It is worth noting that the dense flow
energy can effectively suppress the effects of jittering landmarks near the
face contour, but cannot eliminate them completely. If the landmarks
themselves are jittering, the estimated transformations cannot be very smooth.
Figure 7 shows the problematic case without discarding vertices inside the
wrong regions of the flow map. The handshakes near the face cause wrong motion
directions in the flow map, resulting in instability head transformations.

We then evaluate the contour energy which restricts the face model to align
with the 2D face contour. It is more difficult to detect the landmarks of the
side face compared to the frontal one. We found that the detected landmarks
tend to have offset at the side when they are far away from the camera,
leading to a poor alignment between the face model and its 2D face contour.
For example, the landmarks near the eye have offset as shown in Figure 8(a).
Figure 8(b) and Figure 8(c) show results without and with the contour energy,
respectively. Although both results have similar eye locations, the one using
contour energy achieves better alignment. For an image sequence, aligned
contours result in continuous and stable face transformations.

####  6.3.2. Video reshaping

We first perform comparisons to demonstrate that our approach which combines
MLS and grid optimization is meaningful and effective. Figure 9 shows the
comparison with the MLS-only approach, the optimization-only approach, and our
approach. It shows that the grid optimization is effective in correcting
background distortion. And the MLS approach ensures the coherence of the face
boundaries and the video stability. Besides, optimization-only approach
requires a higher grid resolution (up to four times) to achieve similar
results, but still has incoherent face boundary.

(a) MLS-only (b) Optimization-only (c) Our method

Figure 9. Comparison between the method using MLS only (a), the method using
optimization only (b), and ours (c). Our method achieves better results in
terms of background distortion and face boundary coherence.

We then perform comparisons to prove that using SDF based dense boundary
mapping is effective for preserving face reshaping consistency and the
stability of the video. Figure 10(a) shows the case of not fixing face contour
points, where only the feature points used for reshaping are exploited as
fixed control points. This method does not reflect enough information about
the reshaping, thus the obtained results do not coincide with the projection
of the 3D model after reshaping. Figure 10(b) shows the case of only using a
set of sparse grid points to represent the face boundary. Here we select
suitable initial grid points coincided with the face contour as fixed control
points. Two more energy terms are introduced in Eqn. 17 in order to obtain
reasonable results (see Appendix for details).

(a) Without fixing contour

(b) Sparse mapping

(c) Our method

Figure 10. Comparison of extreme reshaping results between the method that
does not fix contour grid points (a), the method that only uses sparse contour
points mapping (b), and ours (c). Our method achieves better results in terms
of face boundary coherence and reshaping consistency.

The results are better than other baselines. In the case of a small degree of
face reshaping, the defects are almost invisible and it can generally achieve
video coherence. However, if the face is largely deformed, as shown in Figure
10(b), protrusions and gaps in the face boundary appear and they may affect
video continuity. Also, the time consumed by this method is significantly
increased.

Figure 11 shows the result of selecting control points based on dense mapping
from SDF without using the 3D model information. This performs well in most
cases, but has problem when the nose contour affects the face contour.

###  6.4. Implementation Details

We use the optical flow algorithm provided by OpenCV to extract the motion
map, and use Gaussian sampling to get the motion value $\mathbb{U}_{i}$ of
each projected face vertex instead of bilinear interpolation to avoid local
minima and obtain smoother results (Ren et al., 2019). We perform facial
landmark detection as proposed in (Bulat and Tzimiropoulos, 2017). We use
Ceres (Agarwal et al., 2012) to solve optimization in Section 4 and Section 5.
For image warping, we set the grid dimension to $100\times 100$ for all of our
results.

Our implementation is on a desktop PC with AMD Ryzen 9 3950X CPU and 32GB
memory. In the face reconstruction stage, pose estimation takes 120ms per
frame on average, identity estimation computes only once and takes 321ms, and
the expression estimation takes 150ms per frame. In the video reshaping stage,
the 3D face deformation takes 160ms at the beginning. Image warping takes
266ms per frame, where MLS based image deformation and grid optimization take
71ms and 74ms, respectively.

The videos we used are all downloaded from public datasets or Youtube
websites. The sources of each video are listed in Appendix.

(a) Without using 3D information (b) Our method

Figure 11. Comparison of extreme reshaping results between the method that
does not use 3D information to establish the dense mapping (a) and ours (b).
Our method performs better at face boundary.

##  7\. Discussions

All of our results show plausible reshaping without visible artifacts.
However, our method still has some limitations. First, visual distortion may
appear in the surrounding regions close to the face when large shape
deformation is employed. Video inpainting methods (Kim et al., 2019) could be
employed on background regions to further improve the results. Second, we note
the fact that wrinkles will reduce and a double chin will appear when gaining
weight on the face; while wrinkles will increase and a double chin will
diminish when losing weight. But similar to reshaping portrait images (Xiao et
al., 2020), our approach cannot deal with such changes. Further, our current
approach is fully unsupervised, which means it does not require any customized
face priors, thus it can be directly used to process portrait videos in the
wild. On the other hand, with the help of more pre-knowledge such as face
shape, our method can be consequently adapted and accelerated to reshape
portrait videos on the fly.

##  8\. Conclusions

We have presented the first method to reshape portraits in a video. Our video-
based face reconstruction method is able to eliminate the effects of jittering
landmarks and incorrect flow map, resulting in a steady 3D face model sequence
with accurate identities and smooth transformations. We achieve a consistent
video deformation by aligning the face model sequence with the face contours
of the image sequence. We employ an SDF based approach to produce a dense and
smooth mapping from the initial face to the reshaped face, which effectively
minimizes the warping distortion and avoids visual artifacts after video
reshaping. Extensive evaluations and comparisons demonstrate that our method
can generate high-quality portrait video reshaping results.

###### Acknowledgements.

Xiaogang Jin was supported by the National Natural Science Foundation of China
(Grant No. 61972344), the Key Research and Development Program of Zhejiang
Province (Grant no. 2018C03055), and the Ningbo Major Special Projects of the
“Science and Technology Innovation 2025” (Grant No. 2020Z007). Yong-Liang Yang
was partly supported by RCUK grant CAMERA (EP/M023281/1, EP/T022523/1), and a
gift from Adobe.

## References

  * (1)
  * Agarwal et al. (2012) Sameer Agarwal, Keir Mierle, and Others. 2012.  Ceres Solver.  http://ceres-solver.org.
  * Blanz and Vetter (1999) Volker Blanz and Thomas Vetter. 1999.  A morphable model for the synthesis of 3D faces. In _the 26th Annual Conference on Computer Graphics and Interactive Techniques_. ACM, 187–194.
  * Booth et al. (2017) James Booth, Epameinondas Antonakos, Stylianos Ploumpis, George Trigeorgis, Yannis Panagakis, and Stefanos Zafeiriou. 2017.  3D Face Morphable Models ”In-the-Wild”. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 5464–5473.
  * Bulat and Tzimiropoulos (2017) Adrian Bulat and Georgios Tzimiropoulos. 2017.  How Far are We from Solving the 2D & 3D Face Alignment Problem? (and a Dataset of 230, 000 3D Facial Landmarks). In _IEEE International Conference on Computer Vision (ICCV)_. 1021–1030.
  * Cao et al. (2018) Chen Cao, Menglei Chai, Oliver J. Woodford, and Linjie Luo. 2018\.  Stabilized Real-time Face Tracking via a Learned Dynamic Rigidity Prior.  _ACM Transactions on Graphics (TOG)_ 37, 6 (2018), 1–11.
  * Chen et al. (2020) Renwang Chen, Xuanhong Chen, Bingbing Ni, and Yanhao Ge. 2020\.  SimSwap: An Efficient Framework For High Fidelity Face Swapping. In _Proceedings of the 28th ACM International Conference on Multimedia_. 2003–2011.
  * Chen et al. (2013) Tao Chen, Jun-Yan Zhu, Ariel Shamir, and Shi-Min Hu. 2013\.  Motion-Aware Gradient Domain Video Composition.  _IEEE Transactions on Image Processing_ 22, 7 (2013), 2532–2544.
  * Egger et al. (2020) Bernhard Egger, William A. P. Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhöfer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, Christian Theobalt, Volker Blanz, and Thomas Vetter. 2020.  3D Morphable Face Models - Past, Present, and Future.  _ACM Transactions on Graphics (TOG)_ 39, 5 (2020), 157:1–157:38.
  * Huber et al. (2016) Patrik Huber, Guosheng Hu, Rafael Tena, Pouria Mortazavian, Wollem P. Koppen, William Christmas, Matthias Rätsch, and Josef Kittler. 2016.  A Multiresolution 3D Morphable Face Model and Fitting Framework. In _the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP)_ , Vol. 4. 79–86.
  * Jain et al. (2010) Arjun Jain, Thorsten Thormählen, Hans-Peter Seidel, and Christian Theobalt. 2010.  MovieReshape: tracking and reshaping of humans in videos.  _ACM Transactions on Graphics (TOG)_ 29, 6 (2010), 148\.
  * Kaufmann et al. (2013) Peter Kaufmann, Oliver Wang, Alexander Sorkine-Hornung, Olga Sorkine-Hornung, Aljoscha Smolic, and Markus H. Gross. 2013.  Finite Element Image Warping.  _Computer Graphics Forum_ 32, 2 (2013), 31–39.
  * Kim et al. (2019) Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon. 2019\.  Deep Video Inpainting. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 5792–5801.
  * Lewis et al. (2014) John P. Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frédéric H. Pighin, and Zhigang Deng. 2014.  Practice and Theory of Blendshape Facial Models. In _the 35th Annual Conference of the European Association for Computer Graphics_. 199–218.
  * Leyvand et al. (2008) Tommer Leyvand, Daniel Cohen-Or, Gideon Dror, and Dani Lischinski. 2008.  Data-Driven Enhancement of Facial Attractiveness.  _ACM Transactions on Graphics (TOG)_ 27, 3 (2008), 38\.
  * Li et al. (2020) Ruilong Li, Karl Bladin, Yajie Zhao, Chinmay Chinara, Owen Ingraham, Pengda Xiang, Xinglei Ren, Pratusha Prasad, Bipin Kishore, Jun Xing, and Hao Li. 2020.  Learning Formation of Physically-Based Face Attributes. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, Seattle, WA, USA, 3407–3416.
  * Nagano et al. (2019) Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, and Hao Li. 2019.  paGAN: real-time avatars using dynamic textures.  _ACM Transactions on Graphics_ 37 (2019), 1–12.
  * Ren et al. (2019) Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H. Li, Shan Liu, and Ge Li. 2019\.  StructureFlow: Image Inpainting via Structure-Aware Appearance Flow. In _IEEE/CVF International Conference on Computer Vision (ICCV)_. 181–190.
  * Schaefer et al. (2006) Scott Schaefer, Travis McPhail, and Joe D. Warren. 2006\.  Image deformation using moving least squares.  Vol. 25. 533–540.
  * Shih et al. (2019) Yi-Chang Shih, Wei-Sheng Lai, and Chia-Kai Liang. 2019\.  Distortion-Free Wide-Angle Portraits on Camera Phones.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019), 61:1–61:12.
  * Shih et al. (2014) Yi-Chang Shih, Sylvain Paris, Connelly Barnes, William T. Freeman, and Frédo Durand. 2014\.  Style Transfer for Headshot Portraits.  _ACM Transactions on Graphics (TOG)_ 33, 4 (2014), 148:1–148:14.
  * Tewari et al. (2020) Ayush Tewari, Mohamed Elgharib, Mallikarjun B R, Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. 2020.  PIE: portrait image embedding for semantic control.  _ACM Transactions on Graphics_ 39, 6 (2020), 1–14.
  * Thies et al. (2015) Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015.  Real-time Expression Transfer for Facial Reenactment.  _ACM Transactions on Graphics (TOG)_ 34, 6 (2015), 183:1–183:14.
  * Thies et al. (2016) Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016.  Face2Face: Real-Time Face Capture and Reenactment of RGB Videos. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 2387–2395.
  * Tran et al. (2019) Luan Tran, Feng Liu, and Xiaoming Liu. 2019.  Towards High-Fidelity Nonlinear 3D Face Morphable Model. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1126–1135.
  * Tran and Liu (2018) Luan Tran and Xiaoming Liu. 2018.  Nonlinear 3D Face Morphable Model. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 7346–7355.
  * Triggs et al. (1999) Bill Triggs, Philip F. McLauchlan, Richard I. Hartley, and Andrew W. Fitzgibbon. 1999.  Bundle Adjustment - A Modern Synthesis. In _International Workshop on Vision Algorithms_ , Vol. 1883. 298–372.
  * Xiao et al. (2020) Qinjie Xiao, Xiangjun Tang, You Wu, Leyang Jin, Yong-Liang Yang, and Xiaogang Jin. 2020\.  Deep Shapely Portraits. In _MM ’20: The 28th ACM International Conference on Multimedia_. 1800–1808.
  * Zeng et al. (2020) Dan Zeng, Han Liu, Hui Lin, and Shiming Ge. 2020\.  Talking Face Generation with Expression-Tailored Generative Adversarial Network. In _Proceedings of the 28th ACM International Conference on Multimedia_ _(MM ’20)_. 1716–1724.
  * Zhao et al. (2018) Haiming Zhao, Xiaogang Jin, Xiaojian Huang, Menglei Chai, and Kun Zhou. 2018.  Parametric Reshaping of Portrait Images for Weight-change.  _IEEE Computer Graphics and Applications_ 38, 1 (2018), 77–90.
  * Zollhöfer et al. (2018) Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, and Christian Theobalt. 2018.  State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications.  _Computer Graphics Forum_ 37, 2 (2018), 523–550.


# TOWARDS UNIVERSAL PHYSICAL ATTACKS ON CASCADED CAMERA-LIDAR 3D OBJECT
DETECTION MODELS

###### Abstract

We propose a universal and physically realizable adversarial attack on a
cascaded multi-modal deep learning network (DNN), in the context of self-
driving cars. DNNs have achieved high performance in 3D object detection, but
they are known to be vulnerable to adversarial attacks. These attacks have
been heavily investigated in the RGB image domain and more recently in the
point cloud domain, but rarely in both domains simultaneously - a gap to be
filled in this paper. We use a single 3D mesh and differentiable rendering to
explore how perturbing the mesh’s geometry and texture can reduce the
robustness of DNNs to adversarial attacks. We attack a prominent cascaded
multi-modal DNN, the Frustum-Pointnet model. Using the popular KITTI
benchmark, we showed that the proposed universal multi-modal attack was
successful in reducing the model’s ability to detect a car by nearly 73%. This
work can aid in the understanding of what the cascaded RGB-point cloud DNN
learns and its vulnerability to adversarial attacks.

Index Terms—  Adversarial attacks, cascaded multi-modal, 3D object detection,
point cloud, deep learning.

##  1 Introduction

Most modern autonomous vehicles (AVs) employ cameras and LiDAR sensors to
generate a complimentary representation of the scene in the form of dense 2D
RGB images and sparse 3D point clouds. Using these data, deep neural networks
(DNNs) have achieved state-of-the-art performance in 3D object detection. Such
DNNs are however vulnerable to adversarial attacks (mainly in the image
domain) that slightly alter the input but greatly affect the output. As safety
is critical for self-driving cars, this paper investigates the vulnerabilities
of cascaded multi-modal DNNs in 3D car detection.

Fig. 1: Illustrative example: Left: A car is detected with accurate bounding
boxes. Right: The same car is not detected after adding an adversarial object
with learned geometry and texture. The adversarial texture fools the 2D RGB
detection pipeline, and the green points are the rendered LiDAR points added
to the point cloud to fool the 3D point cloud pipeline.

Previous works demonstrated vulnerability of DNNs when the input is either an
image [1, 2] or a point cloud [3, 4]. Most point cloud attacks, simply alter
or add individual points to the point cloud, and are not physically realizable
due to properties of LiDAR sensors. More recently, a physically realizable
adversarial attack on point clouds was made using a perturbed mesh that is
placed in the 3D scene and rendered differentiably by simulating a LiDAR [5,
6]. Existing attacks either target a single modality or are not physically
realizable, limiting their applicability to real world scenarios.

Camera-LiDAR multi-model 3D object detectors can be divided into cascaded or
fusion models. Cascaded models usually use an off-the-shelf 2D image detector
to generate proposals that limit the 3D search space [7, 8, 9]. The points
within the proposed regions are then fed into a 3D object detector for
classification and bounding box regression. Fusion models extract and combine
image and point cloud features in parallel using DNNs, and then a combined
representation is sent to a detector for bounding box regression [10, 11].

.

Fig. 2: The proposed attack pipeline on the cascaded F-PN model [7].
Backpropagation is represented by a dashed red line.

This paper focuses on cascaded models because they are less computationally
expensive and more interpretable than fusion models. They are therefore easier
to train and deploy, especially that they also utilize mature 2D detection
DNNs. While fused models are currently less studied, cascaded models are
closer to industry deployment. Moreover, cascaded models have an implicit bias
that in the post-proposal limited 3D point cloud an object must reside and so
previous point cloud attacks could face a difficulty in attacking these
models. Nonetheless, our proposed attack can extend easily to fusion models
and other RGB-point cloud multi-modal DNNs.

We propose a novel, universal, and physically realizable adversarial attack
against a cascaded 3D car detection DNN. It attacks both image and point
cloud. An adversarial 3D object is placed on top of a car in a 3D scene and
then rendered to both point cloud and the corresponding RGB image by
differentiable renderers. The shape and texture of the object are trainable
parameters that are manipulated adversarially. To the best of our knowledge
this is the first work that explores physically realizable multi-modal
adversarial attacks on cascaded 3D object detection DNNs. This topic is
critical because most self-driving vehicles now have LiDAR and camera sensors,
and many advanced 3D detection models are multi-modal. Since the study of
adversarial attacks on multi-modal DNN has so far been rare, this paper is
towards addressing this research gap. Based on evaluations on the KITTI
dataset, our multi-modal attack reduced the average precision of car detection
from 86% to 28% under easy conditions.

##  2 Related Work

Adversarial attacks are heavily explored in the image domain [12, 1, 2],
starting with digital pixel-wise adversarial perturbations that were not
realistic in the physical world [1]. Then constraints were established to
ensure the physical feasibility of an attack and so patches [12], and
sunglasses [2] were proposed to attack a model. Also, universal attacks were
studied where an attack is not just trained on a single image [13]. Recently,
differentiable renderers were used to make adversarial attacks by altering the
3D geometry of an object or its lighting conditions and then rendering them to
images for an adversarial attack on 2D detection DNNs [14, 15].

Our attack is very different from the aforementioned work: We aim not to learn
a patch or a pixel value, but an adversarial texture on a 3D object that can
attack a model even when viewed from many angles. Moreover, we target point
cloud. The dense representation of RGB images made image attacks easier
compared to point cloud attacks where the representation is very sparse. In
the context of AVs, assuming a physically realistic attack on RGB images, it
still does nothing with regards to a LiDAR sensor that is mainly concerned
with 3D geometry.

Early work on adversarial attacks on point cloud DNNs focused on perturbing
point clouds by slightly moving, adding, or removing a few points [3, 4].
These perturbations are largely theoretical, i.e. they have no guarantees that
they can be realized physically. Also, these attacks were largely made on
relatively dense point clouds that represent shapes and small objects, unlike
the very sparse nature of AVs’ point clouds.

To tackle the physical feasibility of a point cloud attack, [5] used an
adversarial 3D mesh that is placed somewhere in a 3D scene around an AV. But
it was only trained on a single example and an in-house dataset and so it was
not universal, and their attack is not evaluated on a publicly available
benchmark. To remedy these shortcomings, [6] proposed a universal attack on
LiDAR using a mesh that is placed on top of a car, with the objective of
minimizing the detection score. The attack was evaluated on different point
cloud DNNs using the KITTI benchmark. All these approaches focus only on point
cloud detection, while an AV also has a camera sensor.

##  3 Methods

We plan to learn a single adversarial object that is placed on top of a car in
a 3D scene, with the goal that this adversarial object can significantly
reduce the accuracy of a cascaded multi-modal DNN for 3D object detection.
This object is rendered to point cloud and RGB image as if it was present in
the original scene (see Fig. 2). The consequences of such an attack are
especially dangerous, because if those 2 main modalities are attacked, cars
have very little recourse. Here we describe how this object is learned, the
rendering methods, and the victim multi-modal DNN.

###  3.1 Attack method

To make a realistic adversarial attack on multi-modal DNNs that can target
both camera and LiDAR modalities, we require a representation that can
maintain realistic 3D physical geometry and also can be differentiably
rendered to RGB images and LiDAR. We choose a mesh for our adversarial object,
since it provides an efficient representation of a 3D object and can be
rendered to images or point clouds. This adversarial mesh is placed on top of
a vehicle to avoid occlusion, and rotated around the vertical axis to have the
same orientation as the vehicle.

To train the shape of the object, following previous work [5, 6], we start
with an initial mesh $S$ with $V$ vertices, where each initial vertex is
defined as $v_{i}^{0}\in\mathbb{R}^{3}$ $,i\in\\{1,2,...,V\\}$. We deform the
shape of the mesh by displacing each vertex with a displacement vector
$\hat{v_{i}}\in\mathbb{R}^{3}$, which is a learnable parameter to produce an
adversarial mesh $S^{adv}$ with vertices $v_{i}$’s, as in Eqn. (1). We use a
$4\times 4$ transformation matrix $T$ to put the mesh on top of a car and give
it the same orientation:

$v_{i}=T\cdot(v_{i}^{0}+\hat{v_{i}})$ (1)

For the texture of the adversarial mesh, we give each vertex a corresponding
RGB color $c_{i}\in\mathbb{R}^{3}$, which is a learnable parameter. We then
render the mesh to 2D and add it to the original RGB scene using a given
projection matrix. To produce the texture of the mesh, each face is colored by
the interpolation of the colors from the vertices surrounding it.

To ensure our object is realistic, we put constraints on the size and
smoothness of the mesh geometry. Also since we interpolate between vertex
colors to produce the adversarial texture, the result is a realistic smooth
texture without abrupt changes in color (a typical observation in RGB
adversarial attacks using pixel-wise perturbations).

###  3.2 Differentiable Rendering

To realistically render a mesh to point cloud, we need to find which LiDAR
rays would intersect with the mesh if it was placed in the original scene. We
simulate the LiDAR used in capturing the dataset’s point cloud by producing
rays at the same angular frequencies and incorporating a small amount of noise
that is present in this specific LiDAR. We then calculate the intersection
points between these rays and our mesh’s triangles in the 3D scene using the
Möller–Trumbore intersection algorithm [16]. Finally, for each ray with
intersection points we take the nearest point, and add all the resulting
points to the LiDAR point cloud scene.

After placing our mesh on top of all cars in a 3D scene, we render the meshes
to 2D using a given projection matrix. To train our adversarial texture, the
rendering process needs to be differentiable. We therefore use the fast,
differentiable rendering tools developed in [17] to render our adversarial
mesh from the 3D scene to RGB images, as shown in Fig. 2.

###  3.3 Victim Model

As mentioned in section 1, we are attacking cascaded models. We chose the
Frustum-PointNet (F-PN) model [7] for many reasons. First, it’s a pioneering
work in the area of multi-modal DNN for AVs and many works were developed
based on its mythology and architecture [8, 9]. Moreover, it has a PointNet
backbone which is commonly used in many single and multi-modal 3D detection
DNNs, and so this attack could be a threat to other 3D detection DNNs that
rely on PointNet [18]. Finally, it showed competitive results on the KITTI
benchmark.

As shown in Fig. 2, F-PN first takes an RGB image through a 2D detection DNN,
which proposes 2D bounding boxes. These bounding boxes are then projected to
3D space, thus producing frustum-shaped 3D search spaces that surround each
object. Points within each frustum are extracted and sent through two PointNet
based DNNs for 3D instance segmentation, and 3D box estimation. F-PN directly
outputs one 3D bounding box estimation from a given point cloud frustum, since
the assumption is that there is only a single object in a frustum.

For image detection, we use YOLOv3 [19]. It is a standard fast 2D object
detection DNN that outputs region proposals, classifications, and confidence
scores. The F-PN and YOLOv3 models were pre-trained on the KITTI dataset.

###  3.4 Objective Functions

We tried to attack the point cloud network in several ways and found that the
most successful attack is achieved by minimizing the accuracy of the
segmentation network. Assuming $N$ points in $M$ point clouds, the F-PN
segmentation network outputs 2 logits for each point, estimating whether it
belongs to an object or not, $logit_{i}\in\mathbb{R}^{2},i\in\\{1,2,...,N\\}$.
We softmax these logits and extract the highest probability given to a “car”
point, $p_{k}=\max\limits_{j}\\{softmax(logit_{j})\\}$ in a point cloud $k$,
where $j$ is a point classified as car. To attack the 3D segmentation network,
we train the shape of the mesh to minimize this probability. Finally,
following previous work [6], we weighed the objective function by the IoU
score between the ground truth $y_{gt}$ and predicted $\hat{y}$ bounding
boxes:

$\mathcal{L}_{mesh}=\sum_{k\in M}-1*log(1-p_{k})*IoU(y_{gt},\hat{y}).$ (2)

As for the image loss function, we simply minimize the “objectness” score
given to resulting car classifications.

We also want to ensure that the geometry of the mesh is smooth and realistic,
so we minimize a Laplacian loss [20]:
$\mathcal{L}_{lap}=\sum_{i}\left\lVert\delta_{i}\right\rVert^{2}_{2}$, where
$\delta_{i}$ is the distance between a vertex $v_{i}$ and the centroid of its
neighbors $\delta_{i}=v_{i}-\frac{1}{\left\lVert N(i)\right\rVert}\sum_{j\in
N(i)}v_{j}$.

The overall point cloud attack objective function is:

$\mathcal{L}^{pc}_{adv}=\mathcal{L}_{mesh}+\lambda\mathcal{L}_{lap}$ (3)

Fig. 3: An example of the RGB+point cloud attack. Green boxes are ground
truth, yellow boxes are detections without adversarial attack, and red ones
are detections after the adversarial attack.

##  4 Experiments

###  4.1 Experiment Setup

We use the KITTI dataset [21], a popular benchmark for 3D detection in
autonomous driving. Roughly half of the samples are used for training and the
other half for validation. All reported results in the following section are
from the validation set. We use ground truth 2D bounding boxes to generate the
frustums used in training and evaluating the attack on the point cloud
pipeline. We use YOLOv3 generated 2D bounding boxes to measure the effect of
an image-only attack and image + point cloud attack on detection. This is a
universal attack, i.e., the shape and texture of the object are trained on the
entire training set.

We first perturb the geometry of the object, to attack the 3D part. We start
with a 40 cm radius isotropic sphere mesh with 162 vertices and 320 faces. We
apply box constraints to keep the mesh’s 3 dimensions below the initial
levels. An ADAM optimizer is then used to iteratively deform the mesh to
minimize (3). Once we have an adversarial shape we render it to 2D images and
train a universal adversarial texture. We give each vertex an initial color
and then we use an ADAM optimizer to learn which color, for each vertex, would
produce an adversarial texture to attack the RGB pipeline.

###  4.2 Results & Discussion

We measure the success of an attack by the reduction in the average precision
(AP) of car detection due to the introduction of an adversarial mesh to the
scene. Table 1 shows the bird’s eye view (BEV) AP results of our trained F-PN
after different types of attacks. We show the results for 3 difficulty levels
(based on occlusion) with an IoU threshold of $0.7$.

Attack Typea |  Easy |  Moderate |  Hard
---|---|---|---
No Attack |  85.66 |  83.62 |  76.23
PC: Adv Shape |  37.36 |  37.79 |  39.10
Img: Adv Texture |  51.85 |  35.98 |  31.49
PC + Img: Adv Object |  27.50 |  22.67 |  19.21
aAttack on the point cloud (PC) or image (Img) pipeline.
Table 1: BEV 3D car detection AP results

It can be challenging to attack the point cloud part of a cascaded model for
several reasons. First, these models are built with an implicit bias that an
object must reside in a proposed region and so they can be resistant to simple
perturbations. Also, the post-proposal point cloud is very sparse and small in
size. Thus there is not much space for an adversarial attack to take place.
Nonetheless, our adversarial mesh was able to reduce the localization BEV AP
by nearly 45% by just targeting the point cloud pipeline. This can be
attributed to cascaded models’ limited exposure to examples where objects
could be on cars during training. The frustum limits the 3D search space by
focusing only on the body of a car. This is a weakness in cascaded models,
since the limited 3D search space means that it may not learn at all about the
surrounding 3D scene. This poses a grave danger for autonomous driving since
the dimensions and locations of surrounding vehicles and objects contribute
heavily to decision making. As shown in Fig 3, our adversarial object not only
led to some cars going entirely undetected, but also introduced other
detection errors.

The RGB pipeline is the bottleneck of detection performance in camera-LiDAR
cascaded models. The 2D proposals decide exactly where to search for objects
and so if that fails the entire model fails. Using 2D region proposals doesn’t
utilize a main purpose behind LiDAR which is to avoid cases where lighting and
occlusion affect localization and detection. RGB attacks are much simpler and
less computationally expensive than point cloud attacks, and they’re more
easily reproduced in real life which can make them more dangerous.

Many previous work showed that data augmentation can make a DNN more robust to
adversarial attacks. In the training of our network, we included the data
augmentation step in the original work [7] by randomly scaling and shifting
the proposed 2D bounding boxes. This gave the network certain robustness, as
shown in the relatively high AP of car detection even after the RGB attack
under the Easy case. Nonetheless, when occlusion increased, detection accuracy
rapidly decreased. Under moderate circumstances, BEV AP decreased from 83.62%
to 22.67%, a nearly 73% reduction.

##  5 Conclusion

We proposed a novel, universal, and physically realistic adversarial attack on
a cascaded camera-LiDAR DNN for 3D object detection. We manipulated mesh
geometry and texture and used differentiable rendering to study the
vulnerability of cascaded DNNs. We found that cascaded models are vulnerable
to adversarial attacks and that detection accuracy can decrease by nearly 73%
if we attack both camera and LiDAR modalities simultaneously. While we have
applied here the proposed attack to a multi-modal cascaded model, it can be
extended in future work to investigate the vulnerability of different multi-
modal DNN models (e.g., fusion models) for different tasks.

## References

  * [1] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy,  “Explaining and harnessing adversarial examples,”  arXiv preprint arXiv:1412.6572, 2014.
  * [2] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter,  “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition,”  in Proceedings of the 2016 acm sigsac conference on computer and communications security, 2016, pp. 1528–1540.
  * [3] Jiancheng Yang, Qiang Zhang, Rongyao Fang, Bingbing Ni, Jinxian Liu, and Qi Tian,  “Adversarial attack and defense on point sets,”  arXiv preprint arXiv:1902.10899, 2019.
  * [4] Chong Xiang, Charles R. Qi, and Bo Li,  “Generating 3d adversarial point clouds,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [5] Yulong Cao, Chaowei Xiao, Dawei Yang, Jing Fang, Ruigang Yang, Mingyan Liu, and Bo Li,  “Adversarial objects against lidar-based autonomous driving systems,”  2019\.
  * [6] James Tu, Mengye Ren, Sivabalan Manivasagam, Ming Liang, Bin Yang, Richard Du, Frank Cheng, and Raquel Urtasun,  “Physically realizable adversarial examples for lidar object detection,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [7] Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas,  “Frustum pointnets for 3d object detection from rgb-d data,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [8] Zhixin Wang and Kui Jia,  “Frustum convnet: Sliding frustums to aggregate local point-wise features for amodal 3d object detection,”  arXiv preprint arXiv:1903.01864, 2019.
  * [9] K. Shin, Y. P. Kwon, and M. Tomizuka,  “Roarnet: A robust 3d object detection based on region approximation refinement,”  in 2019 IEEE Intelligent Vehicles Symposium (IV), 2019, pp. 2510–2515.
  * [10] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai,  “Epnet: Enhancing point features with image semantics for 3d object detection,”  in European Conference on Computer Vision. Springer, 2020, pp. 35–52.
  * [11] Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel,  “Mvx-net: Multimodal voxelnet for 3d object detection,”  in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 7276–7282.
  * [12] Simen Thys, Wiebe Van Ranst, and Toon Goedemé,  “Fooling automated surveillance cameras: adversarial patches to attack person detection,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019, pp. 0–0.
  * [13] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard,  “Universal adversarial perturbations,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1765–1773.
  * [14] Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang, Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi-Keung Tang, and Alan L Yuille,  “Adversarial attacks beyond the image space,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 4302–4311.
  * [15] Hsueh-Ti Derek Liu, Michael Tao, Chun-Liang Li, Derek Nowrouzezahrai, and Alec Jacobson,  “Beyond pixel norm-balls: Parametric adversaries using an analytically differentiable renderer,”  arXiv preprint arXiv:1808.02651, 2018.
  * [16] Tomas Möller and Ben Trumbore,  “Fast, minimum storage ray-triangle intersection,”  Journal of graphics tools, vol. 2, no. 1, pp. 21–28, 1997.
  * [17] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari,  “Accelerating 3d deep learning with pytorch3d,”  arXiv:2007.08501, 2020.
  * [18] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas,  “Pointnet: Deep learning on point sets for 3d classification and segmentation,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652–660.
  * [19] Joseph Redmon and Ali Farhadi,  “Yolov3: An incremental improvement,”  arXiv preprint arXiv:1804.02767, 2018.
  * [20] Shichen Liu, Weikai Chen, Tianye Li, and Hao Li,  “Soft rasterizer: Differentiable rendering for unsupervised single-view mesh reconstruction,”  arXiv preprint arXiv:1901.05567, 2019.
  * [21] Andreas Geiger, Philip Lenz, and Raquel Urtasun,  “Are we ready for autonomous driving? the kitti vision benchmark suite,”  in 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012, pp. 3354–3361.