# Inverting and Understanding Object Detectors

Ang Cao, Justin Johnson
University of Michigan
{ancao<EMAIL_ADDRESS>
###### Abstract

As a core problem in computer vision, the performance of object detection has
improved drastically in the past few years. Despite their impressive
performance, object detectors suffer from a lack of interpretability.
Visualization techniques have been developed and widely applied to introspect
the decisions made by other kinds of deep learning models; however visualizing
object detectors has been underexplored. In this paper, we propose using
inversion as a primary tool to understand modern object detectors and develop
an optimization-based approach to layout inversion, allowing us to generate
synthetic images recognized by trained detectors as containing a desired
configuration of objects. We reveal intriguing properties of detectors by
applying our layout inversion technique to a variety of modern object
detectors, and further investigate them via validation experiments: they rely
on qualitatively different features for classification and regression; they
learn canonical motifs of commonly co-occurring objects; they use different
visual cues to recognize objects of varying sizes. We hope our insights can
help practitioners improve object detectors. 111The code is
https://github.com/Caoang327/vis_det

##  1 Introduction

Object detection, which requires jointly classifying and localizing objects,
is a core computer vision task widely used in applications such as autonomous
vehicles, biomedical imaging, and content recommendation. Deep learning has
enabled swift progress on this task, making detectors based on convolutional
networks accurate [15, 50, 19] and efficient [32, 45, 30, 4] on challenging
datasets [31, 17].

Despite their impressive performance, object detectors (like other deep
learning models) suffer from a lack of interpretability. Their internal
representations and decisions are opaque, and the visual cues used to make
these decisions are nonobvious. An improved understanding of object detectors
may help researchers diagnose errors and ultimately improve their object
detection models.

Figure 1: Layout Inversion: An object detector predicts a _scene layout_ from
an image (solid arrow). We develop methods for _layout inversion_ (dotted
arrow) that invert trained detectors, and generate images recognized as
containing the target layout.

Prior work has developed techniques for visualizing and understanding other
types of deep models, including convolutional [66], recurrent [25, 28] and
adversarial [3] networks. Generalizing these techniques to object detectors is
nontrivial due to their highly structured outputs: while an image
classification model gives a single category label per image, an object
detector emits a variably-sized set of detections, each involving
classification, localization, and (optionally) segmentation [18, 19]. Object
detectors also rely on non-differentiable operations like non-max suppression
(NMS) which break the gradient pathway between predicted objects and image
pixels, preventing the use of naïve gradient ascent used to visualize other
model types.

This paper extends network visualization techniques to modern object detectors
to gain insight into these models and better understand them. As our primary
tool for investigation, we propose an optimization-based approach to _layout
inversion_ (Figure 1) that generates synthetic images recognized by trained
detectors as containing a desired configuration of objects. We circumvent non-
differentiabilities with an alternating optimization algorithm inspired by the
alternating direction method of multipliers (ADMM) [5].

In Section 4.1 we show that our layout inversion method can be applied to a
wide variety of modern object detectors that vary in meta-architecture
(single-stage vs. two-stage, anchor-based vs. anchor-free) and that are
trained for different tasks (object detection vs. instance segmentation).
These visualizations reveal qualitative differences between the visual cues
used by different detectors for recognition.

In Section 4.2 we use _inversion transfer_ to quantify the performance of our
layout inversions by checking whether an image generated from one detector can
be recognized by another detector. This verifies the effectiveness of our
layout inversion approach, and also allows us to measure whether different
detectors rely on differing visual cues.

Object detectors predict both _what_ objects are present and _where_ they are
located. In Section 4.3 we disentangle the effects of these subtasks by
performing layout inversion targeting different combinations of the
classification, regression, and segmentation heads of detectors. These
experiments show that detectors rely on qualitatively different visual
features for classification and localization. We investigate further by using
_attribution methods_ to show that classification and regression rely on
different image regions.

Full layout inversion requires finding an image where a target set of regions
are recognized by the detector as objects, and where all other regions are
dismissed as background. In Section 4.4 we probe the priors learned by object
detectors by performing layout inversions requiring only a single object to be
recognized as foreground, and omitting the requirement that other regions be
dismissed.

In Section 4.4.1 we investigate the role of _context_ , and demonstrate that
detectors learn canonical motifs of commonly co-occurring objects. We thus
hypothesize that detectors rely heavily on context to perform recognition.

In Section 4.4.2 we observe that varying _object scale_ causes detectors to
rely on qualitatively different features for recognition. Small objects are
recognized primarily by shape and context, while large objects are recognized
as amalgamations of part and texture. We quantify this hypothesis by
eliminating object texture via masked style transfer.

Our contributions are two-folded. First, our experiments show that the
proposed layout inversion is a versatile tool for introspecting object
detectors. Second, we gain novel insights into these models’ inner workings
via inversion and design experiments to validate these hypotheses. We hope
these insights can help practitioners better understand and improve these
models’ performance in the future.

##  2 Related Work

Object Detection. Classical approaches use a sliding-window approach, densely
applying a classifier to image patches [27, 60, 8, 11]. More recently the two-
stage R-CNN framework [15, 20, 14, 50] has become a dominant paradigm, where
one neural network applied to images predicts regions of interest (RoIs) and a
second per-region network classifies objects. This versatile framework has
been extended to tasks including instance segmentation [19], keypoint
estimation [19], and 3D shape prediction [16]. Single-stage detectors form a
second category of recent methods, in which a single fully-convolutional
network is applied per-image to jointly propose and classify regions [32, 45,
46, 30, 47]. Single and two-stage methods typically regress boxes from a set
of convolutional anchors; recently a number of anchor-free approaches have
been proposed that model object corners [26] or centers [71, 10, 59] or use
transformers [6]. In this paper we focus primarily on RetinaNet [30], Faster
R-CNN [50] and Mask R-CNN [19] as representatives of the common anchor-based
single and two-stage approaches; we also work on FCOS [59] as a representative
of anchor-free methods.

Neural Network Visualization. Techniques have been developed for introspecting
the decisions made by convolutional [53, 66], recurrent [25, 28, 57] and
adversarial [3] networks. A typical approach is synthesizing images which
maximize predicted category scores or internal neuron values [53, 66, 65, 39];
other approaches invert feature vectors [33, 9]. Images are typically
generated via gradient descent on pixels, though some approaches use
evolutionary search [38]. Image regularizers are used to encourage generated
images to appear natural [53, 65, 35, 39, 37, 36] Alternatives to synthesizing
images include measuring agreement between neuron activation maps and semantic
maps [69, 68, 2], or finding patches that maximize hidden activations [66]. To
date these techniques have been mainly applied to image classification models.
Object detectors are only be visualized before the deep learning era, with HOG
features and SVM [61]; we generalize them to modern object detectors.

Attribution Methods. Some methods introspect deep models by determining which
part of an input image was responsible for the network’s decision by computing
gradients of the network output with respect to the input image [53, 66, 56,
54] or using saliency maps [70, 44, 12, 52]. Other methods fuse information
from gradients, activations, and weights using classification activation
mapping (CAM) [70, 52, 44], or measure how occlusions to input images affect
network outputs [66, 42, 12]. These methods have been applied to image-level
tasks like classification [53, 66], question answering [70], and captioning
[52]; we extend them to the region-based task of object detection.

Image Generation from Layouts. Some conditional generative models synthesize
images from different kinds of scene layouts such as keypoints [49, 48], scene
graphs [24, 1], bounding boxes [67, 58], or segmentation maps [22, 72, 7, 62,
41]. We also generate images from layouts, but rather than training
conditional generative models we invert pretrained object detectors as an
introspection tool.

##  3 Inverting Object Detectors

Figure 2:  (a) Gradient descent cannot be used to invert single-stage
detectors since non-differentiable post-processing $f$ blocks gradient flow
between the image $I$ and loss $\ell$. (b) We overcome this problem by
introducing auxiliary _pseudo-targets_ $Z$. We alternate between updating the
image $I$ so the network output $\Phi(I)$ match the pseudo-targets $Z$, and
updating $Z$ to be a minimal perturbation of $\Phi(I)$ that would give the
target layout $\hat{L}$ after applying $f$.

Our goal in this section is to develop a general method for inverting
universal object detectors. This is challenging because modern detectors
comprise both differentiable neural network layers and non-differentiable
operators which preclude the use of naïve gradient-based optimization commonly
used to invert image classification models.

We overcome this challenge with an alternating optimization procedure inspired
by the alternating direction method of multipliers (ADMM) [5]. Introducing
additional auxiliary variables splits the intractable objective to
differentiable terms that can be optimized via gradient ascent, and non-
differentiable terms that can be optimized heuristically.

###  3.1 Problem Setup

Considered as a black box, an object detector is a function $\mathcal{F}$
which inputs an image $I$ and returns a scene layout $L$ as a collection of
detected objects: $\mathcal{F}(I)=L=\\{(c_{i},b_{i},m_{i})\\}$. Each object
has category label $c_{i}\in\mathcal{C}$ from a fixed set of categories
$\mathcal{C}$, a bounding box $b_{i}=(x_{i},y_{i},h_{i},w_{i})$, and
(optionally, if the model performs instance segmentation) a binary
segmentation mask $m_{i}$.

Given a trained detector $\mathcal{F}$, we wish to introspect its decisions by
inverting it: starting from a target layout $\hat{L}$, find an image $I^{*}$
such that $\mathcal{F}(I^{*})=\hat{L}$. By analogy with methods commonly used
to visualize image classification models [53, 34], one path forward might be
defining a loss function $\ell$ between the detector’s output and the desired
layout, an image regularizer $R$ encouraging natural images, and optimize via
gradient descent:

$I^{*}=\arg\min_{I}\ell(\mathcal{F}(I),\hat{L})+R(I)$ (1)

However, this method cannot work when the detector $\mathcal{F}$ involves non-
differentiable components such as eliminating detections below a confidence
threshold, and non-maximum suppression (NMS) to prevent duplicate detections.
As shown in Figure 2, these non-differentiabilities block the gradient flow
between the loss and the input image, precluding the use of naïve gradient
ascent.

We proceed by writing the detector $\mathcal{F}$ as a composition of
differentiable neural networks $\Phi_{i}$ and nondifferentiable steps $f_{i}$.
Single-stage detectors can generally be written in the form
$\mathcal{F}(I)=f_{1}(\Phi_{1}(I))$, while two-stage detectors can be written
as $\mathcal{F}(I)=f_{2}(\Phi_{2}(f_{1}(\Phi_{1}(I)),I))$. Like ADMM, we
introduce auxiliary variables $Z_{i}$ and constraints encouraging $Z_{i}$ to
be equal to the output of $\Phi_{i}$. We then minimize via alternating
optimization, updating $I$ and each $Z_{i}$ in turn. We will describe the
setup for single-stage detectors in detail, then summarize its extension to
two-stage methods; see the supplementary material for a more explicit
discussion.

Figure 3:  We show updating the pseudo-targets $Z$ of raw network outputs
$\Phi(I)$ with several candidate detection examples. The predicted
probabilities and categories are annotated at the right-bottom and left-top of
boxes. Green boxes represent matched detections (with the highest IoU with
ground truth objects), and blue boxes represent unmatched detections (with low
IoU and kept after post-processing). The ground truth categories and box
positions are assigned to matched detections (green boxes), while unmatched
detections (blue box) are assigned background categories as classification
targets. $Z$ results in $f(Z)=\hat{L}$.

###  3.2 Inverting Single-Stage Detectors

Let $\mathcal{F}(I)=f(\Phi(I))$ be a single-stage detector. $\Phi$ is a
network receiving the image $I$ and outputing a fixed-size tensor $\Phi(I)$ of
shape $N\times(D_{cls}+D_{reg})$ giving raw classification scores and box
positions for $N$ candidate detections. These raw outputs are converted into a
set of detected objects by non-differentiable post-processing $f$.

Inspired by ADMM, we introduce auxiliary variables $Z$ with the same shape as
$\Phi(I)$. This allows us to rewrite Equation 1 as a constrained optimization
problem:

$I^{*},Z^{*}=\arg\min_{I,Z}\ell(f(Z),\hat{L})+R(I)\textrm{ with }Z=\Phi(I)$
(2)

We can then relax this back to an unconstrained problem:

$\displaystyle I^{*},Z^{*}=\arg\min_{I,Z}L_{\lambda}(I,Z)$ (3) $\displaystyle
L_{\lambda}(I,Z)=\ell(f(Z),\hat{L})+\lambda d(Z,\Phi(I))+R(I).$ (4)

Here $\lambda>0$ is a hyperparameter and $d$ is a differentiable distance
function. We choose $\ell$ to be an indicator function that is $0$ when
$f(Z)=\hat{L}$ and $+\infty$ otherwise. We now minimize Equation 3 by
alternating updates to $I$ and $Z$:

$\displaystyle I^{k+1}$ $\displaystyle=\arg\min_{I}L_{\lambda}(I,Z^{k})$ (5)
$\displaystyle=\arg\min_{I}\lambda d(Z^{k},\Phi(I))+R(I)$ (6) $\displaystyle
Z^{k+1}$ $\displaystyle=\arg\min_{Z}L_{\lambda}(I^{k+1},Z)$ (7)
$\displaystyle=\arg\min_{Z}\ell(f(Z),\hat{L})+\lambda d(Z,\Phi(I^{k+1}))$ (8)

We can update $I$ via gradient descent since all terms of Equation 6 are
differentiable. Since $\ell$ is an indicator, Equation 8 is equivalent to
finding $Z$ which minimizes $d(Z,\Phi(I^{k+1}))$ subject to $f(Z)=\hat{L}$. In
other words, updating $Z$ amounts to finding a _minimal perturbation_
(measured by $d$) of the current raw detector outputs $\Phi(I)$ that would
produce the target layout $\hat{L}$ after post-processing.

Intuitively, we can think of $Z$ as _pseudo-targets_ for the raw outputs of
the network $\Phi$. We alternate between updating the image $I$ so the network
outputs $\Phi(I)$ are closer to the pseudo-targets, and updating the pseudo-
targets $Z$ to be the smallest change to the current network outputs $\Phi(I)$
which would result in the target layout.

Updating Pseudo-Targets. We update the pseudo-targets $Z$ heuristically by
analyzing the post-processing $f$: it filters out most candidate detections,
and few detections are kept in the final layout. Therefore, the pseudo-targets
$Z$ for raw detector outputs $\Phi(I)$ with a minimal perturbation can be
reached by assigning ground-truth categories and positions as targets to
several detections in $\Phi(I)$ while letting targets equal to outputs for
other detections.

Like Figure 3, we match ground-truth instances in target layouts with
candidate detections of the highest IoU and assign the ground-truth categories
and box positions as targets to these detections. For candidate detections
which are not matched to any ground-truth instances and would be kept after
post-processing, we assign background as target predictions so that it would
be eliminated by post-processing.

Loss Function. We use a differentiable loss $d$ to measure the agreement
between the raw network outputs $\Phi(I)$ and the pseudo-targets $Z$. Inspired
by losses commonly used to train object detectors, we define $d$ as a weighted
combination of a cross-entropy loss $\ell_{cls}$ and a GIoU loss [51]
$\ell_{reg}$:

$d(\Phi(I),Z)=\lambda_{cls}\ell_{cls}(\Phi(I),Z)+\lambda_{reg}\ell_{reg}(\Phi(I),Z)$

(9)

###  3.3 Inverting Two-Stage Detectors

We use the same idea to invert two-stage detectors by introducing auxiliary
variables $Z_{1}$, $Z_{2}$ with constraints $Z_{1}=\Phi_{1}(I)$ and
$Z_{2}=\Phi_{2}(f_{1}(\Phi_{1}(I)),I)$. $\Phi_{1}(I)$ is the output of RPN
network $\Phi_{1}$, and $\Phi_{2}(f_{1}(\Phi_{1}(I)),I)$ is the prediction
result of $\Phi_{2}$ on every RoI. Loss
$\ell_{rpn}(f_{1}(\Phi_{1}(I)),\hat{L})$ is added to improve the quality of
proposed RoIs:

$\displaystyle\begin{split}\ell(f_{2}(Z_{2}),\hat{L})+\ell_{rpn}(f_{1}(Z_{1}),\hat{L})+\lambda_{1}\ell_{1}(Z_{1},\Phi_{1}(I))\\\
+\lambda_{2}\ell_{2}(Z_{2},\Phi_{2}(f_{1}(\Phi_{1}(I)),I))+R(I)\end{split}$
(10)

This could be solved by alternating optimization between $I,Z_{1}$ and $Z_{2}$
following the same manner as single-stage methods. Intuitively, $Z_{1}$ and
$Z_{2}$ are the pseudo-targets for outputs of network $\Phi_{1}$ and
$\Phi_{2}$. $\ell_{1}$ and $\ell_{2}$ are the loss functions between network
outputs and pseudo-targets for $\Phi_{1}$ and $\Phi_{2}$ respectively. See
more details in supplementary.

##  4 Experiments

We apply layout inversion as our primary visualization tool to a wide variety
of object detectors, where the inverted images reveal the learned features and
representative visual cues for detections. In Section 4.1, we show the
qualitative difference between inversions of varying detectors. We further
quantify the inversion performance by _inversion transfer_ in Section 4.2,
which also reflects the similarity of visual cues learned by different object
detectors.

On the other hand, object detection is a highly structured task: predicting
both _what_ is present and _where_ things are located for multiple objects in
a scene. To this end, we disentangle subtasks and invert individual objects.

We disentangle subtask losses in Section 4.3 by performing layout inversion
targeting with loss combinations, to reveal the visual cues learned by each
head. We also use _attribution_ to find image regions responsible for each
subtask.

We visualize individual objects in Section 4.4. Inspired by observations, we
investigate the role of _contexts_ in Section 4.4.1 and measure the feature
difference used for detecting objects with varying scales in Section 4.4.2.

Layout |  |  |  |  |  |
---|---|---|---|---|---|---
Mask |  |  |  |  |  |
Faster |  |  |  |  |  |
Retina |  |  |  |  |  |
FCOS |  |  |  |  |  |
Figure 4:  Visualization results with layout information input for Mask R-CNN,
Faster R-CNN, RetinaNet, and FCOS, all using ResNet-50-FPN backbones. The
layout information is the bounding box, category annotations, with additional
instance masks only for Mask R-CNN. All generated images reproduce the desired
layouts when passed to detectors. Zoom in for details.

Network We Visualize. We focus on a set of models spanning different meta-
architectures [21], which we believe are a representative subset of modern
detectors. We use RetinaNet [30] as a single-stage method. It uses a fully-
convolutional network to jointly classify and regress box predictions from a
set of convolutional anchors. We use Faster R-CNN [50] as a two-stage method.
It uses a fully-convolutional region proposal network (RPN) to predict regions
of interest (RoIs), which are classified and refined with a second per-region
network. We use Mask R-CNN [19] as a instance segmentation method. It augments
Faster R-CNN with a fully-convolutional segmentation network applied per-RoI
to predict per-object binary masks. We use FCOS [59] as an anchor-free method.
It uses a fully-convolutional network to classify and regress box predictions
on a set of points on the feature map. We use reference models trained on COCO
[31], provided by Detectron2 [64]. All models use ResNet-50-FPN backbones
[29].

Optimization Details. We conduct 1000 alternating iterations, and in each
iteration, we update $I$ using one-step SGD. We use periodically blurring
[65], TV-norm [33] and $p$-norm [53] as regularizer. See details in
supplementary.

###  4.1 Inverting Scene Layouts

We invert scene layouts of images from the COCO validation set; results are
shown in Figure 4. In all cases, the generated images reproduce the target
layout when passed to the detector. The inversions of different object
detectors share some intriguing features. Inversions show clear object
structures and category-specific details: shoes, pants, shirts, and faces for
people (col. 2); wheels, windows, car lights and traffic lights (col. 7);
canonical shapes of furniture (col. 5). These features imply that detectors
have learned these canonical traits as visual cues for detections.

invert\test |  Retina |  Faster |  Maskbox |  Maskmask |  FCOS
---|---|---|---|---|---
Retina |  91.6 |  59.4 |  55.5 |  17.5 |  61.0
Faster |  74.6 |  94.8 |  86.1 |  25.7 |  75.1
Mask-b |  69.4 |  86.3 |  94.3 |  26.8 |  69.3
Mask-m |  63.1 |  82.0 |  93.6 |  85.1 |  63.0
FCOS |  56.2 |  54.6 |  52.7 |  16.6 |  95.3

Table 1:  Inversion Transfer. We invert 1k layouts from COCO’17 val using
different object detectors; for Mask R-CNN we invert only boxes (Mask-b) and
boxes+masks (Mask-m) (similar to Figure 5). We run all generated images
through each detector and compute AP to see whether detectors recognize
objects similarly. Every row represents the detection results by different
detected models with the images inverted from the same network. Maskmask is AP
of instance segmentation using Mask R-CNN.

Moreover, the object structure seems context-dependent, and objects of the
same category have different modalities given different contexts. People have
different forms and actions under umbrellas (col. 2), on horses (col. 2) or
motorcycles (col. 6), next to a table (col. 4).

We compare inversions from different detectors to analyze visual cues learned
by differing models. Inverting layouts from Mask R-CNN more faithfully
reproduce the original image since its layouts include mask information.
However, inverting layouts with Faster R-CNN also results in clear object
boundaries and internal parts, such as the horse’s ear, eye, leg, and hoof
(col. 2), bicycle wheels and body (col. 6). It again implies that detectors
have a good understanding of shapes and internal structures of objects.

Inverting layouts with RetinaNet and FCOS gives qualitatively different
results from two-stage methods. Smaller objects show clear boundaries and
structures, but larger objects may instead appear as an amalgamation of parts.
We hypothesize that single-stage methods, especially RetinaNet, struggle to
model the global structures of large objects and resort to more local features
for detections, while RoI Align may help two-stage methods handle large
objects, resulting in explicit boundaries and structures. Besides superior
detection performance, inverting FCOS also shows finer results than RetinaNet,
implying better features.

Layout | Cls+Reg+Mask | Mask | Cls | Reg | Cls+Reg | Faster R-CNN
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
Figure 5:  Mask R-CNN produces three outputs per RoI: object classification
(cls); regression from RoI to output box (reg); and segmentation mask. We
perform layout inversion on Mask R-CNN where different subsets of the per-RoI
outputs must match the layout while don’t optimize the rest. We also show
results using Faster R-CNN, which matches the (cls) and (reg) outputs only for
comparison with cls+reg.

Sheep | Horse | Banana | Elephant
---|---|---|---
Image |  |  |
Grad-CAM [52] |  |  |
Mask [12] |  |  |
NormGrad [44] |  |  |

Figure 6:  We use attribution methods to find image regions responsible for
classification in Mask R-CNN’s second stage. We truncate Grad-CAM [52] results
to the RoI; for Mask [12] we preserve area equal to half the ground-truth
mask. See Supplementary for more examples on Mask R-CNN and other object
detectors.

###  4.2 Inversion Transfer.

We quantify our inversion and further measure the similarity between different
object detectors via _inversion transfer_. We invert a layout using one
detector, then pass the generated image to another detector and check whether
it also reproduces the layout. If two detectors similarly recognize objects or
learn similar visual cues, we should expect high inversion transfer. Table 1
evaluates inversion transfer using 1k layouts from COCO’17 val, following the
evaluation protocol of COCO.

Inverting and evaluating with the same model gives high but imperfect APs;
inversions may not be perfect for small objects, crowded scenes, and bad
annotated instances. In particular, imperfect APs come from box positions:
achieving 100 AP under the 0.95 test IoU threshold requires the inverted
bounding boxes to be exactly the same as annotated boxes for small objects at
pixel level. Besides, image regularizers used in visualization (such as blur,
jitter) for better recognition may also affect the AP number.

Inversions are highly transferable between Faster and Mask R-CNN, showing that
they use similar cues for recognition. Inversions are less transferable
between single (RetinaNet, FCOS) and two-stage (Faster and Mask R-CNN)
detectors, partially supported by the qualitative difference of visualizations
in Figure 4. Inversions are also less transferable between detection and
instance segmentation models: FCOS and RetinaNet have higher APs on inversions
from Faster R-CNN than from Mask R-CNN, and their inversions are better
detected by Faster R-CNN as well, implying distinct features learned for
instance segmentation task.

dx | dy | dx | dy
---|---|---|---
Image |  |  |
Grad-CAM [52] |  |  |
Mask [12] |  |  |
NormGrad [44] |  |  |

Figure 7:  We use attribution methods to find image regions responsible for
box regressions in Mask R-CNN’s second stage. See Supplementary for more
examples.

###  4.3 Disentangling Losses.

Layout inversion involves a combination of classification, bounding box
regression, and mask prediction (for Mask R-CNN). To understand distinct
visual cues learned for each task, in Figure 5, we disentangle losses by
performing layout inversion on Mask R-CNN using different combinations of
losses.

Inverting masks reproduces object boundaries but loses internal object
structure (bird eyes, feather textures) present in full layout inversions.
Inverting category predictions only gives object parts without clear
structures, similar to classification network visualization. Inverting box
regressions gives blobs in the correct locations that evoke coarse category-
specific shape (bird body and beak). Inverting class and box predictions with
Mask and Faster R-CNN give similar results; the former’s mask supervision
seems not to affect its qualitative understanding of object appearance.

By observations, we hypothesize that regression head is trained to detect
structures and boundaries of objects, while classification head relies more on
category-distinct visual cues like textures. We employ _attribution methods_
to validate this idea by pointing to the image regions responsible for a
model’s decision on input images. We adapt several popular attribution methods
developed for classification models to object detectors, showing attributions
of classification and regression in Figure 6 and Figure 7.

RetinaNet |  |  |  |
---|---|---|---|---
Faster |  |  |  |
Carrot | Person | Cat | Dog | Couch

Figure 8:  We generate category-specific images by optimizing the predictions
of RetinaNet and Faster R-CNN on a single anchor. Anchors are shown in blue,
and predicted object boxes in red.

---

Figure 9:  Detection results of individual object visualizations in Figure 8.
While only maximizing prediction scores on an anchor, detectable contexts
emerge with high probability.

Attribution methods reveal that classification head and regression head pay
attention to different regions, which supports our hypothesize. Classification
head focuses on local features like animal heads, hair textures. The
regression head has a better understanding of global structures and pays
category-specific attention to boundaries and objects’ corners. From this
perspective, we suggest the popular sibling head setting in object detector
(classification and regression heads share almost the same weights) may not be
the best choice since these two heads focus on different regions. We find our
hypothesis could be supported by some new object detector design of sibling
heads [55, 23, 63].

###  4.4 Visualizing Individual Objects.

Inverting entire scene layouts requires both recognition of foreground objects
and suppression of background regions. To further explore the category-
specific information learned by detectors, we visualize individual objects in
Figure 8. We optimize the image to maximize only a single anchor’s probability
of being classified as the chosen category without constraining its regressed
box position or any outputs from other anchors.

Visualizing individual objects in this way allows us to more precisely probe
the priors learned by detectors. In Section 4.4.1 we see that detectors learn
consistent contexts for objects, and in Section 4.4.2 we find that detectors
rely on different features to detect objects at different scales.

####  4.4.1 Object Context.

In Figure 8 we only maximize the classification score of a single anchor. As
expected, this causes the specified object to appear. Surprisingly other
background objects also consistently emerge: broccoli near the carrot; a bed
and couch below the dog and cat; and a potted plant near the couch.

Figure 10: We display the surrounding contexts with highest frequency while
visualizing individual car, cat, horse and keyboard object for RetinaNet.
Surrounding contexts show a consistent and category-specific pattern. See
supplementary for more data. Figure 11:  We show the frequency of relative
positions of contextual objects when visualizing people on RetinaNet.

These emergent _contextual objects_ are visible to detectors as well, as
demonstrated in Figure 9. This suggests that detectors may learn _motifs_ of
commonly occuring objects – e.g. that carrots occur in groups near other
vegetables – which may be important signals for recognition.

We quantify these motifs by scaling up the experiments of Figures 8 and 9. For
each category, we generate 1000 visualizations with a fixed anchor and
different random seeds. We feed these images back to the detector, and keep
track of other confidently detected objects ($>0.5$).

Contextual Frequency. For each category, we count the frequency with which
objects of other categories emerge. Figure 10 shows the six most frequently
apearing objects when generating images containing a person, carrot, dog, or
bed.222Instances of the same category are only counted once per image. We see
detectors learn contexts which are consistent but not universal: surfboards
appear near most but not all people. Contexts are category-specific, and may
be asymmetric: people emerge near $>20\%$ of beds, but beds rarely emerge near
people.

Relative Position. We further analyze not only _what_ contextual objects
emerge, but also _where_ they are located. We classify each emergent
contextual object as above/below and left/right of the central object based on
object centers. Figure 11 shows the frequency of contextual objects appearing
at different positions when visualizing people. The detector learns consistent
spatial trends: surfboards and boats appear below people, while birds and
umbrellas appear above people. Left/right biases may reveal photography bias
in the dataset (such as birds right of people).

One might expect detectors to be less context-dependent than classifiers since
they are trained for localization. However our experiments show that detectors
learn canonical patterns of co-occurring objects, including common spatial
positions. It seems likely that detectors also use these common contexts to
help recognize objects, which accords with the research of objects and their
context in early vision literature [40, 43]. We show context statistic in
supplementary.

Scale 1 |  |  |  |
---|---|---|---|---
Scale 2 |  |  |  |
Scale 3 |  |  |  |
Scale 4 |  |  |  |
Scale 5 |  |  |  |
Person | Car | Dog | Cat | Horse

Figure 12:  We optimize the predictions of Faster R-CNN (col 1-3) and
RetinaNet (col 4-5) on a single anchor (like Figure 8) at different anchor
scales. Anchors are shown in blue and predicted boxes in red. For both
detectors, along with scale increasing, objects gradually lose global
structures and show disconnected parts.

####  4.4.2 Varying Object Scale.

Unlike classification models, detectors are expected to recognize objects of
different scales, often with Feature Pyramid Network (FPN). In Detectron2
settings, objects are categorized into scale 1-5 with object areas of
$\\{32^{2},64^{2},128^{2},256^{2},512^{2}\\}$. To this end, it is of great
interest to visualize individual objects of different scales. We show such
visualization in Figure 12.

We see that detectors rely on qualitatively different cues when recognizing
the same class at different scales. Small objects (scales 1 and 2) rely
heavily on context and possess good global structures: dogs are near people,
and cars are on roads or next to houses. Medium-sized objects (scale 3) rely
on context (e.g. car behind motorcycle) and also have clear structures: dogs
have faces and legs; cars have wheels and windows. Large objects (scale 4 and
especially 5) have little coherent context or structure; detectors seem to
recognize them disconnected parts or textures.

(a) Original Image

(b) Probability change vs Scale

(c) Masked stylized image

(d) IoU change vs Scale

Figure 13:  We eliminate object texture by performing masked style transfer.
We measure how this changes detectors’ predicted probability and IoU with
ground-truth as a function of object scale.

The qualitative difference between different scales indicates the shifting of
visual cues for object detectors: from relying on global shape and context at
small scales, to large structure and texture at large scales. This may help
explain texture-bias in ImageNet-trained classification models [13], since
most objects in ImageNet are large.

Inspired by [13], we use masked style transfer to quantify the reliance of
object detectors on texture. As shown in Figure 13, we use an object’s ground-
truth mask to perform style transfer keeping background pixels unchanged,
obfuscating the object’s texture. We pass the original and stylized images to
RetinaNet and Faster R-CNN and record the change in predicted probability and
box location. We repeat for all objects in the `val2017` split and plot the
results as a function of object scale.

As object scale increases, prediction probability for stylized objects drops
sharply. This indicates that detectors rely more on texture for large objects
than for small objects.

Comparing IoU and probability changes after stylization, probability is
decreased significantly with stylization for objects of all scales, while IoU
is less affected, and even slightly improved. This supports our claim in
Section 4.3 that classification and regression heads rely on different
features; specifically texture is less important for the regression head. The
marginal improvement in IoU may be due to sharper boundaries caused by
stylization.

Comparing the detectors, Faster R-CNN has smaller drops in probability and
larger increases in IoU, suggesting that Faster R-CNN relies less on texture
and more on shape than RetinaNet, though the effect is small. This helps
explain the layout inversions in Figure 4, where RetinaNet struggles with
boundaries of large objects.

##  5 Conclusion

In this paper, we extend visualization to modern object detector and propose
an optimization-based approach for layout inversion. Our visualization reveals
intriguing properties of object detectors. We hope these insights can help
practitioners diagnose and improve object detectors.

## References

  * [1] Oron Ashual and Lior Wolf.  Specifying object attributes and relations in interactive scene generation.  In ICCV, 2019.
  * [2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network dissection: Quantifying interpretability of deep visual representations.  In CVPR, 2017.
  * [3] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba.  Gan dissection: Visualizing and understanding generative adversarial networks.  arXiv preprint arXiv:1811.10597, 2018.
  * [4] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  YOLOv4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [5] Stephen Boyd, Neal Parikh, and Eric Chu.  Distributed optimization and statistical learning via the alternating direction method of multipliers.  Now Publishers Inc, 2011.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  arXiv preprint arXiv:2005.12872v2, 2020.
  * [7] Qifeng Chen and Vladlen Koltun.  Photographic image synthesis with cascaded refinement networks.  In ICCV, 2017.
  * [8] Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In CVPR, 2005.
  * [9] Alexey Dosovitskiy and Thomas Brox.  Inverting visual representations with convolutional networks.  In CVPR, 2016.
  * [10] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In ICCV, 2019.
  * [11] Pedro Felzenszwalb, David McAllester, and Deva Ramanan.  A discriminatively trained, multiscale, deformable part model.  In CVPR, 2008.
  * [12] Ruth Fong, Mandela Patrick, and Andrea Vedaldi.  Understanding deep networks via extremal perturbations and smooth masks.  In ICCV, 2019.
  * [13] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  arXiv preprint arXiv:1811.12231, 2018.
  * [14] Ross Girshick.  Fast R-CNN.  In ICCV, 2015.
  * [15] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In CVPR, 2014.
  * [16] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh R-CNN.  In ICCV, 2019.
  * [17] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5356–5364, 2019.
  * [18] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik.  Simultaneous detection and segmentation.  In ECCV, 2014.
  * [19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In ICCV, 2017.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  TPAMI, 2015.
  * [21] Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al.  Speed/accuracy trade-offs for modern convolutional object detectors.  In CVPR, 2017.
  * [22] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In CVPR, 2017.
  * [23] Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang.  Acquisition of localization confidence for accurate object detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 784–799, 2018.
  * [24] Justin Johnson, Agrim Gupta, and Li Fei-Fei.  Image generation from scene graphs.  In CVPR, 2018.
  * [25] Andrej Karpathy, Justin Johnson, and Li Fei-Fei.  Visualizing and understanding recurrent networks.  arXiv preprint arXiv:1506.02078, 2015.
  * [26] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In ECCV, 2018.
  * [27] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural computation, 1989.
  * [28] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky.  Visualizing and understanding neural models in nlp.  In NAACL, 2015.
  * [29] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In ICCV, 2017.
  * [30] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [31] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In ECCV, 2014.
  * [32] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  SSD: Single shot multibox detector.  In ECCV, 2016.
  * [33] Aravindh Mahendran and Andrea Vedaldi.  Understanding deep image representations by inverting them.  In CVPR, 2015.
  * [34] Aravindh Mahendran and Andrea Vedaldi.  Visualizing deep convolutional neural networks using natural pre-images.  IJCV, 2016.
  * [35] Alexander Mordvintsev, Christopher Olah, and Mike Tyka.  Inceptionism: Going deeper into neural networks, 2015.  URL https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural., 2015\.
  * [36] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.  Plug & play generative networks: Conditional iterative generation of images in latent space.  In CVPR, 2017.
  * [37] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  In NeurIPS, 2016.
  * [38] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In CVPR, 2015.
  * [39] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks.  arXiv preprint arXiv:1602.03616, 2016.
  * [40] Aude Oliva and Antonio Torralba.  The role of context in object recognition.  Trends in cognitive sciences, 11(12):520–527, 2007.
  * [41] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In CVPR, 2019.
  * [42] Vitali Petsiuk, Abir Das, and Kate Saenko.  Rise: Randomized input sampling for explanation of black-box models.  arXiv preprint arXiv:1806.07421, 2018.
  * [43] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos, Eric Wiewiora, and Serge Belongie.  Objects in context.  In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
  * [44] Sylvestre-Alvise Rebuffi, Ruth Fong, Xu Ji, and Andrea Vedaldi.  There and back again: Revisiting backpropagation saliency methods.  arXiv preprint arXiv:2004.02866, 2020.
  * [45] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In CVPR, 2016.
  * [46] Joseph Redmon and Ali Farhadi.  YOLO9000: better, faster, stronger.  In CVPR, 2017.
  * [47] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [48] Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando De Freitas.  Parallel multiscale autoregressive density estimation.  In ICML, 2017.
  * [49] Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee.  Learning what and where to draw.  In NeurIPS, 2016.
  * [50] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [51] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 658–666, 2019.
  * [52] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual explanations from deep networks via gradient-based localization.  In ICCV, 2017.
  * [53] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [54] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [55] Guanglu Song, Yu Liu, and Xiaogang Wang.  Revisiting the sibling head in object detector.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11563–11572, 2020.
  * [56] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for simplicity: The all convolutional net.  arXiv preprint arXiv:1412.6806, 2014.
  * [57] Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush.  Lstmvis: A tool for visual analysis of hidden state dynamics in recurrent neural networks.  IEEE transactions on visualization and computer graphics, 2017.
  * [58] Wei Sun and Tianfu Wu.  Image synthesis from reconfigurable layout and style.  In ICCV, 2019.
  * [59] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  FCOS: Fully convolutional one-stage object detection.  In ICCV, 2019.
  * [60] Régis Vaillant, Christophe Monrocq, and Yann Le Cun.  Original approach for the localisation of objects in images.  IEE Proceedings-Vision, Image and Signal Processing, 1994.
  * [61] Carl Vondrick, Aditya Khosla, Tomasz Malisiewicz, and Antonio Torralba.  Hoggles: Visualizing object detection features.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1–8, 2013.
  * [62] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In CVPR, 2018.
  * [63] Yue Wu, Yinpeng Chen, Lu Yuan, Zicheng Liu, Lijuan Wang, Hongzhi Li, and Yun Fu.  Rethinking classification and localization for object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10186–10195, 2020.
  * [64] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2, 2019.
  * [65] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.  Understanding neural networks through deep visualization.  arXiv preprint arXiv:1506.06579, 2015.
  * [66] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In ECCV, 2014.
  * [67] Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal.  Image generation from layout.  In CVPR, 2019.
  * [68] Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba.  Interpreting deep visual representations via network dissection.  TPAMI, 2018.
  * [69] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Object detectors emerge in deep scene CNNs.  arXiv preprint arXiv:1412.6856, 2014.
  * [70] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In CVPR, 2016.
  * [71] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.
  * [72] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, 2017.


$\tilde{n}^{\ell}_{A}(i_{j})=\lvert\\{i_{k}\in\mathcal{N}^{L}_{A}\left(P_{\bm{\alpha}}\right)\,|\,i_{k}>i_{j}\,\\}\rvert+\lvert\\{i_{m}\in\mathcal{N}^{L}_{A}\left([P_{\ell,i_{j}},P_{\bm{\alpha}}]\right)\,|\,i_{m}<i_{j}\,\\}\rvert$
(58)

Indeed, it counts all the Pauli gates in $\tilde{W}_{\ell}$ that anti-commute
with $P_{\bm{\alpha}}$ if they are located after $P_{\ell,i_{j}}$, and those
that anti-commute with $[P_{\ell,i_{j}},P_{\bm{\alpha}}]$ if they are located
before $P_{\ell,i_{j}}$ (see Figure 15).

As $\mathbb{E}_{\bm{\Theta}}[(\mathcal{L}_{\bm{\alpha}}^{(L)})^{2}]$ depends
on $P_{\bm{\alpha}}$ and all possible nested commutators including $P_{L,i}$,
it is extremely complicated to generalize the exact equation. Therefore, for
the general case, we will compute an exact lower bound depending on $\sigma$.
However, this bound does not provide much information about the scaling with
respect to the system’s size without further assumption on the circuit
architecture. In Appendix C.3, we consider an EfficientSU2 architecture as a
specific example and provide some approximate bounds which show the polynomial
scaling of the variance for the leading order in $\sigma$. These bounds,
despite being approximate, are rather tight as supported by our numerics.

First of all, let us consider the case with $\sigma\ll 1$. As
$e^{-2\sigma^{2}}$ is close to $1$, $\frac{1+e^{-2\sigma^{2}}}{2}$ dominates
over $\frac{1-e^{-2\sigma^{2}}}{2}$, and therefore, the first term will mainly
contribute in the lower bound :

$\mathbb{E}_{\bm{\Theta}}[(\mathcal{L}_{\bm{\alpha}}^{(L)})^{2}]>\big{(}\frac{1+e^{-2\sigma^{2}}}{2}\big{)}^{n_{A}^{L}\left(P_{\bm{\alpha}}\right)}\mathbb{E}_{\bm{\Theta}}[(\mathcal{L}^{(L-1)}_{\bm{\gamma}})^{2}]>\big{(}\frac{1+e^{-2\sigma^{2}}}{2}\big{)}^{N_{A}^{L}}\mathbb{E}_{\bm{\Theta}}[(\mathcal{L}^{(0)}_{\bm{\eta}})^{2}],$
(59)

where we obtain the last inequality by induction. Combining Eq. (42) and Eq.
(59), we have the lower bound for the variance
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\alpha}}^{(L)}]$ :

$\text{Var}[\mathcal{L}_{\bm{\alpha}}]=\mathbb{E}[(\mathcal{L}_{\bm{\alpha}})^{2}]-\mathbb{E}[\mathcal{L}_{\bm{\alpha}}]^{2}\geq\Big{(}\frac{1+e^{-2\sigma^{2}}}{2}\Big{)}^{N^{L}_{A}}\mathbb{E}_{\bm{\Theta}}[(\mathcal{L}^{(0)}_{\bm{\eta}})^{2}]-(e^{-\sigma^{2}})^{N^{L}_{A}}\mathbb{E}_{\bm{\Theta}}[(\mathcal{L}^{(0)}_{\bm{\eta}})]^{2}.$
(60)

In particular, if there exist a $\bm{\lambda}^{*}$ such that
$\eta_{j}=\lambda_{j}$ for all $\eta_{j}\neq 0$, we have :

$\text{Var}[\mathcal{L}_{\bm{\alpha}}]=\mathbb{E}[(\mathcal{L}_{\bm{\alpha}})^{2}]-\mathbb{E}[\mathcal{L}_{\bm{\alpha}}]^{2}>c_{\bm{\lambda}^{*}}^{2}\Big{(}\frac{1+e^{-2\sigma^{2}}}{2}\Big{)}^{N^{L}_{A}+w(P_{\bm{\eta}})}-c_{\bm{\lambda}^{*}}(e^{-\sigma^{2}})^{N^{L}_{A}+w(P_{\bm{\eta}})},$
(61)

otherwise, $\mathbb{E}_{\bm{\Theta}}[\mathcal{L}^{(L)}_{\bm{\eta}}]=0$, and
thus,

$\text{Var}[\mathcal{L}_{\bm{\alpha}}]=\mathbb{E}[(\mathcal{L}_{\bm{\alpha}})^{2}]>\sum_{\bm{\lambda}}c_{\bm{\lambda}}^{2}\Big{(}\frac{1+e^{-2\sigma^{2}}}{2}\Big{)}^{N^{L}_{A}+\sum_{j\in\mathcal{J_{\bm{\eta}}}}\delta_{\eta_{j}\lambda_{j}}}\Big{(}\frac{1-e^{-2\sigma^{2}}}{2}\Big{)}^{\sum_{j\in\mathcal{J_{\bm{\eta}}}}\bar{\delta}_{\eta_{j}\lambda_{j}}}.$
(62)

On the other hand, if $\sigma\gg 0$, the identity initialization is not valid
anymore, and a $\text{poly}(n)$ depth unstructured circuit is sufficiently
expressive to induce exponential concentration of the loss function.

###  C.3 Case study: Absence of barren plateau in EfficientSU2 ansatz

(a) Pairwise entanglement (b) Circular entanglement

Figure 16: Circuit architecture of the EfficientSU2 ansatz [101]. The circuit
is initialized with two orthogonal layers of single-qubit Pauli rotation gates
and $L$ layers of entanglement layers with CNOT gates and single-qubit
rotation gates. The entanglement gates can have either (a) a pairwise
structure, or (b) a circular structure.

In this section, we will analyze the decay of the gradient variance in the
EfficientSU2 ansatz shown on Figure 16, which consists only of single-qubit
Pauli rotations and CNOT gates. The single-qubit rotations can be chosen in
different combinations, and the entanglement map can be either pairwise or
circular. Although we only use the EfficientSU2 ansatz for the study, the
result can also be generalized to other types of quantum circuits. Our results
show that with small $\sigma$ the variance of the loss scales polynomially
with the number of qubits. We note that while we rely on some approximations
to obtain these approximate lower bounds of the variance (i.e., the bounds are
written in the leading order of $\sigma$), these bounds are rather tight, as
supported by our numerics.

We will analyze how the loss function behaves depending on the type of
observables and the entanglement map for both local and global observables.
Let us start with two different local observables
$P_{\bm{\alpha}}=\sigma_{\alpha_{0}}\otimes\mathbb{I}^{\otimes n-1}$ and
$P_{\bm{\beta}}=\sigma_{\beta_{0}}\otimes\mathbb{I}^{\otimes n-1}$ such that
$\alpha_{0}\neq\beta_{0}$. Furthermore, we assume that $\rho$ only consists of
$P_{\bm{\lambda}}$ such that $\lambda_{j}\in\\{0,\alpha_{0}\\}$ for all
$\bm{\lambda}$. This is the case if we have zero initial state
$\rho=\ket{0}\bra{0}^{\otimes n}=(\mathbb{I}+Z)^{\otimes n}/2^{n}$ with
$\sigma_{\alpha_{0}}=Z$ and $\sigma_{\beta_{0}}=X$.

Table 5 summarizes $\mathcal{L}_{\bm{\alpha}}$ and $\mathcal{L}_{\bm{\beta}}$
for all possible combinations of $\mu_{0}$ and $\nu_{0}$ in $L=0$ case.

| $\mathcal{L}_{\alpha_{0}}$ | $\mathcal{L}_{\beta_{0}}$ | $\mathbb{E}[\mathcal{L}_{\alpha_{0}}]$ | $\mathbb{E}[\mathcal{L}_{\beta_{0}}]$ | $\mathbb{E}[\mathcal{L}_{\alpha_{0}}^{2}]$ | $\mathbb{E}[\mathcal{L}_{\beta_{0}}^{2}]$ | $\mathbb{E}[\mathcal{L}_{\alpha_{0}}\mathcal{L}_{\beta_{0}}]$
---|---|---|---|---|---|---|---
$\alpha_{0}=\nu_{0},\beta_{0}=\mu_{0}$ | $\cos(\omega_{j})$ | $\sin(\omega_{j})\sin(\phi_{j})$ | $e^{-\frac{\sigma^{2}}{2}}$ | 0 | $\frac{1+e^{-2\sigma^{2}}}{2}$ | $\left(\frac{1-e^{-2\sigma^{2}}}{2}\right)^{2}$ | 0
$\alpha_{0}=\nu_{0},\beta_{0}\neq\mu_{0}$ | $\cos(\omega_{j})$ | $\sin(\omega_{j})\cos(\phi_{j})$ | $e^{-\frac{\sigma^{2}}{2}}$ | 0 | $\frac{1+e^{-2\sigma^{2}}}{2}$ | $\frac{1-e^{-4\sigma^{2}}}{4}$ | 0
$\alpha_{0}\neq\nu_{0},\beta_{0}=\mu_{0}$ | $\cos(\omega_{j})\cos(\phi_{j})$ | $\cos(\omega_{j})\sin(\phi_{j})$ | $e^{-\sigma^{2}}$ | 0 | $\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)^{2}$ | $\frac{1-e^{-4\sigma^{2}}}{4}$ | 0
$\alpha_{0}=\mu_{0},\beta_{0}=\nu_{0}$ | $\cos(\phi_{j})$ | $0$ | $e^{-\frac{\sigma^{2}}{2}}$ | 0 | $\frac{1+e^{-2\sigma^{2}}}{2}$ | $\frac{1-e^{-2\sigma^{2}}}{2}$ | 0
$\alpha_{0}\neq\mu_{0},\beta_{0}=\nu_{0}$ | $\cos(\phi_{j})\cos(\omega_{j})$ | $\sin(\omega_{j})$ | $e^{-\sigma^{2}}$ | 0 | $\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)^{2}$ | $\frac{1-e^{-2\sigma^{2}}}{2}$ | 0
$\alpha_{0}=\mu_{0},\beta_{0}\neq\nu_{0}$ | $\cos(\phi_{j})$ | $\sin(\phi_{j})$ | $e^{-\frac{\sigma^{2}}{2}}$ | 0 | $\frac{1+e^{-2\sigma^{2}}}{2}$ | $\frac{1-e^{-4\sigma^{2}}}{4}$ | 0
Table 5: Summary of the loss functions $\mathcal{L}_{\alpha}$ and
$\mathcal{L}_{\beta}$ for different circuit architectures. The covariance
$\mathbb{E}_{\bm{\Theta}}\left[\mathcal{L}_{\alpha_{0}}\mathcal{L}_{\beta_{0}}\right]$
vanish for all cases as $\mathcal{L}_{\alpha_{0}}\mathcal{L}_{\beta_{0}}$ is
always odd with respect to $\omega_{j}$ or $\phi_{j}$ over $\mathbb{R}$.

As the table shows, the covariance between $\mathcal{L}_{\bm{\alpha}}$ and
$\mathcal{L}_{\bm{\beta}}$ vanishes, as assumed in Appendix C.2 and thus, Eq.
(44) holds.

It is more straightforward to show the lower bound of
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\alpha}}]$, as the
$\mathbb{E}_{\bm{\Theta}}[\mathcal{L}^{(0)}_{\bm{\alpha}}]$ scales following
the dominant term, $(1+e^{-2\sigma^{2}})/{2}$. As EfficientSU2 ansatz only
consists of CNOT gates among the Clifford gates, the Pauli strings
$C^{\ell\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell}$, in particular,
$C^{1\dagger}_{L}P_{\bm{\alpha}}C^{L}_{1}=P_{\bm{\eta}}$ , only consists of
$\sigma_{\alpha_{0}}$. Furthermore, the ansatz only contains single-qubit
Pauli rotation gates, hence, the number of anti-commuting Pauli operators,
$n^{\ell}_{A}\left(C^{\ell+1\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell+1}\right)$
will be proportional to the weight
$w\left(C^{\ell+1\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell+1}\right)$ of the
Pauli string at each layer $\ell$ and, as a result, the total number of anti-
commuting gates will be proportional to the sum of weights,
$N^{L}_{A}\approx\sum_{\ell=1}^{L}w\left(C^{\ell+1\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell+1}\right)$.

As $N_{A}^{L}$ depends on the position of the observable and the types of the
entanglement map, it is complex to find a general formula. Therefore, we will
assume that the non-trivial Pauli matrices can span over $n$-qubits for
$C^{\ell\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell},\ell=1,...,n$ , i.e. for all
$d\in\\{1,...,n\\}$, there exists an $\ell\in\\{1,...,n\\}$ such that
$w(C^{\ell\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell})=d$. Under this assumption,
we take the average over $w(C^{\ell\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell})$
to have $w(P_{\bm{\eta}})\approx n/2$ and $N^{L}_{A}\approx(n^{2}+n)/2$. In
addition, we also consider the extreme-case scenario for the polynomial depth
circuit where the $N^{L}_{A}\approx Ln=n^{2}$ , and $w(P_{\bm{\eta}})=n$. For
$\sigma\ll 1$ , Eq. (61) and Table 5 lead to :

$\displaystyle\text{Var}[\mathcal{L}_{\bm{\alpha}}]$
$\displaystyle\geq\min\left[c_{\bm{\lambda}^{*}}^{2}\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)^{\frac{n^{2}}{2}+n}-c_{\bm{\lambda}^{*}}\left(e^{-\sigma^{2}}\right)^{\frac{n^{2}}{2}+n},c_{\bm{\lambda}^{*}}^{2}\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)^{n^{2}+n}-c_{\bm{\lambda}^{*}}\left(e^{-\sigma^{2}}\right)^{n^{2}+n}\right]\;,$
(63)

with $\bm{\lambda}^{*}$ such that $\bm{\lambda}^{*}=\bm{\eta}$. In particular,
in the case of the zero state, the equation simplifies with
$c_{\bm{\lambda}^{*}}=1$. It is important to consider both cases, because the
interplay between
$\mathbb{E}_{\bm{\Theta}}\left[\mathcal{L}_{\bm{\alpha}}\right]^{2}$ and
$\mathbb{E}_{\bm{\Theta}}\left[(\mathcal{L}_{\bm{\alpha}})^{2}\right]$ depends
greatly on the value of $\sigma$ and $n$. Taking into account only the extreme
case will make the calculation deviate too much from the real lower bound.

We also compute the scaling of the variance while assuming the initial zero
state. Taking a Taylor expansion with respect to $\sigma$ around $0$ gives us:

$\displaystyle\text{Var}[\mathcal{L}_{\bm{\alpha}}]$
$\displaystyle>(1-\sigma^{2}+\sigma^{4})^{n^{2}+n}-(1-\sigma^{2}+\frac{\sigma^{4}}{2})^{n^{2}+n}$
(64)
$\displaystyle>\frac{1}{2}n(n+1)\sigma^{4}-\frac{1}{2}(n^{2}(n+1))^{2}\sigma^{6}$
(65) $\displaystyle\approx\frac{1}{2}n^{2}\sigma^{4}(1-n^{2}\sigma^{2})$ (66)
$\displaystyle>\frac{1}{n^{b}}\;.$ (67)

with a $b>1$ independent of $n$, which implies that
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\alpha}}]$ decays polynomially.
Rearranging Eq. (67), we can conclude that
$\text{Var}_{\bm{\Theta}}\left[\mathcal{L}_{\bm{\alpha}}\right]$ will scale as
$\mathcal{O}(1/n^{b})$ with $b>2$ if :

$\sigma\in\Theta\left(\frac{1}{n}\right)\;.$ (68)

Note that the
$\max\left(w\left(C^{\ell\dagger}_{L}P_{\bm{\alpha}}C^{L}_{\ell}\right)\right)=n$
and $\max\left(N^{L}_{A}\right)=n^{2}$, and thus, the lower bound given by Eq.
(63) also applies to the global observable case with
$P_{\bm{\alpha}}=\sigma_{\alpha_{0}}^{\otimes n}$ as shown on Figure 18. On
Figure 17 and 18, we display $\text{Var}[\mathcal{L}^{(L)}_{\bm{\alpha}}]$ for
$L=n$ and its lower bound calculated with Eq. (63) with respect to the number
of qubits. Furthermore, for $\sigma\geq 1$, we take the lower bound as
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\alpha}}]=2^{-n}$ as justified in
Appendix C.2. The figures clearly show that the variance follows the computed
lower bound for both local and global observables in the case of $\sigma\ll 1$
while it decays exponentially for large $\sigma$.

Figure 17: Variance of $\mathcal{L}_{\bm{\alpha}}$ for the local Z observable
and its lower bound in polynomial depth circuit with different initialization
ranges. The blue cross represents the simulated variance obtained in
EfficientSU2 ansatz with $\text{poly}(n)$ depth and the gray dashed line its
lower bound calculated with Eq. (63).

(a) Pairwise entanglement

(b) Circular entanglement

Figure 18: Variance of $\mathcal{L}_{\bm{\alpha}}$ for the global Z observable
and its lower bound in the polynomial depth circuit with different
initialization ranges. The blue cross represents the simulated variance
obtained in the EfficientSU2 ansatz with $\text{poly}(n)$ depth and the gray
dashed line its lower bound calculated with Eq. (63). We observe that
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\alpha}}]$ for both (a) pairwise
and (b) circular entanglement also follows the theoretical lower bound, and
does not exhibit a barren plateau with an appropriate choice of
initialization.

On the other hand, understanding the variance of
$\mathbb{E}_{\bm{\Theta}}[\mathcal{L}_{\bm{\beta}}]$ requires additional
insight due to the presence of sine terms. As shown on Table 5,
$\mathbb{E}_{\bm{\Theta}}[\mathcal{L}_{\bm{\beta}}]$ vanishes regardless of
the ansatz, thus it suffices to find the scaling of
$\mathbb{E}_{\bm{\Theta}}[\mathcal{L}_{\bm{\beta}}^{2}]$. We start by
rewriting Eq. (57) for the EfficientSU2 ansatz with $K_{\ell}=2$ (c.f. Figure
16) as follows :

$\displaystyle\mathbb{E}_{\bm{\Theta}}\left[\left(\mathcal{L}^{(L)}_{\bm{\beta}}\right)^{2}\right]=$
$\displaystyle\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)^{2}\mathbb{E}_{\bm{\Theta}}\left[\left(\mathcal{L}^{(L-1)}_{\bm{\gamma}^{\prime}}\right)^{2}\right]+\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)\left(\frac{1-e^{-2\sigma^{2}}}{2}\right)\mathbb{E}_{\bm{\Theta}}\left[\left(\mathcal{L}^{(L-1)}_{\bm{\delta}_{1}^{\prime}}\right)^{2}\right]$
$\displaystyle+\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)\left(\frac{1-e^{-2\sigma^{2}}}{2}\right)\mathbb{E}_{\bm{\Theta}}\left[\left(\mathcal{L}^{(L-1)}_{\bm{\delta}_{2}^{\prime}}\right)^{2}\right]+\mathcal{O}\left(\frac{1-e^{-2\sigma^{2}}}{2}\right)^{2}$
(69)

where we denote $P_{\bm{\gamma}^{\prime}}=C^{\dagger}P_{\bm{\beta}}C$,
$P_{\bm{\delta}_{1}^{\prime}}=-iC^{\dagger}\frac{1}{2}[P_{L,1},P_{\bm{\beta}}]C$
and
$P_{\bm{\delta}_{2}^{\prime}}=-iC^{\dagger}\frac{1}{2}[P_{L,2},P_{\bm{\beta}}]C$.

Without losing generality, we assume that
$[P_{L,i_{j}},P_{\bm{\beta}}]=2i\sigma_{\alpha_{0}}\otimes\mathbb{I}^{n-1}$
for $i_{j}\in\\{1,2\\}$. This corresponds to the case of an $X$ observable
with RZ or RY rotations in $\tilde{W}_{\ell}$. This leads
$\mathbb{E}_{\bm{\Theta}}\left[\left(\mathcal{L}^{(L-1)}_{\bm{\delta}_{i_{j}}^{\prime}}\right)^{2}\right]$
back to the previous case for $\mathcal{L}_{\bm{\alpha}}$, scaling as
$((1+e^{-2\sigma^{2}})/2)^{n^{2}+n}$. On the other hand, since
$P_{{\bm{\gamma}}^{\prime}}$ only contains $\sigma_{\beta_{0}}$, in the worst
case scenario, we have $w(C^{\ell\dagger}_{L}P_{\bm{\beta}}C^{L}_{\ell})=n$,
leading to the the first term scaling as $((1-e^{-2\sigma^{2}})/2)^{n}$.
Therefore, the second term will be the dominant term in case of the small
angle initialization and we can conclude that :

$\displaystyle\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\beta}}^{2}]$
$\displaystyle>c_{\bm{\lambda}^{**}}\left(\frac{1+e^{-2\sigma^{2}}}{2}\right)^{n^{2}+n}\left(\frac{1-e^{-2\sigma^{2}}}{2}\right),$
(70)

with $c_{\bm{\lambda}^{**}}$ such that
$P_{\bm{\lambda}^{**}}=C^{1\dagger}_{L}[P_{L,i_{j}},P_{\bm{\beta}}]C^{L}_{1}$.
The equation is confirmed with Figure 19 showing
$\text{Var}[\mathcal{L}^{(L)}_{\bm{\beta}}]$ for $L=n$ and its lower bound
calculated with Eq. (70).

Figure 19: Variance of $\mathcal{L}_{\bm{\beta}}$ for the local X observable
and its lower bound in polynomial depth circuit with different initialization
ranges. The blue cross represents the simulated variance obtained in
EfficientSU2 ansatz with $\text{poly}(n)$ depth and the pairwise entanglement
map. The gray dashed line corresponds to the lower bound calculated with Eq.
(70).

For the zero initial state, by taking the Taylor expansion with respect to
$\sigma$ around $0$, Eq. (70) can be approximated as :

$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\beta}}^{2}]>(1-\sigma^{2})^{n^{2}+n}\sigma^{2}>\sigma^{2}-(n^{2}+n)\sigma^{4}>\sigma^{2}-2n^{2}\sigma^{4}>\frac{1}{n^{b}},$
(71)

resulting in the same conclusion as before, that
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\beta}}^{2}]$ decays as
$\mathcal{O}(1/n^{2})$ if $\sigma\in\Theta(1/n)$.

(a) Pairwise entanglement

(b) Circular entanglement

Figure 20: Variance of $\mathcal{L}_{\bm{\beta}}$ for the global X observable
and its lower bound in the polynomial depth circuit with different
initialization ranges. The blue cross represents the simulated variance
obtained in the EfficientSU2 ansatz with $\text{poly}(n)$ depth and the gray
dashed line its lower bound calculated with Eq. (70). We observe that
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\beta}}]$ decays exponentially for
the (a) pairwise entanglement independent of $\sigma$, while it follows the
theoretical lower bound given for (b) circular entanglement. This discrepancy
comes from the fact that $C^{\ell\dagger}_{L}P_{\bm{\beta}}C^{\ell}_{L}$
behaves differently depending on the entanglement map.

Unlike the Z observable, which exhibits similar behavior in both the global
and local observables, the behavior of the global X observable varies
depending on the type of entanglement. In Figure 20, we observe that the
variance decays exponentially regardless of $\sigma$ with the pairwise
entanglement map. However, with the circular entanglement map, it follows the
lower bound given by Eq. (70), which was computed for the local observable.

Although it is complicated to justify the result mathematically, we can
explain this difference by analyzing
$C^{\ell\dagger}_{L}P_{\bm{\beta}}C^{L}_{\ell}$. For the circular entanglement
map, it is straightforward to see that
$P_{\bm{\gamma}^{\prime}}=C^{\dagger}X^{\otimes
n}C=\mathbb{I}X\mathbb{I}^{\otimes(n-2)}$. This brings us back to the case of
the local X observable with $\mathcal{L}^{(L-1)}_{\bm{\gamma}^{\prime}}$ in
Eq. (69), resulting in the same lower bound. Conversely, for the pairwise
entanglement map, we have $P_{\bm{\gamma}^{\prime}}=C^{\dagger}X^{\otimes
n}C=(\mathbb{I}X)^{\otimes n/2}$, introducing an additional contribution of
$((1-e^{-2\sigma^{2}})/2)^{n}$ in Eq. (70). Indeed, we can easily find that
$w(C^{\ell\dagger}_{L}P_{\bm{\beta}}C^{L}_{\ell})=n$ or $n/2$ for all
$\ell=1,...,L$. Therefore, this higher weight of the X observable leads to the
exponential decay of the variance.

Figure 21 shows the variance of the loss function
$\text{Var}_{\bm{\Theta}}[\mathcal{L}_{\bm{\alpha}}]$ of EfficientSU2 ansatz
with $\text{poly}(n)$ depth versus the number of qubits $n$ using the
initialization range varying as $\sigma=1/n$ for Z and X observables. As
expected, the loss function scales as $\mathcal{O}(1/n^{b})$ with $b>2$,
clearly proving the mitigation of the barren plateau in the polynomial depth
circuit.

(a) Local Z

(b) Global Z

(c) Local X

Figure 21: Variance of the loss function with respect to $n$ using
$\sigma=1/n$. The blue cross corresponds to the simulated result, the dashed
line its polynomial fit, and the green points the theoretical bounds computed
with (a), (b) Eq. (63) and (c) Eq. (70). As predicted, the variance decays as
$\mathcal{O}(1/n^{2})$ with $\sigma$ scaling as $1/n$.

##  Appendix D Study of BP with Other Circuits

Previously, in Section V, we only displayed the results for the variance of
the gradients calculated with Circuit1 (see Figure 3) as a quantum generator
in LaSt-QGAN . In this section, we compute the variance for other types of
circuits to confirm that this absence of barren plateau is not only limited to
a specific type of circuit. As shown on Figure 22, we observe that
$\text{Var}_{\bm{\Theta},\bm{\phi}}[\partial_{\nu}\mathcal{L}_{G}]$ decays
polynomially with a small angle initialization for other circuits as well.
Notably, the varying $\sigma=1/n$ found in Appendix C.3 ensures a polynomial
decay, with the slope decreasing as the system size increases.

(a) EfficientSU2

(b) Circuit3

Figure 22: Variance of the partial derivative of $\mathcal{L}_{G}$ versus the
number of qubits $n$ using polynomial depth quantum generator with different
circuit architecture in LaSt-QGAN . The variance is computed with
$\mathcal{D}_{\mathbf{z}}=n$ for different initialization ranges, $\delta$,
and averaged over the parameters of the first layer. The quantum generator
consists of different circuits presented in Figure 3 with polynomial depth,
$d=\lfloor\log(n)\rfloor$.