# Mind the Backbone: Minimizing Backbone Distortion for
Robust Object Detection

Kuniaki Saito1, Donghyun Kim2, Piotr Teterwak1, Rogerio Feris2, Kate Saenko1,3
1Boston University, 2MIT-IBM Watson AI Lab , 3Meta AI
{keisaito, piotrt<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Building object detectors that are robust to domain shifts is critical for
real-world applications. Prior approaches fine-tune a pre-trained backbone and
risk overfitting it to in-distribution (ID) data and distorting features
useful for out-of-distribution (OOD) generalization. We propose to use
Relative Gradient Norm (RGN) as a way to measure the vulnerability of a
backbone to feature distortion, and show that high RGN is indeed correlated
with lower OOD performance. Our analysis of RGN yields interesting findings:
some backbones lose OOD robustness during fine-tuning, but others gain
robustness because their architecture prevents the parameters from changing
too much from the initial model. Given these findings, we present recipes to
boost OOD robustness for both types of backbones. Specifically, we investigate
regularization and architectural choices for minimizing gradient updates so as
to prevent the tuned backbone from losing generalizable features. Our proposed
techniques complement each other and show substantial improvements over
baselines on diverse architectures and datasets.

##  1 Introduction

Fine-tuning a large-scale pre-trained vision model has become the defacto
transfer learning paradigm that significantly improves performance on diverse
downstream tasks compared to training from scratch [12, 28, 6, 49]. Although
naive fine-tuning performs well on the downstream task’s training distribution
(in-distribution; ID), it has subpar performance on data unseen during
training (out-of-distribution; OOD) [21]. Kumar _et al_. [21] suggest that
fine-tuning without unnecessarily distorting the pre-trained backbone is the
key to achieving high performance in both ID and OOD. They show that “linear
warm-up” training, _i.e_. training a linear head first while freezing the
backbone before fine-tuning all layers, is effective for robust image
classification.

Figure 1: Top: The standard fine-tuning approach in object detection can cause
overfitting to ID and hurt the performance on OOD data (Orange dot with Plain
Tuning) or achieve minimal improvements (Blue dot with Plain Tuning) depending
on the network architecture. Based on a thorough analysis, we propose a new
fine-tuning recipe that can achieve good performance on both ID and OOD
performance for diverse pre-trained backbones (Blue and Orange dots with Our
Tuning). Results are on Pascal [9], see Table 4. Bottom: We visualize detected
results in Foggy Cityscape.

Inspired by this, we explore the effect of backbone distortion during fine-
tuning in object detection. Although there exist generalizable detection
approaches that use data augmentation [5], feature augmentation [10], or
advanced objectives [40], our work is the first to ask whether biasing toward
the initial model is effective for this task.

We study an equivalent of “linear warm-up” training for object detection.
Specifically, we first train the decoder (all detector modules except for the
backbone) while keeping the backbone frozen, which we refer to as Decoder-
Probing (DP). After DP, we perform full fine-tuning of the detector (including
the backbone), which we call Decoder-Probing Fine-Tuning (DP-FT).
Surprisingly, we observe that the improvement in OOD performance of DP-FT over
DP depends on the architecture of the backbone. To analyze this phenomenon, we
investigate the relationship between performance improvement from fine-tuning
and the amount of gradient in each parameter. Specifically, two detectors are
compared for each backbone to compute the performance change after tuning the
backbone: (1) a DP model and (2) fine-tuning all modules (FT). We then compute
the ratio of gradient norm to parameter norm, i.e., Relative Gradient Norm
(RGN) [22], calculated on the training (in-distribution) data using DP.
Intuitively, RGN indicates how much updating feature extractors require to fit
ID data. We measure RGN’s correlation with the performance improvement on OOD
relative to that on ID. Our novel findings are three-fold: (1) a model with
small RGN is more likely to improve performance on both OOD and ID data after
fine-tuning, (2) RGN depends on the network architecture, (3) in particular,
some modules such as squeeze excitation (SE) blocks [17] can reduce the
gradient, thus lowering RGN and improving both OOD and ID performance after
fine-tuning.

Based on these findings, we explore two techniques to preserve the
generalizability of a diverse set of pre-trained models (see Fig. 1). The
first is a regularizer that minimizes the distance from the initial model in
the parameter space to prevent feature distortion. The backbone is constrained
to find a point performing well on ID, yet not too far away from the initial
model. This simple regularization effectively boosts OOD performance for
diverse architectures, yet is overlooked in generalizable object detection.
Second, we study the effect of the decoder’s architecture and training
iterations on decreasing the RGN of the backbone. A decoder producing a lower
RGN tends to train a detector more generalizable to OOD. Specifically,
inserting an SE-Block into the backbone significantly decreases RGN.

Our contributions can be summarized as follows:

  * •

We reveal a novel finding that some backbones lose more OOD robustness after
fine-tuning than others. At the same time, some backbones can significantly
increase robustness due to an architecture that prevents the parameters from
being too far away from the initial model, such as having an SE-Block.

  * •

We present methods to boost OOD robustness for both types of backbones.
Specifically, weight regularization and decoder design are investigated, and
the two techniques are shown to complement each other.

  * •

Our methods show substantial improvements over baselines when evaluating on
diverse architectures and object detection datasets.

##  2 Related Work

Domain Generalization in Image classification. Robust fine-tuning, namely,
adapting a pre-trained model to a downstream task without losing
generalization to unseen domains, has been popular in recent work. Gulrajani
_et al_. [13] find that simple ERM, _i.e_. fine-tuning on source with well-
tuned hyperparameters, shows strong generalization. Kim _et al_. [19] and
Angarano _et al_. [1] show that larger models have stronger OOD performance.
Kumar _et al_. propose warm-up with a linear head [21], while Wortsman _et
al_. propose to interpolate the weights of a fine-tuned and a zero-shot model
to produce a single robust model [43]. This technique is extended to ensemble
diverse models trained from the same initialization [42, 32], and to ensemble
several checkpoints from one training trajectory [2]. Regularizing the change
w.r.t the pre-trained model’s features is another promising way to tackle the
problem [4, 3]. Surgical fine-tuning [22] changes only a subset of layers to
adapt a model to a downstream task. Their goal is to improve the performance
on the downstream task given a small number of training data. They find that
tuning a block with a large RGN achieves high performance on the downstream
task. In contrast, our findings suggest that large RGN can indicate the risk
of overfitting the training data.

Backbone | Tuning | Comic | Water | Clipart | ID: Pascal
---|---|---|---|---|---
ResNet50 Instagram [29] | DP | 15.7 | 21.2 | 15.3 | 44.6
FT | 7.5 (-8.2) | 19.4 (-1.8) | 11.4 (-3.9) | 50.4 (+5.8)
DP-FT | 9.1 (-6.6) | 21.0 (-0.2) | 12.9 (-2.4) | 52.6 (+8.0)
EfficientNet-B2 JFT [46] | DP | 12.6 | 20.4 | 15.1 | 40.2
FT | 17.1 (+4.5) | 27.2 (+6.8) | 18.0 (+2.9) | 53.4 (+13.2)
DP-FT | 17.4 (+4.7)) | 29.4 (+9.0) | 20.7 (+5.6) | 55.3 (+15.1)
ConvNeXt Base IN21K [27] | DP | 11.7 | 17.3 | 14.0 | 39.7
FT | 11.5 (-0.2) | 22.9 (+5.6) | 16.8 (+2.8) | 60.6 (+20.9)
DP-FT | 13.6 (+2.9) | 24.7 (+7.4) | 19.1 (+5.1) | 62.3 (+22.6)

Table 1: Analysis of DP (Detector Probing, _i.e_. tuning only the decoder,
freezing backbone), FT (Fine-Tuning all modules), and DP-FT (tuning all layers
after DP training) on the three OOD domains and ID (training) domain.
Interestingly, EfficientNet-B2 FT improves OOD performance over DP while
ResNet50 FT degrades OOD performance. Overall, DP-FT improves performance over
FT.

Data Augmentation. Data augmentation effectively expands the training domain
[52, 47] _e.g_., Augmix [16] exploits diverse synthetic data augmentation, and
augmentation with style transfer can further diversify training images [51].
Our experiments show that the backbone can lose generalization even with
augmentation, needing regularization to maintain OOD robustness.

Domain Generalization in Object detection. Wang  _et al_. employ temporal
frames to ensure the consistency of features [40]. Normalization perturbation
(NP) [10], concurrent with our work, uses style randomization in feature
space. Det-AdvProp [5] combines adversarial and clean examples to train a
model. The settings of NP and Det-AdvProp are comparable to ours, but we do
not compare against their approaches since they do not publish code to train
models. Instead, we show the compatibility of our approach with data
augmentation. This indicates the potential compatibility with their methods,
which use data augmentation for training. There exist multiple source
generalizable object detection [23, 39], but we tackle a single-source
scenario with no domain labels. Vasconcelos  _et al_. [38] demonstrate that
training only a strong decoder on top of a frozen backbone can improve
detection models. They focus on evaluating the model on ID, but we inspect the
effectiveness of OOD generalization and fine-tuning the backbone.

Continual Learning. Our study connects with early work on continual learning
[20] in considering the distance from a pre-trained model in weight space.
They find that elastic weight consolidation (EWC), which accounts for the
importance of each weight on the older task, is useful for maintaining
performance on older tasks. Mirzadeh _et al_. [31] show that wider neural
networks forget less and have sparser gradient updates. Xuhong  _et al_. [48]
inspect fine-tuning for retaining the features learned on the source task,
compare several regularization methods including EWC, and observe that a
simple L2 penalty on the pre-trained model mitigates overfitting well, which
we also observe in experiments (see appendix for more details). Wortsman  _et
al_. [43] also use the L2 penalty as a baseline in an image classification
task but the penalty is not explored in object detection. We highlight that we
are the first to show that weight regularization is an overlooked, yet
powerful tool for keeping OOD performance in object detection.

Figure 2: RGN of three DP models on Pascal. The depth is normalized by the
total number of layers. Interestingly, RGN values differ a lot by models
(legend shows RGN averaged over layers). ResNet50 has a large RGN in all
layers, while EfficientNet has a very small RGN in all layers, which seems to
support the generalization in OOD.

##  3 Analysis of Feature Distortion

In this section, we aim to answer three questions; (1) how the model’s
distortion can be quantified, (2) the correlation between OOD performance and
the distortion, and (3) whether the distortion depends on the backbone
architecture. We start by analyzing three tuning strategies; (1) a decoder-
probing (DP) model, which freezes the feature extractor while training the
remaining detector modules, _i.e_., decoder, (2) fine-tuning (FT) model, which
tunes both the extractor and detector, (3) a decoder-probing before fine-
tuning (DP-FT), which applies DP training before FT.

Set-up. We use the Faster R-CNN model [33] with a feature pyramid network [24]
as our detector since it is the typical detector architecture. We regard the
pyramid modules and detector head as the decoder, wherein only the decoder is
trained for a DP model. We follow the protocol of detectron2 [45] for hyper-
parameter selection, _e.g_., the shallow CNN layers (stages 1 and 2) and all
BatchNorm parameters are frozen by the default setting of the library. DP and
FT train for the same number of iterations, DP-FT initializes the decoder with
the DP model and trains both decoder and backbone as in FT training. Backbones
are chosen mainly from models pre-trained with large-scale data such as
Instagram [29], JFT [36], and ImageNet21K [8] (IN21K) since our primary goal
is the investigation of robust models. We utilize Pascal [9] (16,551 training
images) or Cityscapes [7] (2,965 training images) as training data. We employ
watercolor, comic, and clipart [18] as OOD in Pascal, FoggyCityscapes [34] and
BDD [50] as OOD in Cityscapes. We focus on the setting where the training data
is not very large as this is the case in many applications. We employ the mean
Average Precision (mAP) used in COCO [25] as the evaluation metric.

(a) Comic: -0.78 (mAP 10.3)

(b) Clipart: -0.48 (mAP 14.8)

(c) Watercolor: -0.23 (mAP 22.0)

(d) BDD: -0.73 (mAP 5.9)

(e) Foggy: -0.60 (mAP 10.9)

Figure 3: A negative correlation exists between RGN and performance
improvement in OOD over ID by fine-tuning. RGN (x-axis) indicates whether the
network will degrade performance on OOD by fine-tuning. Y-axis indicates the
ratio of mAP improvement in OOD to the improvement in ID by fine-tuning.

The performance improvement by DP-FT depends on the architecture. First, we
conduct experiments to see the effect of DP-FT in Table 1. Interestingly, the
performance improvement over DP depends on the pre-trained models.
EfficientNet [37, 46] improves both OOD and ID performance by fine-tuning
feature extractor, but ResNet50 [29] significantly degrades OOD performance.
Also, DP-FT improves performance compared to FT but still underperforms DP in
OOD for ResNet50. Overall, ConvNeXt improves OOD performance, but the gain is
marginal, considering its high performance on ID. Prior work on image
classification [21] only reports that naive fine-tuning tends to decrease OOD
generalization. Our novel finding is that the performance decrease or increase
depends on the architecture of the pre-trained models.

Relative Gradient Norm differs by architectures. The analysis above motivates
us to study the factors that improve OOD performance with simple fine-tuning.
From the insight on linear-warm-up training for image classification [21], if
the amount of updates, _i.e_., model distortion, by fine-tuning is large, the
model will likely lose generalization on OOD. To quantify the model
distortion, we compute the variant of the ratio of gradient norm to parameter
norm (RGN) [22].

Let $\mathbf{W}\in\mathbb{R}^{C_{i}\times C_{o}\times F}$ denote a weight
parameter in a convolution layer, where $C_{i},C_{o},F$ denote input, output
channel size, and the kernel size of filters. Since we are interested in the
scale of updates per each filter, we take summation within each filter to
compute RGN in layer $L$,
$\mathbf{RGN}^{L}_{ij}=\frac{\sum_{k}|\nabla{\mathbf{W}_{ijk}}|}{\sum_{k}|\mathbf{W}_{ijk}|}$.
Then, $\mathbf{RGN}^{L}=\frac{1}{C_{i}C_{o}}\sum_{i,j}\mathbf{RGN}^{L}_{ij}$.
To get the RGN value for one model, we take the average over layers, $L$. We
utilize the DP model to compute the gradient of detection loss; thus, RGN
indicates how aggressively $W$ will be updated by fine-tuning. RGN is averaged
over training images. Note that since we do not update the DP model during
this computation. RGN shows the expected update at the starting point of DP-
FT.

Fig. 2 plots RGN values for the three models on Pascal, with different layers,
used in Table 1. RGN differs significantly by model; EfficientNet has a small
RGN in most layers, ConvNeXt has a large RGN in the first few layers, and
ResNet50 has large RGN across all layers. Also, note that ResNet50 loses OOD
generalization by fine-tuning while the other two networks improve it in Table
1. The RGN averaged over all layers are 1.34 (ResNet50), 0.13 (ConvNeXt), and
0.06 (EfficientNet), respectively. This observation is consistent with
previous findings, _i.e_., the performance on OOD will be high if a pre-
trained model does not have to aggressively update parameters to fit a
downstream task [21]. However, our novel finding is that the model distortion
is specific to each pre-trained model. Next, we conduct a more extensive study
to investigate whether the difference stems from architecture or the dataset
used to train the model.

Relative Gradient Norm (RGN) has a negative correlation with performance
improvement in OOD. In Fig. 3, we investigate the relationship between RGN and
the ratio of improvement on OOD to improvement on ID, _i.e_.
$\frac{(\rm{OOD_{FT}}-OOD\rm{{}_{DP}})}{(\rm{ID_{FT}}-ID_{DP})}$, where
$\rm{OOD}$ and $\rm{ID}$ denote the mAP on OOD and ID respectively. We employ
14 models with diverse architectures and different pre-training datasets,
_e.g_., ResNet [15], SeNet [17], EfficientNet [37], ConvNeXt [27], MobileNetV2
[35]. We observe that RGN and relative improvement has a negative correlation,
indicating that RGN is a good measurement of the performance improvement on
OOD gained by fine-tuning on ID. Intuitively, the target domain with low mAP
is far from the ID. Then, the correlation becomes more evident in these plots
as OOD gets farther away from ID. For example, the correlation is more evident
in Comic (mAP: 10.3) while it is not evident in Watercolor (map: 22.0).

Backbone | Dataset | SE | RGNpascal | RGNcity
---|---|---|---|---
ResNet50 | IN1K | ✗ | 0.31 | 0.53
ResNet50 | IN1K + Augmix | ✗ | 0.57 | 0.48
ResNet50 | IN1K + Insta | ✗ | 1.34 | 0.89
SeNet50 | IN1K | ✓ | 0.05 | 0.04

Table 2: Relationship among RGN and architecture and dataset used to train the
model. RGN of SeNet is much smaller than that of ResNet. Note that the
difference between SeNet50 and ResNet50 is whether SE-Block is plugged into
the model. Also, RGN is calculated except for SE-Block for a fair comparison.

Pre-training dataset and SE-Block affect RGN. In Table 2, we pick ResNet50
trained with different datasets and SeNet50 to show their RGN. First,
surprisingly, using more data tends to increase RGN, as shown in the results
of ResNet50. If a model is trained on diverse data, it may need to lose much
representation to be specific to the downstream task. Second, SeNet has a much
smaller RGN than ResNet50 does. The only difference between SeNet and ResNet50
is the existence of SE-Block. Therefore, we conjecture that one of the keys to
reducing RGN is in SE-Block and investigate it in the next paragraph.

Figure 4: Visualization of SE-Block activations. Note that most values are
close to either zero or one, which works to suppress gradients from the upper
layer.

Analysis of SE-Block. Squeeze-and-excitation block performs channel-wise
attention in the residual block as follows:

$\hat{\mathbf{X}}=\mathbf{X}+\sigma(S(R(\mathbf{X})))\circ R(\mathbf{X}),$ (1)

where $\sigma$, $S$, $R$, and $\circ$ denote sigmoid activation, multi-layer
perceptron after average pooling, convolution layers, and channel-wise
attention, respectively. This block applies sigmoid activation to scale
outputs from 0 to 1. Therefore, the block masks several outputs from $R(X)$
resulting in a sparse activation pattern. In Fig. 4, we visualize the
histogram of the activation from four SE-Blocks, computed on the SeNet [17]
pre-trained on ImageNet [8]. We observe that many units are close to either
zero or one.

For simplicity, let $M(\mathbf{X})$ denote $S(R(\mathbf{X}))$. Then, we can
calculate the derivative as follows:

$\diffp{\hat{\mathbf{X}}}{\mathbf{X}}=I+\diffp{\sigma(M(\mathbf{X}))}{\mathbf{X}}\circ
R(\mathbf{X})+\sigma(M(\mathbf{X}))\circ\diffp{R(\mathbf{X})}{\mathbf{X}}.$
(2)

Note that the second term includes a derivative of the sigmoid activation,
which is nearly zero if the activation is close to one or zero:

$\diffp{\sigma(M(\mathbf{X}))}{\mathbf{X}}=(I-\sigma(M(\mathbf{X})))\sigma(M(\mathbf{X}))\diffp{M(\mathbf{X})}{\mathbf{X}}.$
(3)

Also, $\sigma(M(\mathbf{X})))$ in the third term of Eq. 2 can mask the
derivative from $R(\mathbf{X})$. From this analysis and empirical findings, we
hypothesize that SE-Block is masking gradient from upper layers and promotes
robust fine-tuning.

In summary, in this analysis, we introduce RGN to measure the distortion of
the model by fine-tuning, revealing that the large distortion is correlated
with lower OOD performance. Also, some backbones lose OOD robustness during
fine-tuning, but others gain robustness because their architecture prevents
distortion.

##  4 Regularization for Robust Fine-tuning

Given the findings in the previous section, we aim to find a fine-tuning
recipe for achieving a robust object detector. In the previous section, we
have seen that large RGN distorts features of the pre-trained model, and SE-
Block can effectively reduce the amount of gradient. In addition, DP-FT boosts
OOD performance compared to FT. We build two techniques inspired by these
observations on top of DP-FT.

Figure 5: Illustration of decoder-probing with SE-Block and (left) and its fine-tuning (right). Backbone | Method | ID: Pascal | ID: Cityscapes
---|---|---|---
Comic | Watercolor | Clipart | ID | Foggy | BDD | ID
ResNet50 Instagram [29] | DP | 15.7 | 21.2 | 15.3 | 44.6 | 13.9 | 7.7 | 28.3
FT | 7.5 | 19.4 | 11.4 | 50.4 | 12.8 | 5.1 | 33.5
DP-FT | 9.1 | 21.0 | 12.9 | 52.6 | 14.8 | 5.5 | 34.7
DP-FT + WR | 16.8 | 26.5 | 17.6 | 52.9 | 19.3 | 9.6 | 34.5
ConvNeXt IN21K [27] | DP | 11.7 | 17.3 | 14.0 | 39.7 | 14.7 | 7.8 | 31.1
FT | 11.5 | 22.9 | 16.8 | 60.6 | 18.1 | 9.7 | 35.8
DP-FT | 13.6 | 24.7 | 19.1 | 62.3 | 20.5 | 11.5 | 37.1
DP-FT + WR | 14.6 | 27.8 | 19.7 | 61.4 | 21.1 | 11.7 | 37.2
Eff-B2 JFT [46] | DP | 12.6 | 20.4 | 15.1 | 40.2 | 11.1 | 6.9 | 25.2
FT | 17.1 | 27.2 | 18.0 | 53.4 | 10.7 | 5.1 | 31.5
DP-FT | 17.4 | 29.4 | 20.7 | 55.3 | 12.9 | 7.3 | 32.9
DP-FT + WR | 19.5 | 30.0 | 22.0 | 54.2 | 13.5 | 7.6 | 32.5
Table 3: Effect of weight regularization. DP, FT, and WR denote decoder-
probing, fine-tuning, and weight regularization.

###  4.1 Weight Regularization

Our analysis shows that RGN is a key factor in losing generalizability. To
reduce RGN and incorporate an explicit inductive bias of a pre-trained model,
we investigate the effect of introducing the distance from the initial value
as a regularizer, which we denote $\Omega(w)$, and optimize it with detection
loss as follows:

$L(w)=L_{det}(w)+\lambda\Omega(w),$ (4)

where $L_{det}(w)$ denotes the loss of object detection. Note that the decoder
modules are trained without regularization. There are several options for
$\Omega(w)$ as indicated by previous work on transfer learning [20, 48].

$L^{2}$ penalty. One simple option for the distance function is the L2
distance,

$\Omega(w)=\sum_{i}\lVert w_{i}-{w_{i}}^{pre}\rVert^{2}.$ (5)

The distance looks naive, yet it empirically works well to preserve robustness
to OOD.

$L^{2}$ penalty weighted by RGN. We also study weighting the $L^{2}$ penalty
by RGN:

$\Omega(w)=\sum_{i}{{\mathbf{RGN}}_{i}}\lVert w_{i}-{w_{i}}^{pre}\rVert^{2},$
(6)

where $\mathbf{RGN}_{i}$ denotes RGN value for the parameter $w_{i}$. The RGN
value needs to be computed after DP training.

Elastic weight consolidation (EWC). EWC [20] utilizes a fisher-information
matrix to weight the regularizer,

$\Omega(w)=\sum_{i}{{F}_{i}}\lVert w_{i}-{w_{i}}^{pre}\rVert^{2},$ (7)

where $F$ denotes the fisher matrix. Since Kirkpatrick  _et al_. [20] employ
diagonal approximation to compute the Fisher matrix, the biggest difference
between the penalty with RGN and this penalty is in the normalization by the
norm of the parameter. Empirically, we find that these techniques are almost
equally effective in boosting the performance of OOD, that is, by choosing
proper $\lambda$, they show comparable performance in both OOD and ID. But,
the regularization with RGN excels at transferring $\lambda$ across different
pre-trained models probably because RGN considers the scale of gradient and
weights, making regularization’s strength consistent across different
architectures. In experiments, we employ the regularization with RGN and leave
the comparison among regularizers in the appendix.

###  4.2 Regularization by Decoder Design

Decoder-Probing with SE-Block. Although the SE-Block effectively avoids
distorting features, not all pre-trained models are built with it. Thus, we
study inserting SE-Blocks into several layers of pre-trained models and
perform decoder-probing fine-tuning. By default, ResNet-like architectures
split the network into four stages. We insert the block at the end of each
stage, as shown in Fig. 5. We tune the decoder and SE-Block during decoder-
probing training, then tune all modules during fine-tuning. The block is
expected to filter the gradient from the upper layers, thus leading to better
OOD performance. Additionally, since the block is a lightweight plug-in
module, the parameter and running time increase during inference is very
small.

Stronger decoder. One simple way to reduce the gradient updates to the
backbone model is to use a strong decoder in decoder-probing training, then
fine-tune the whole network. If the strong decoder decreases the loss, the
backbone will not be significantly updated. In experiments, we investigate
increasing training iterations in decoder-probing and decoder architecture.

##  5 Experiments

In this section, we first study the effect of weight regularization and SE-
Block. After that, we analyze the combination of the two techniques and other
regularization, such as data augmentation. We follow the experimental protocol
presented in Sec. 3. We set $\lambda$ in Eq. 4 as 0.1 based on the performance
on Pascal to Watercolor using ResNet50, and apply the same value to all other
settings with weight regularization. We analyze the sensitivity to $\lambda$
in the appendix.

Effect of weight regularization. Table 3 shows results with three pre-trained
models. The results show the effectiveness of using weight regularization for
all architectures. This means that even for the architecture with a robust
structure, _e.g_., EfficientNet, adapting to downstream tasks without losing
generalization is not trivial, thus, regularization is useful. Large $\lambda$
values in Eq. 4 impose strong regularization on the training, that is, larger
$\lambda$ should improve OOD generalization while limiting the improvements on
ID performance. We obtain empirical results consistent with this intuition in
the appendix.

Backbone | Method | ID: Pascal
---|---|---
Comic | Water | Clipart | ID
ResNet50 Instagram | DP | 15.7 | 21.2 | 15.3 | 44.6
FT | 7.5 | 19.4 | 11.4 | 50.4
DP-FT | 9.1 | 21.0 | 12.9 | 52.6
DP-SE-FT | 10.2 | 22.5 | 15.1 | 53.4
\+ WR | 18.9 | 27.5 | 21.4 | 52.2
ConvNeXt IN21K | DP | 11.7 | 17.3 | 14.0 | 39.7
FT | 11.5 | 22.9 | 16.8 | 60.6
DP-FT | 13.6 | 24.7 | 19.1 | 62.3
DP-SE-FT | 15.4 | 27.5 | 20.9 | 61.6
\+ WR | 17.8 | 29.3 | 23.8 | 61.0

Table 4: Effect of inserting SE-Block (SE). SE-Block improves generalization
and combining it with weight regularization (WR) in fine-tuning further
increases performance.

Inserting SE-Block into the backbone. Table 4 investigates the effectiveness
of inserting SE-Block into the pre-trained model following Sec. 4.2. Although
it does not necessarily outperform the DP model in OOD, it consistently
improves FT and DP-FT, showing that SE-Block improves generalization.
Additionally, combining the technique with weight regularization (see +WR)
significantly improves performance, showing the compatibility of the two
techniques. Table 5 compares DP and DP-SE models in terms of RGN and the
detection loss computed on the checkpoints. The decrease in loss can reduce
the RGN value, but even when the loss value increases a little (see ConvNeXt),
DP-SE substantially reduces RGN value. This indicates that SE-Block is
important in preventing large gradients from passing to lower layers.

Distance from the pre-trained model. Table 6 shows the distance between fine-
tuned (DP-FT) models and an initial model. Since weight regularization uses
the distance as a penalty, the fine-tuned model gets closer to the initial
one. Also, SE-Block, the architecture level regularization, reduces the
distance. Although the plain training renders the model far from the initial
one, its performance on ID is worse than in some regularized models.
Overfitting can cause performance degradation even ID.

Backbone | Method | RGN | Loss
---|---|---|---
ResNet50 Instagram | DP | 1.34 | 0.34
DP-SE | 0.36 | 0.27
ConvNeXt IN21K | DP | 0.13 | 0.35
DP-SE | 0.04 | 0.36
Table 5: Comparison between DP and DP-SE (decoder-probing inserting SE-Block)
in terms of RGN.

SE | WR | Distance from Init | AP (OOD) | AP (ID)
---|---|---|---|---
|  | 7.91 | 14.3 | 52.6
✓ |  | 6.41 | 15.9 | 53.4
| ✓ | 0.34 | 19.9 | 52.9
✓ | ✓ | 0.04 | 22.6 | 52.2

Table 6: Distance from pre-trained models for DP-FT models. We use ResNet50
and Pascal. Both SE-Block and weight regularization decrease the distance from
the initial model, improving the generalization in OOD.

(a) Longer DP training.

(b) Stronger decoder.

Figure 6: Analysis of the decoder design. Left: longer DP training increases OOD performance. Right: Investigation on the stronger decoder (NAS-FPN [11]). RGN value is shown next to each plot of DP. Strong decoder model increases RGN, which causes significant decrease in OOD performance in fine-tuning.  Method | DP-FT | Augmix | ID: Pascal | ID: Cityscape
---|---|---|---|---
Comic | Watercolor | Clipart | ID | Foggy | BDD | ID
FT | ✗ | ✗ | 11.5 | 22.9 | 16.8 | 60.6 | 18.1 | 9.7 | 35.8
FT | ✗ | ✓ | 15.5 | 28.6 | 20.7 | 61.4 | 25.4 | 14.5 | 37.6
CR | ✓ | ✓ | 17.5 | 31.1 | 23.3 | 59.9 | 26.9 | 14.3 | 38.5
CR + WR | ✓ | ✓ | 19.1 | 32.2 | 24.7 | 58.6 | 27.8 | 14.9 | 39.2
WR + SE | ✓ | ✗ | 17.8 | 29.3 | 23.8 | 61.0 | 21.5 | 12.0 | 37.6
WR + SE | ✓ | ✓ | 20.0 | 32.9 | 25.4 | 61.3 | 27.1 | 14.8 | 38.8
Table 7: Compatibility of our weight regularization (WR) and SE-Block training
(SE) techniques with other domain generalization techniques using ConvNeXt.
CR: contrastive regularization, Augmix: data augmentation.

Decoder producing small RGN helps OOD generalization. Fig. 6(a) compares DP,
DP-FT, and DP-FT plus weight regularization by increasing the number of
training iterations in the DP stage. We conduct training on Pascal with the
ResNet50 model and report the performance averaged over the three OOD domains.
We have two findings; (1) longer training in DP reduces RGN and improves OOD
generalization in all approaches, yet, (2) the backbone still loses
generalization by fine-tuning (DP vs. DP-FT), and (3) weight regularization is
helpful to maintain the generalization. Fig. 6(b) further investigates the
capacity of the decoder model using NAS-FPN [11], by increasing the stack of
feature pyramid modules. Stacking more modules increases the generalization on
OOD (NAS-FPN DP), but fine-tuning from such decoders significantly reduces
generalization (NAS-FPN DP-FT). The decoder with more FPN stacks has a larger
RGN, reducing generalization by fine-tuning. These models underperform a plain
decoder with DP-FT trained with weight regularization (from Fig. 6(a)),
showing that a stronger decoder does not necessarily produce a generalizable
detector. We also try longer DP training for this strong decoder but do not
see improvements in OOD performance because the strong decoder easily overfits
the ID data. These observations suggest that training a lightweight decoder
longer in the DP stage and applying weight regularization during fine-tuning
is the best recipe to achieve an OOD robust detector.

Compatibility with other DG approaches. Since the prior work on single-domain
generalization for object detection [5], comparable to our setting, does not
publish training code, we develop several DG approaches inspired by DG image
classification to see the compatibility with our techniques in Table 7. First,
to see the compatibility with data augmentation, we study the Augmix data
augmentation [16]. Comparing the second row (FT) and the last row (WR+SE), our
approach and data augmentation have an orthogonal effect; data augmentation
expands the training domain while our approach retains the knowledge of the
pre-trained model. From this result, we conjecture that our approach is
compatible with other augmentation, such as adversarial augmentation [5].
Second, to ensure consistency with a pre-trained model, we introduce a model
regularized with feature consistency loss. Specifically, given the DP model,
this approach minimizes detection loss and contrastive loss between region-
wise features from the DP and the tuned model, wherein Augmix [16] is employed
as data augmentation. Although the contrastive loss to ensure consistency with
the pre-trained model is explored in image classification [4], our work is the
first to test in object detection. The regularization approach, CR,
outperforms FT trained with Augmix. Note that, thanks to the simplicity of
weight regularization, it is easy to plug in, computationally efficient, and
consistently improves the performance of CR (CR+WR). These results indicate
that our proposed recipes are compatible with other generalization techniques.

Method | ID: Pascal
---|---
Comic | Water | Clipart | ID
DP | 17.9 | 27.2 | 22.6 | 49.5
FT | 13.7 | 29.0 | 23.1 | 62.3
DP-FT | 15.4 | 30.7 | 23.2 | 62.8
DP-FT + WR | 19.5 | 33.7 | 26.3 | 61.9
DP-SE-FT + WR | 19.8 | 33.1 | 26.8 | 60.6

Table 8: Results on Swin Transformer.

Transformer. We further investigate the effectiveness of the regularization in
Swin-Transformer [26]. We employ Swin-B pre-trained with ImageNet-21K in Table
8. While both FT and DP-FT decrease performance on OOD or the improvements are
marginal, adding the weight regularization remarkably improves OOD
generalization with a small decrease in ID performance. The advantage of using
SE-Block is not evident in this experiment. Further exploration of gradient-
preserving layers in transformers is part of future work.

DP-FT | WR | SE | COCO | COCO-C | rPC (%)
---|---|---|---|---|---
|  |  | 36.5 | 20.2 | 55.3
✓ |  |  | 38.5 | 22.7 | 58.9
✓ | ✓ |  | 37.7 | 25.1 | 66.5
✓ | ✓ | ✓ | 37.2 | 25.7 | 69.0

Table 9: Results of ResNet50 trained on COCO (AP). rPC denotes the relative
performance under corruption. Introducing weight regularization and SE-Block
substantially improves robustness over FT (the first row).

Experiments on COCO. We evaluate our approach using COCO [25] and COCO-C [30]
containing images with diverse types and severity of image corruptions. Table
9 shows that regularization boosts robustness to corruptions by more than five
points over the plain fine-tuning (FT).

Method | Bbox | Seg
---|---|---
AP | APr | APc | APf | AP | APr | APc | APf
DP | 20.1 | 10.2 | 19.2 | 25.5 | 20.9 | 11.8 | 20.4 | 25.5
DP-FT | 22.1 | 9.9 | 20.3 | 29.5 | 22.2 | 11.4 | 20.9 | 28.4
DP-FT + WR | 22.4 | 10.7 | 21.1 | 29.0 | 22.9 | 11.9 | 22.3 | 28.5

Table 10: Results on long-tailed instance segmentation using Lvis v1.0 [14].

Analysis on the long-tailed instance segmentation. Although we focus on the
input-level distribution shift in evaluation, one of the important shifts is
in label distribution. Especially, long-tailed recognition aims to train a
model which generalizes well on diverse categories from data with imbalanced
label distribution. We hypothesize that the large-scale pre-trained backbone
has representations effective at recognizing diverse categories, but it can
lose the representations if trained on the imbalanced data; thus, the
regularization on a backbone can be effective. Then, we conduct experiments on
Lvis v1.0 [14] in Table 10. We see that DP-FT degrades performance on rare
categories (APr) compared to DP, while weight regularization improves the
performance on all types of categories. This analysis indicates the
effectiveness of a large-scale pre-trained model for long-tailed recognition.

##  6 Conclusion

We study ways of achieving a robust object detector by fine-tuning from a
robust pre-trained model. We explore warmup fine-tuning in object detection,
i.e., decoder-probing followed by full fine-tuning, and find that it gives
architecture-specific improvements. Given this, we explore two complementary
techniques to preserve generalizable representations: weight regularization to
preserve backbone features, and the design of the decoder. These techniques
prove useful for diverse architectures and datasets. We believe they will be
good baselines for generalizable object detection.

##  7 Acknowledgement

This work was supported by DARPA LwLL.

##  Appendix A Experimental Details

Datasets. The number of test images used for evaluation is listed in Table A.
For experiments on COCO, we use the validation split for evaluation.

ID: Pascal | ID: Cityscape
---|---
Comic | Water | Clipart | ID | Foggy | BDD | ID
1,000 | 1,000 | 500 | 1,999 | 500 | 5,346 | 500
Table A: Number of test images.

Training Details. We set up the hyper-parameters following the instructions
provided by detectron2 [45], _e.g_., we train the model for 80,000 iterations
with a batch size of 2 in Pascal and 24,000 iterations with a batch size of 8
in Cityscape. In Cityscape, MaskRCNN is trained and evaluation is done on the
instance segmentation task. For other architectural choices _e.g_., the
architecture of the feature pyramid and detector head, we employ the default
configuration. One exception is in the training of ConvNeXt [27], where we use
the group normalization [44] in the feature pyramid module to stabilize the
training. For pre-trained models, we employ weights available in PyTorch Image
Models [41]. We will publish the code used for training, including each
configuration, upon acceptance.

##  Appendix B Additional Results

Method | Foggy | BDD | Cityscape
---|---|---|---
DP | 14.7 | 7.8 | 31.1
FT | 18.1 | 9.7 | 35.8
DP-FT | 20.5 | 11.5 | 37.1
DP-SE-FT | 22.0 | 11.3 | 36.6
DP-FT + WR | 21.1 | 11.7 | 37.2
DP-SE-FT + WR | 21.7 | 11.8 | 37.2
Table B: Results on Cityscapes using Convnext.

Results on Cityscape. Table B shows the full ablation study on Cityscape using
ConvNeXt. Overall, combining all modules performs well in many domains.

Data | ID: Pascal
---|---
Comic | Water | Cart
IN1K | 10.9 (+3.6) | 22.5 (+2.1) | 16.5 (+4.7)
IN1K+Augmix | 12.7 (+3.1) | 25.7 (+2.6) | 17.3 (+2.5)
Instagram | 16.8 (+9.3) | 26.5 (+7.1) | 17.6 (+6.2)
Table C: Results of ResNet50 pre-trained with different datasets or
augmentation. We show the results with DP-FT + WR and improvement over FT
baseline with blue numbers.

Analysis of the dataset used for pre-training. Table C describes the
performance of ResNet50 models pre-trained with different datasets or data
augmentation, where we apply weight regularization to train these models.
Generally, pre-training on diverse data makes the model generalize well on OOD
datasets. Also, improvements over the FT baseline by the regularization get
more significant in the pre-training. In other words, pre-trained on diverse
data, the model will likely forget the learned representations with standard
training.

Figure A: Sensitivity to the hyper-parameter.

(a) RGN weighted regularization

(b) EWC

(c) L2 distance

Figure B: Comparison among difference regularization. (a): RGN weighted
regularization. (b): EWC. (c): Plain weight regularization. X-axis varies the
coefficient for regularization. Y-axis denotes the MAP on the training domain
(Pascal) with each coefficient over the MAP of the DP-FT model. Intuitively,
Y-axis denotes the strongness of the regularization.

(a) MAP on ID (Pascal).

(b) MAP on OOD.

(c) MAP on ID (Pascal).

(d) MAP on OOD.

Figure C: Effect of longer training in DP with NAS-FPN (a)(b) and the default
FPN (c)(d).

ID-OOD Trade-off by the coefficient. Fig. A shows a trade-off in ID-and-OOD
performance controlled by the regularization coefficient, $\lambda$ in Eq. 4,
which is measured using Pascal with ResNet50-Instagram. Increasing $\lambda$
adds more regularization to keep the parameter near the initial model,
reflected as the performance decrease in ID (Pascal). Clipart and Comic
increase the performance with more regularization, while Watercolor peaks near
0.1. The peak of OOD performance should differ by the similarity with ID
dataset.

Data | EfficientNet | ResNet50
---|---|---
OOD | ID | OOD | ID
L2-penalty | 22.3 | 50.5 | 20.5 | 52.6
EWC | 23.3 | 55.2 | 21.0 | 53.5
RGN | 23.8 | 54.3 | 20.3 | 52.9
Table D: Performance comparison among different regularization.

Comparison among different regularizations. Table D compares different
regularization, _i.e_., weight regularization with L2-distance, EWC, and RGN
weighted regularization, using DP-FT. We train models using Pascal and pick
$\lambda$ based on the Watercolor domain and show the average over three
domains as OOD. We do not see a significant difference among the three
regularizations, in particular, in OOD performance. We further investigate the
strength of the regularization on each $\lambda$ in Fig. B. The Y-axis shows
the MAP on Pascal with each coefficient over the MAP of the DP-FT model. With
more regularization, the value decreases. Therefore, the value implies the
strength of the regularization. Then, we see that the strength of the
regularizer is similar across different architectures by RGN while the other
two show different strengths. This indicates that RGN can add similar strength
of the regularization on different architectures by the same $\lambda$. This
is because the regularization is weighted by the relative gradient norm, which
should adjust the strength of the regularization depending on the norm of the
gradient and parameter. In short, L2-distance and EWC may require additional
hyper-parameter $\lambda$ tuning when the architecture is changed while RGN
weighted regularization is less sensitive to the changes in the architecture.

DP-FT | WR | SE | Clean | Gauss. | Shot | Impulse | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contrast | Elastic | Pixel | JPEG
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  | 36.5 | 18.5 | 19.2 | 16.8 | 24.9 | 17.4 | 21.2 | 6.4 | 12.8 | 19.1 | 20.8 | 31.9 | 21.1 | 28.0 | 23.6 | 20.5
✓ |  |  | 38.5 | 21.0 | 21.8 | 19.2 | 26.9 | 18.7 | 22.7 | 6.7 | 14.2 | 21.7 | 32.1 | 33.9 | 23.7 | 29.8 | 26.0 | 21.9
✓ | ✓ |  | 37.7 | 23.4 | 23.8 | 21.7 | 27.1 | 20.7 | 24.2 | 7.9 | 16.6 | 23.0 | 34.1 | 33.8 | 27.8 | 30.4 | 32.8 | 28.6
✓ | ✓ | ✓ | 37.2 | 22.7 | 23.0 | 20.8 | 28.6 | 23.0 | 25.4 | 8.1 | 17.4 | 23.5 | 34.5 | 34.2 | 28.7 | 30.7 | 33.2 | 31.0
Table E: Detailed results on the robustness evaluation using COCO.

Analysis on NAS-FPN. Fig. C shows the effect of training iterations for the DP
model of NAS-FPN and FPN, where we set the number of stacks in NAS-FPN as 3.
The NAS-FPN shows strong performance on ID by longer training but reduces the
performance on OOD. By contrast, FPN improves the performance on both types of
distribution through longer training. A strong decoder can easily overfit ID
data; thus, OOD performance can decrease with longer training.

Figure D: Performance on each corruption severity.

Analysis on the robustness to image corruptions. Table E shows the results on
the robustness to image corruptions using COCO. Using our regularization, the
trained detector improves the robustness to all types of corruption.
Interestingly, the use of SE-Block decreases the robustness to the high-
frequecy noise, _e.g_., gaussian noise, while improving the robustness to the
other types of corruptions. The architectural difference seems to cause this
change. Fig. D shows the performance on each severity of the corruptions.
Trained with regularization (WR, SE), the detector performs better on the
diverse level of severities than the model without regularization.

Figure E: Detection results on Pascal. Figure F: Detection results on
Cityscapes.

Qualitative Results. Fig. E and F show qualitative results on Pascal and
Cityscape. Note that, using our regularization, more objects are detected,
_e.g_., middle in Fig. E and top right in Fig. F, or correctly classified,
_e.g_., top left in Fig. E. Several objects are not localized by both models,
which indicates the difficulty of this task.

## References

  * [1] Simone Angarano, Mauro Martini, Francesco Salvetti, Vittorio Mazzia, and Marcello Chiaberge.  Back-to-bones: Rediscovering the role of backbones in domain generalization.  arXiv preprint arXiv:2209.01121, 2022.
  * [2] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park.  Swad: Domain generalization by seeking flat minima.  Advances in Neural Information Processing Systems, 34:22405–22418, 2021.
  * [3] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun.  Domain generalization by mutual-information regularization with pre-trained models.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIII, pages 440–457. Springer, 2022.
  * [4] Wuyang Chen, Zhiding Yu, Shalini De Mello, Sifei Liu, Jose M Alvarez, Zhangyang Wang, and Anima Anandkumar.  Contrastive syn-to-real generalization.  arXiv preprint arXiv:2104.02290, 2021.
  * [5] Xiangning Chen, Cihang Xie, Mingxing Tan, Li Zhang, Cho-Jui Hsieh, and Boqing Gong.  Robust and accurate object detection via adversarial learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16622–16631, 2021.
  * [6] Brian Chu, Vashisht Madhavan, Oscar Beijbom, Judy Hoffman, and Trevor Darrell.  Best practices for fine-tuning visual classifiers to new domains.  In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pages 435–442. Springer, 2016.
  * [7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In CVPR, 2016.
  * [8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 88(2):303–338, 2010.
  * [10] Qi Fan, Mattia Segu, Yu-Wing Tai, Fisher Yu, Chi-Keung Tang, Bernt Schiele, and Dengxin Dai.  Normalization perturbation: A simple domain generalization method for real-world domain shifts.  arXiv preprint arXiv:2211.04393, 2022.
  * [11] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le.  Nas-fpn: Learning scalable feature pyramid architecture for object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7036–7045, 2019.
  * [12] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * [13] Ishaan Gulrajani and David Lopez-Paz.  In search of lost domain generalization.  arXiv preprint arXiv:2007.01434, 2020.
  * [14] Agrim Gupta, Piotr Dollar, and Ross Girshick.  Lvis: A dataset for large vocabulary instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019.
  * [15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [16] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  arXiv preprint arXiv:1912.02781, 2019.
  * [17] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [18] Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Cross-domain weakly-supervised object detection through progressive domain adaptation.  In CVPR, 2018.
  * [19] Donghyun Kim, Kaihong Wang, Stan Sclaroff, and Kate Saenko.  A broad study of pre-training for domain generalization and adaptation.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 621–638. Springer, 2022.
  * [20] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  arXiv preprint arXiv:1612.00796, 2016.
  * [21] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang.  Fine-tuning can distort pretrained features and underperform out-of-distribution.  arXiv preprint arXiv:2202.10054, 2022.
  * [22] Yoonho Lee, Annie S Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn.  Surgical fine-tuning improves adaptation to distribution shifts.  arXiv preprint arXiv:2210.11466, 2022.
  * [23] Chuang Lin, Zehuan Yuan, Sicheng Zhao, Peize Sun, Changhu Wang, and Jianfei Cai.  Domain-invariant disentangled network for generalizable object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8771–8780, 2021.
  * [24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, pages 2117–2125, 2017.
  * [25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021.
  * [27] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022.
  * [28] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [29] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten.  Exploring the limits of weakly supervised pretraining.  In Proceedings of the European conference on computer vision (ECCV), pages 181–196, 2018.
  * [30] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S Ecker, Matthias Bethge, and Wieland Brendel.  Benchmarking robustness in object detection: Autonomous driving when winter is coming.  arXiv preprint arXiv:1907.07484, 2019.
  * [31] Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar.  Wide neural networks forget less catastrophically.  In International Conference on Machine Learning, pages 15699–15717. PMLR, 2022.
  * [32] Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord.  Diverse weight averaging for out-of-distribution generalization.  arXiv preprint arXiv:2205.09739, 2022.
  * [33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NeurIPS, 2015.
  * [34] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Semantic foggy scene understanding with synthetic data.  IJCV, 2018.
  * [35] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [36] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In Proceedings of the IEEE international conference on computer vision, pages 843–852, 2017.
  * [37] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [38] Cristina Vasconcelos, Vighnesh Birodkar, and Vincent Dumoulin.  Proper reuse of image classification features improves object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13628–13637, 2022.
  * [39] Xudong Wang, Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos.  Towards universal object detection by domain attention.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7289–7298, 2019.
  * [40] Xin Wang, Thomas E Huang, Benlin Liu, Fisher Yu, Xiaolong Wang, Joseph E Gonzalez, and Trevor Darrell.  Robust object detection via instance-level temporal cycle confusion.  International Conference on Computer Vision (ICCV), 2021.
  * [41] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [42] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al.  Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.  In International Conference on Machine Learning, pages 23965–23998. PMLR, 2022.
  * [43] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959–7971, 2022.
  * [44] Yuxin Wu and Kaiming He.  Group normalization.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [45] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [46] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687–10698, 2020.
  * [47] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian.  A fourier-based framework for domain generalization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14383–14392, 2021.
  * [48] LI Xuhong, Yves Grandvalet, and Franck Davoine.  Explicit inductive bias for transfer learning with convolutional networks.  In International Conference on Machine Learning, pages 2825–2834. PMLR, 2018.
  * [49] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  Advances in neural information processing systems, 27, 2014.
  * [50] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636–2645, 2020.
  * [51] Zhun Zhong, Yuyang Zhao, Gim Hee Lee, and Nicu Sebe.  Adversarial style augmentation for domain generalized urban-scene segmentation.  arXiv preprint arXiv:2207.04892, 2022.
  * [52] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang.  Domain generalization with mixstyle.  arXiv preprint arXiv:2104.02008, 2021.


# A study of HTTP/2’s Server Push Performance Potential

Rui Meireles⋆, Junrui Liu⋆, Peter Steenkiste†

⋆Computer Science Department, Vassar College, USA †Computer Science
Department, Carnegie Mellon University, USA

###### Abstract

Modern web pages have complex structures comprised of up to hundreds of
different resources, such as scripts and images. Server push is an HTTP/2
feature enabling servers to preemptively send resources to clients before they
realize they need them to render a page. The objective is to reduce the amount
of time the browser has to wait for data to be transferred, and consequently
total page load time. Our goal in this work is to quantify how much server
push can actually reduce web page load times. We approach the problem from
both theoretical and experimental perspectives. We start by deriving an upper
bound for the load time reduction afforded by server push. Then we proceed to
actually evaluate an idealized push implementation on the Alexa Top 100 global
sites[1], against a non-push HTTP/2 baseline. Our results show a linear
relationship between latency and the benefit of server push. Moreover, pages
with taller dependency trees tend to benefit the most from it.

##  I Introduction

After its inception in the early 1990s, the web quickly became the Internet’s
killer application. Minimizing web page load time is of critical importance to
improve usability and consequently user satisfaction. A 2018 study[2] found
that a $2\text{\,}\mathrm{s}\text{/}$ increase in load time halves the sales
conversion rate for retail sites. Another study[3] found that
$53\text{\,}\mathrm{\char 37\relax}\text{/}$ of mobile site visits are
abandoned if the page takes longer than $3\text{\,}\mathrm{s}\text{/}$ to
load.

Over the years, web pages have become more and more complex, a trend that is
likely to continue. At present, many of the most popular pages contain
hundreds of embedded objects and tens of scripts, leading to longer load
times. Given the speed of today’s computers and smartphones, network transfer
is most often the bottleneck in the page loading process.

HTTP/2[4] was designed to optimize network transfer in two ways. The first is
by reducing the amount of data to be transferred, which was done by changing
the protocol from textual to binary and compressing the headers. The second is
by improving network transfer pipelining. Ideally, when laid out in a
timeline, transfers would all be neatly concatenated without gaps. In practice
this has proven difficult to achieve.

With HTTP/1.0[5] clients needed to wait for the reply to an outstanding
request before issuing another one. This meant that for each new request,
there was one Round-Trip Time (RTT) without any data flow. HTTP/1.1[6]
introduced pipeline support which allows multiple requests to be in-flight
simultaneously. However, requests are still served in order, so a slow request
at the head of the queue will block subsequent requests, introducing a
_pipeline bubble_. In practice, HTTP/1.1 browsers reduce the impact of a slow
request by using multiple parallel connections. Even if one connection
experiences a bubble, the other connections can still deliver objects to the
browser. But using multiple connections uses more resources and increases
overhead, e.g., each connection needs to go through slow start.

HTTP/2 fixes this problem by having HTTP sessions that feature multiple
_streams_ , where each stream supports multiple pipelined HTTP requests,
similar to an HTTP 1.1 session. The server can transfer data for streams in
any order, depending on what streams have data available. This solves the
head-of-line blocking problem of HTTP/1.1 and obviates the need for multiple
connections. The order in which requests are served is determined by a server-
side scheduler, based on priorities assigned to each stream by either server
or client.

HTTP/2’s other transfer-optimizing feature is server push, which is the focus
of our work. It consists of having servers preemptively send page-embedded
objects to clients right after the page’s HTML file, without waiting for an
explicit request. Pushed objects are saved in the browser’s cache, eliminating
the RTT of latency that a network request would entail.

Push is performed by having the server send a _push promise_ frame announcing
its intention to send the client a specific object. The client then decides to
either accept (by doing nothing) or reject (by resetting the stream) the push
promise. Rejection is useful when the client already has the object in its
cache, for example. If the stream is not reset, the server creates a new
stream and sends the data after the main HTML. This can be done for
potentially any page-embedded object.

While the concept of server push is simple, its practical benefits and server-
side implementation are not. For example, not all objects embedded in the
original main HTML are necessarily used since scripts can rewrite part of the
web page. In this paper we study HTTP/2’s server push performance potential in
terms of page load time reduction, both analytically and experimentally. Our
contributions are as follows:

  * •

We identify and validate a theoretical upper bound for the performance
potential of server push (§II).

  * •

We create a framework to support the execution of coherent and replicable
HTTP/2 experiments (§III).

  * •

We experimentally evaluate HTTP/2’s server push against a non-push HTTP/2
baseline on Alexa’s Top 100 websites [1] (§IV).

  * •

We identify the fundamental challenges in server push deployment and suggest
ways to address them (§V).

Figure 1: Web page loading process.

##  II Theoretical analysis

In this section we start by analyzing how modern browsers load web pages, with
the goal of understanding the sources of non-optimality. This then leads to
the specification of a model for the page load time reduction potential of
server push.

###  II-A Web page loading process

An overview of the page loading process is depicted in Fig. 1. A navigation
event, for instance a link click, triggers the retrieval of the page’s main
HTML file. Parsing it yields a Document Object Model (DOM) tree describing the
page’s structure. As embedded objects are discovered during parsing, they are
also retrieved and integrated into the DOM tree. After parsing, formatting
information from any Cascading Style Sheets (CSSs) is added to the DOM tree
nodes, resulting in a render tree which is finally painted on screen.

While both server- and client-side computations also impact the overall page
load time [7], we focus on the network transfer component of the load process.
Furthermore, we assume TCP has been configured in such a way that slow start
is not a performance bottleneck. We can thus define a lower bound for Page
Load Time (PLT) on a cold connection as:

$PLT\geq 4\times RTT+\dfrac{psize}{bw},$ (1)

where $4\times RTT$ represents the TCP/TLS handshake time, in seconds, and
$\frac{psize}{bw}$ represents the amount of time it takes to transfer the
total amount of data needed to load the web page, $psize$ bits111For
simplicity, we ignore protocol overhead when calculating $psize$., over a
channel with bandwidth $bw$ $\text{\,}\mathrm{bit}\text{/}\mathrm{s}$.

To approach this bound, data transfers must all be neatly pipelined one after
another so the network is never idle.

HTTP/2 multiplexing aims to enable this, but is not a complete solution. There
are multiple reasons why embedded objects can introduce pipeline bubbles.
First, the browser must start parsing the main HTML in order to discover and
request the first embedded object. Even if the browser does this as early as
possible – i.e., even before the HTML finishes downloading – a pipeline bubble
will be introduced if the RTT is longer than the HTML transfer time remaining
at the time the request is issued. Since multiple requests can be issued in
parallel, subsequent dependencies found while parsing the same file will not
cause additional bubbles. The same logic applies to all embedded objects
requiring parsing (i.e. HTML, Javascript, and CSS files): the first dependency
request may trigger a bubble.

Second, Javascript dependencies pose an additional problem. Since its
execution can alter the DOM tree through calls to document.write(), the
inclusion of an external script traditionally blocks parsing until it is
retrieved from the network, delaying the discovery of any additional
dependencies. Developers can mark scripts that do not change the DOM with the
async tag to prevent blocking, but often do not. In practice, browsers avoid
blocking by continuing processing on the assumption that the script will not
change the DOM. If this later turns out to be false, results are discarded.

Finally, when a page embeds an object hosted on a different server, the
browser will need to connect to it prior to issuing the request, introducing a
delay of up to four RTTs, and potentially a pipeline bubble. This can not be
avoided. Consistent with our goal of assessing server push’s maximum
potential, in this work we assume a whole-site serving paradigm, in which all
page resources are downloaded from the same server.

###  II-B Server push performance potential upper bound

We now derive a theoretical upper bound for the absolute Page Load Time (PLT)
reduction provided by server push, which we shall refer to as SPR (for Server
Push Reduction). I.e., $SPR=PLT_{nopush}-PLT_{push}$, where $PLT_{nopush}$ and
$PLT_{push}$ represent the load times of a given page without and with push,
respectively. Throughout, we assume network transfer to be the bottleneck of
the loading process.

For our purposes, we can describe a web page as a dependency tree of
resources. Fig. 2 shows three examples:

a single, dependency-free html file.

a single html file with a single image dependency.

a single html file with two dependencies: an image and a script. The script
has, itself, one image dependency.

Figure 2: Example web page dependency trees.

(a) Web page recording

(b) Page load time measurement

Figure 3: Experimental setup used for web page recording and for measuring
page load times with and without server push.

As argued in the previous subsection, for each new level in the dependency
tree, a pipeline bubble of length up to RTT $\text{\,}\mathrm{ms}\text{/}$ may
be introduced. An ideal server push implementation can eliminate these bubbles
entirely. Therefore, we can write an upper bound for SPR as:

$SPR\leq RTT\times h,$ (2)

where $h$ is the height of the page’s dependency tree. Using the experimental
setup later described in §III, we validated this bound by loading the pages of
Fig. 2 using both push and non-push versions of the same HTTP/2 server.
Minimal HTML and Javascript were used. The images were
$128\text{\,}\mathrm{KiB}\text{/}$ each, and bandwidth was held constant at
$100\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$.

Tab. I shows the mean SPRs measured over 10 loads for various RTTs. Results
matched expectations. Small deviations are normal since our model does not
account for computation time or protocol overhead. p0, p1 and p2 all showed
improvements roughly equal to RTT times their dependency trees’ heights, which
are 0, 1, and 2, respectively.

Page/RTT | 25 ms | 50 ms | 100 ms | 250 ms
---|---|---|---|---
p0 | -0.77 | -0.68 | 1.19 | -1.76
p1 | 29.14 | 48.93 | 104.57 | 248.66
p2 | 53.17 | 99.63 | 201.89 | 510.8

TABLE I: Measured SPR for the pages of Fig. 2

.

Note however, that even with an ideal push implementation capable of
eliminating all pipeline bubbles, Eq. 2 is a loose upper bound for SPR. This
is because the bubble introduced at a given depth $d$ in the dependency tree
will be partially or completely masked if there is still data from lower-depth
dependencies being transferred. E.g., consider the discovery of imgb.jpg when
loading p2 from Fig. 2. It is possible for this dependency to be found while
imga.jpg is still being transferred. If the remaining transfer time for
imga.jpg is longer than the RTT, server push will yield no practical benefit,
as there will be no bubble to begin with. We can leverage this knowledge to
write a tighter upper bound for SPR:

$SPR\leq\sum_{d=1}^{h}\max\\{RTT-\dfrac{rsize_{0}^{d-1}}{bw},0\\},$ (3)

where $d$ represents the different depths of nodes in the dependency tree, and
$rsize_{0}^{d-1}$ the amount of data still to be transferred for dependencies
at depth levels in $[0,d-1]$ at the time of the discovery of the first
dependency at depth $d$.

For each depth $d$, the gain from push can vary between 0, if there is enough
data to completely mask the RTT, and RTT $\text{\,}\mathrm{ms}\text{/}$, if
there is none. This simple model lets us infer that server push will be most
helpful in high bandwidth, high latency scenarios, as there will not be enough
inflight data to mask the bubbles. Also, pages with deeper dependency trees
will tend to benefit more from push.

##  III Experimental setup

The goal of our experiments was to measure the potential page load time
reduction from server push (SPR) on a representative set of web pages. Our
strategy was to record loads for the Alexa top global 100 sites[1] and then
reload them with server push turned on and off, noting the difference.

###  III-A Web page recording

Fig. 3a depicts the setup used for recording the page loads. We employed a
custom version of the Chromium browser, release 72.0.3626.112. We used it to
save to disk the contents and headers of every downloaded resource. These were
then used to populate page and server push databases for use with our HTTP/2
server.

Page load can be affected by client-side nondeterminism. For instance, pseudo-
random numbers in Javascript may trigger different code paths, or be part of a
URL requested from the server. In order to ensure reproducibility, we had to
remove all sources of nondeterminism from the browser. This involved fixing
the date, output of the pseudo-random number generator, and size of all
graphical elements.

###  III-B Page load time measurement

Fig. 3b represents the setup used for performance evaluation. We defined the
metric of interest, Page Load Time (PLT), as time elapsed between the page
navigation command and the firing of the browser’s page load event. Pages were
served by a custom, node.js [8]-based, HTTP/2 server. It features a push
control flag that can be turned on or off, so we could isolate its effect.

In order to evaluate push’s maximum performance potential, we pushed all
dependencies that were requested in the original page load. I.e., the main
HTML was the only explicit request from the browser, everything else was
pushed.

The priority with which resources are pushed is important. E.g., if images are
pushed before scripts, parsing will be delayed, increasing PLT. After
extensive experimentation, we decided on the following order:

  1. 1.

CSSes are pushed first, in the order they were originally requested. We found
that since they’re involved in rendering, pushing them first improves
performance.

  2. 2.

All other resources follow, in the order they were requested in the original
load.

We used the Chromium browser as a client, this time modified to intercept all
network requests and redirect them to our server, allowing it to impersonate
the original one. Browser cache was disabled. And, in order to eliminate
external factors, both server and client were deployed on the same machine.

In §II we identified transfer rate and RTT as the key parameters influencing
SPR for a given web page. We then needed to select appropriate ranges for
them. The 2020 Worldwide Broadband Speed League[9] measured mean download
rates on a per-country basis. Worldwide, the values ranged between
$1\text{\,}\text{/}$ and $230\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$, with a
mean of $25\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$. The mean for Western
Europe and North America combined was
$77\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$. These values are expected to
steadily increase over time.

Figure 4: Measured RTTs for the Alexa top global 100 sites

For RTT we could not find relevant data, so we collected some ourselves. From
our location (omitted due to double-blindness), we pinged the Alexa Top 100
sites every 15 minutes over a week-long period. Fig. 4 shows the result’s
Empirical Cumulative Distribution Function (ECDF). Values of
$5\text{\,}\text{/}$-$40\text{\,}\mathrm{ms}\text{/}$ were prevalent, and
entries above $250\text{\,}\mathrm{ms}\text{/}$ residual.

Based on this information, we picked a representative set of bandwidths
($20\text{\,}\text{/}$-$500\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$) and RTTs
($25\text{\,}\text{/}$-$200\text{\,}\mathrm{ms}\text{/}$) for our experiments
(server push has negligible effect on very low latency connections). We used
macOS’ dnctl utility to control both RTT and bandwidth. For each parameter
combination, we loaded each page a total of 15 times: 5 warmup loads, and 10
measured ones. Mean PLTs were computed from the latter. To ensure loads were
completely cold, we restarted both browser and server between each load.

Our browser modifications and custom server created unsolvable issues, such as
freezes, for a small portion of pages. Out of the 100 sites, we were able to
successfully test 91.

Finally, we wanted to understand how efficient both push and non-push page
loads are in absolute terms. To this end we created a simplified version of
each web page, composed of a single HTML file whose size is equal to the size
of the totality of the resources that make up the original page. The contents
of this HTML were an appropriately-sized HTML comment (i.e., something of the
form <!--...-->), so the browser did not need to render anything. We then
measured the amount of time it took to load these simplified pages. Since they
do not have any dependencies, their load times can be used as a lower bound
for the load time of the actual pages.

##  IV Experimental results

We analyze the effect of enabling HTTP/2 server push on Page Load Time (PLT),
using the setup described in §III.

(a) $25\text{\,}\mathrm{ms}\text{/}$ RTT

(b) $50\text{\,}\mathrm{ms}\text{/}$ RTT

(c) $100\text{\,}\mathrm{ms}\text{/}$ RTT

(d) $200\text{\,}\mathrm{ms}\text{/}$ RTT

Figure 5: Page load time Empirical Cumulative Distribution Function (ECDF) for
different load types: HTTP/2 without push, with push; and size-optimal.
Bandwidth was held constant at $100\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$
while RTT was varied between $25\text{\,}\text{/}$ and
$100\text{\,}\mathrm{ms}\text{/}$.

#### Page load time function of RTT

Fig. 5 shows the ECDF of page load time across the test set, for the different
types of load under consideration: HTTP/2 with and without push, and optimal.
Bandwidth was held at $100\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$ while RTT
varied between $25\text{\,}\text{/}$ and $100\text{\,}\mathrm{ms}\text{/}$.

Two things are clear from the plots: i) enabling server push did reduce Page
Load Times (PLTs), and ii) the effect was positively correlated with RTT. Both
are consistent with our expectations. Server push reduces load times by
eliminating pipeline bubbles where the browser is waiting for the network. And
the larger the RTT, the larger the bubbles.

Server push was unable to match the load times of the optimal reference scheme
that loaded a single HTML file of the correct size. This can be explained by
two factors:

  1. 1.

Load time depends not only on how long it takes to transfer the required data,
but also on how long it takes to process it. Having a single resource reduces
server-side processing time, as well as client-side parsing and rendering
times. As the HTML contains only a comment, there is little to parse, and
naught to render.

  2. 2.

Downloading a single resource, instead of multiple, minimizes protocol
overhead. E.g., no push promise frames need to be sent. This means fewer bytes
need to be transferred overall.

Ergo, due to its nature, the optimal scheme represents a desirable goal but is
impossible to match in practice.

RTT (ms) | Median page load time (ms) | Delta (ms)
---|---|---
Optimal | Server push
25 | 499.8 | 743.1 | 243.3
50 | 625.9 | 875.7 | 249.8
100 | 886.3 | 1143.7 | 257.4
200 | 1071.2 | 1317.0 | 245.8
TABLE II: Median PLT delta between optimal and push loads.

(a) SPR function of RTT ($100\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$
bandwidth)

(b) SPR function of bandwidth ($200\text{\,}\mathrm{ms}\text{/}$ RTT)

Figure 6: Impact of the RTT and bandwidth parameters on the load time
reduction achieved by enabling server push (SPR).

It is interesting to note that the delta between the optimal and server push
schemes appears to be almost independent of RTT, as Tab. II demonstrates for
median deltas.

The fact that both schemes remained separated by around
$250\text{\,}\mathrm{ms}\text{/}$ across RTTs indicates that server push
successfully eliminated all pipeline bubbles. If there were bubbles left, they
would grow larger with RTT, increasing the load time delta. The difference
between push and optimal loads must therefore be attributed to different
processing times and protocol overheads, as described earlier.

#### Server Push time Reduction (SPR) function of RTT

Let us now focus on the relationship between the load time reduction afforded
by server push (SPR) and RTT. From the analysis of §II we anticipate SPR to
grow linearly with RTT. Fig. 6a, which shows SPR violin plots for different
RTTs (bandwidth was fixed at $100\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$),
confirms this bore out in practice. Each violin displays the SPR median,
quartiles, and probability density. We can see that doubling the RTT roughly
doubled the median SPR, e.g., when RTT goes from $100\text{\,}\text{/}$ to
$200\text{\,}\mathrm{ms}\text{/}$, median SPR goes from
$\sim$$200\text{\,}\text{/}$ to $\sim$$400\text{\,}\mathrm{ms}\text{/}$.

A least mean-squares linear regression on SPR as function of RTT yields
$SPR=1.99RTT+27.16$, which is shown as a blue line in Fig. 6a. Using this
equation for SPR prediction yields a Median Absolute Deviation (MAD) of
$65.65\text{\,}\mathrm{ms}\text{/}$, which is reasonable given that page
structure also affects SPR.

Our theoretical analysis also determined that the product of RTT and
dependency tree height forms an SPR upper bound (Eq. 2). The mean dependency
tree height of the tested pages was $2.7\text{\,}\text{/}$, and the median
$2\text{\,}\text{/}$. The $1.99\text{\,}\text{/}$ regression coefficient is
consistent with these values.

Figure 7: SPR for BBC.com (dep. tree height 6).

#### SPR function of bandwidth

Eq. 3 sets our expectations for the effect of bandwidth on SPR. If, when a new
request is issued, data from prior requests is still pending, the latency
associated with the new request will be at least partly hidden. Since lower
bandwidth yields slower transfers, we anticipated that if we decreased
bandwidth, SPR would decrease as well. In practice, that was not the case.

Fig. 6b shows SPR violin plots for different bandwidths, with RTT being held
at $200\text{\,}\mathrm{ms}\text{/}$. SPR appears to be almost unchanged
across the range of tested bandwidths. From studying the page loading logs,
for higher bandwidths server push isn’t able to fill in all the RTT gaps
because the TCP window is not large enough on a cold connection. So loading
halts waiting for TCP to catch up. Further TCP parameter tuning or usage of
warm connections, whose windows are already large, would be necessary to
overcome this issue.

Fig. 7 shows SPR plotted against both RTT and bandwidth for bbc.com’s
homepage. A clear linear relationship between RTT and SPR can be seen. In
contrast, the effect of bandwidth is minor. Both are representative of our
overall results.

#### SPR function of dependency tree height

Finally, we investigate the impact of page structure on SPR. Our analysis
identified resource dependency tree height as the key structural factor. In
most instances, Chromium provides information about which resource triggered a
given network request. We used this information to reconstruct each page’s
resource dependency tree as well as possible.

Fig. 8 depicts SPR as a function of dependency tree height for one of our
experiments ($100\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$ bandwidth,
$200\text{\,}\mathrm{ms}\text{/}$ RTT), and is representative of the overall
results. Positive correlation between tree height and SPR can be observed,
which matches our expectations. In particular, for the subset of pages with
dependency trees of heights between 1 and 3, which represents
$79\text{\,}\mathrm{\char 37\relax}\text{/}$ of all tested pages, there is a
clear linear relationship between tree height and SPR. This is not the case,
however, for pages with taller trees. This can be explained in part by the
fact that the sample size for taller tree heights was smaller. Because of
this, other page characteristics such as parsing and rendering times may have
skewed the results.

Also interesting to note is the fact that, for some pages, SPR exceeded the
theoretical limit of Eq. 2: dependency tree height times RTT. This is because
Eq. 2 does not take processing time into account. In practice, finishing
resources early means parsing can start earlier as well, and execute with
fewer interruptions, leading to additional time gains, e.g., from less context
switching. Furthermore, our push implementation prioritized CSS resources,
allowing rendering to start early.

Figure 8: SPR function of dependency tree height
($100\text{\,}\mathrm{Mbit}\text{/}\mathrm{s}$,
$200\text{\,}\mathrm{ms}\text{/}$ RTT).

##  V Implementation challenges

In our performance evaluation we used pages that always loaded in the same
exact fashion, and were therefore able to precompute a list of resources to
push for each page. In this section we describe the major hurdles to
transitioning from that idealized scenario to a practical server push
implementation.

###  V-A Nondeterministic requests

The feasibility of server push hinges on the server being able to predict
future client requests. If requests are nondeterministic, that becomes
impossible.

We have identified four major sources of nondeterminism, all stemming from the
fact that requested URLs can include the following types of data, obtained
from the Javascript API:

Values generated by calls to the Math.random() function.

Either the client’s clock value, provided by Date.now(), or page load timing
information, provided by the window.performance object.

Values such as window size and position, browsing history length, local
storage, etc. Provided by the window and document objects.

Values such as screen size, from the window.screen object, and geographical
location, from the navigator.geolocation object.

We believe the best the server can do is to avoid pushing any nondeterministic
resources. This could be accomplished for instance by parsing the page before
serving it, looking for any URLs including nondeterministic values. It would
however introduce additional complexity and delay.

###  V-B Dynamic content

Although the web started as an information archive of largely immutable
documents, over time it became more and more dynamic. Today, web page content
is extremely fluid. Refreshing a Facebook feed, for instance, will yield a
different page every time. Also, content will differ depending on the user’s
identity, each user seeing a different feed, even if both are loaded
simultaneously. Finally, content can also vary according to the device used to
access it, as a way to deal with heterogeneous screen and computation
capabilities. Moreover, with the current trend towards an _Internet Of Things_
, the need for device-based customization is likely to increase.

In summary, modern web pages are time-, user- and device-sensitive. Ergo,
there is no single list of resources to push for each web page. Rather, each
page load is unique.

Potential strategies include:

  * •

Before serving it, the server can parse the page to find dependent resources
to push. However, this introduces delay, partially offsetting the potential
benefit. Kansal [10] has proposed the use of a neural network for this. After
training, the network chooses what to push on the fly, using page structure
and client browser attributes as input.

  * •

Dynamic pages are generated by server-side programs. The code that generates
the page could, as it builds it, take note of the resources being referenced
and communicate them to the HTTP/2 server, an idea similar in spirit to that
of Netravali et al. [11]. The downside is that this requires support from the
page-generating application.

###  V-C Cached content

Browsers maintain content caches. Cached content is redundant and should not
be pushed. There are multiple ways to achieve this. One way to do this is to
have the browser reject the push promises for resources that it has in its
cache. However, some bandwidth will still be spent on the push promises and
their replies. An alternative is for the browser to send a bloom filter [12]
of its cache contents as part of a cookie in the initial page request. A bloom
filter is a compact data structure that enables fast set membership testing.
Crucially, it is free from false negatives, meaning no cached content will be
pushed. It does however, have a non-zero false positive probability, meaning
non-cached resources may end up not being pushed. The filter should be made
large enough to limit the false positive probability to a reasonable number.

###  V-D External resources

Currently, many web pages include resources that are not hosted by the server
serving the page’s HTML. Ads and social networking plugins are examples of
resources that are often obtained from external servers. This limits server
push because since they are external, they can not be pushed.

An option would be to have external servers push the resources they are
responsible for. But this would be difficult to implement, as the external
server would need to know which resources go together without having access to
the original main HTML file. One may think of creating a mapping, and having
the client send a token key to index said mapping, as part of a cookie. But
since many pages are generated on the fly, this would require tight coupling
between the different servers, which would defeat the purpose of external
resources.

An alternative is to simply not push these resources. This would reduce
performance but, since they tend to be served by Content Delivery Networks
(CDNs), which locate servers near clients to reduce RTT, the penalty should be
small.

Also, there is interest in having CDNs transition towards whole-site serving,
which would obviate this limitation.

##  VI Related work

We consider work in three areas related to page load optimization: HTTP/2
server push specifically, web prefetching at large, and dependency analysis.

#### HTTP/2 server push

There have been multiple studies on the performance of HTTP/2 server push.

Wang _et al._ [13] briefly studied push as part of SPDY [14], the protocol
that eventually developed into HTTP/2. Their results showed push performs best
when every resource is pushed and when RTT is high. Rosen _et al._ [15] also
experimentally evaluated the performance potential of server push. Their
results agree with ours, with push performing best on high latency and/or
lossy scenarios.

Zimmermann _et al._ [16] analyzed the impact of server push strategies, i.e.,
what resources to push and in what order, on page load time. They discovered
optimal performance requires custom per-site strategies. Pushing every
resource actually reduced performance for some pages. This is further evidence
of the difficulty of designing practical push implementations, leading to low
market penetration. In 2017 [17], the same authors surveyed over 5 million
HTTP/2 sites and measured push adoption at only around
$0.01\text{\,}\mathrm{\char 37\relax}\text{/}$. In 2019, adoption rate within
the Alexa top 10,000 sites was reportedly $0.9\text{\,}\mathrm{\char
37\relax}\text{/}$ [10].

By providing not only experimental results but also a theoretical framework to
help better understand push behavior, our work complements all these.

#### Web prefetching

Prefetching is a technique where the client predicts what content the user
will need in the future, and prefetches it before the user actually requests
it. Prefetching web content has a long history. Some studies [18] have focused
on reducing latency at the cost of more traffic, caused by inaccurate
predictions of future content needs. Other designs [19] provide more modest
performance gains while minimizing the adverse effects of prediction error.

In summary, both prefetching and push try to perform transfers in advance, to
improve performance. The key difference is that prefetching is controlled by
the client, while push is controlled by the server. While prefetching has the
advantage of knowing client state (e.g., cache contents), server push can
typically be more effective since it knows all files associated with a page
and the dependencies between them. Both techniques come with the danger of
wasted bandwidth due to bad predictions, however.

#### Dependency analysis

Modern web pages feature complex dependency graphs, hence a lot of effort has
been dedicated to optimizing the load of dependent resources.

Netravali _et al._ [11] proposed profiling web pages offline and embedding the
dependency graph into the page’s main HTML. A dynamic client-side scheduler
then loads the objects in optimal order with respect to the dependency graph.

In Vroom [20] the server also creates a dependency graph. But then, it uses
server push for local dependencies, and hints of URLs for ones that need to be
downloaded from other servers. This improves performance when resources from
multiple servers are needed, as the client can fetch the hinted URLs before
discovering them during parsing.

Other works [21, 22] have advanced a split-browser approach, introducing a
powerful proxy between client and server. The proxy quickly downloads the page
by using its greater resources relative to the client. It then identifies the
needed resources, and preemptively sends them to the client.

##  VII Conclusions

We studied the potential for HTTP/2 server push to reduce web page load times.
Our theoretical analysis uncovered a linear relationship between the potential
benefit of server push and RTT, as well as a positive correlation with the
height of the page’s resource dependency tree.

Our experimental evaluation using an idealized setup confirmed our analytical
findings. Having the server push every resource brought load times closer to
the optimum, with gains growing linearly with RTT. Bandwidth, on the other
hand, had little effect on results.

We conclude server push is most appealing for structurally-complex web pages
delivered over high-latency connections.

However, a practical implementation of push has major hurdles to overcome,
especially the nondeterministic nature of client requests and the dynamic
nature of modern web pages. Server push adoption has been slow [17, 10], and
we suspect that will remain the case in the foreseeable future.

## References

  * [1] “Alexa top 500 sites on the web.” [Online]. Available: https://www.alexa.com/topsites/ (accessed in Feb. 2019).
  * [2] “The State of Online Retail Performance,” Akamai Technologies, 2018.
  * [3] “The need for mobile speed,” DoubleClick by Google, 2016.
  * [4] M. Belshe, R. Peon, and E. M. Thomson, “Hypertext Transfer Protocol Version 2 (HTTP/2),” RFC Editor, RFC 7540, 2015. [Online]. Available: http://www.rfc-editor.org/rfc/rfc7540.txt
  * [5] T. Berners-Lee, R. Fielding, and H. Frystyk, “Hypertext Transfer Protocol – HTTP/1.0,” RFC Editor, RFC 1945, 1996. [Online]. Available: http://www.rfc-editor.org/rfc/rfc1945.txt
  * [6] R. Fielding, J. Gettys, J. Mogul, H. Frystyk, L. Masinter, P. Leach, and T. Berners-Lee, “Hypertext Transfer Protocol – HTTP/1.1,” RFC Editor, RFC 2616, 1999. [Online]. Available: http://www.rfc-editor.org/rfc/rfc2616.txt
  * [7] X. S. Wang, A. Balasubramanian, A. Krishnamurthy, and D. Wetherall, “Demystifying Page Load Performance with WProf,” in _Proc. of the 10th USENIX Symposium on Networked Systems Design and Implementation_ , ser. NSDI ’13, 2013.
  * [8] “Node.js Javascript runtime,” OpenJS Foundation. [Online]. Available: https://nodejs.org (accessed in Mar. 2021).
  * [9] “2020 Worldwide Broadband Speed League,” Cable.co.uk and M-Lab. [Online]. Available: https://www.cable.co.uk/broadband/speed/worldwide-speed-league/ (accessed in Mar. 2021).
  * [10] N. Kansal, “Alohamora: Reviving HTTP/2 Push and Preload by Adapting Policies On-the-Fly,” Master’s thesis, University of California, Los Angeles, 2019.
  * [11] R. Netravali, J. Mickens, and H. Balakrishnan, “Polaris: faster page loads using fine-grained dependency tracking,” in _Proc. of the 13th USENIX Symposium on Networked Systems Design and Implementation_ , ser. NSDI ’16, 2016\.
  * [12] B. H. Bloom, “Space/Time Trade-Offs in Hash Coding with Allowable Errors,” _Commun. ACM_ , vol. 13, no. 7, 1970.
  * [13] X. S. Wang, A. Balasubramanian, A. Krishnamurthy, and D. Wetherall, “How speedy is SPDY?” in _Proc. of the 11th USENIX Symposium on Networked Systems Design and Implementation_ , ser. NSDI ’14, 2014.
  * [14] M. Belshe and R. Peon, “SPDY protocol,” 2012.
  * [15] S. Rosen, B. Han, S. Hao, Z. M. Mao, and F. Qian, “Push or Request: An Investigation of HTTP/2 Server Push for Improving Mobile Performance,” in _Proc. of the 26th International Conference on World Wide Web_ , ser. WWW ’17, 2017.
  * [16] T. Zimmermann, B. Wolters, O. Hohlfeld, and K. Wehrle, “Is the Web Ready for HTTP/2 Server Push?” in _Proc. of the 14th International Conference on Emerging Networking EXperiments and Technologies_ , ser. CoNEXT ’18, 2018.
  * [17] T. Zimmermann, J. Rüth, B. Wolters, and O. Hohlfeld, “How HTTP/2 pushes the web: An empirical study of HTTP/2 server push,” _2017 IFIP Networking Conference (IFIP Networking) and Workshops_ , 2017.
  * [18] L. Fan, P. Cao, W. Lin, and Q. Jacobson, “Web prefetching between low-bandwidth clients and proxies: potential and performance,” in _ACM SIGMETRICS Performance Evaluation Review_ , vol. 27, no. 1, 1999.
  * [19] R. P. Klemm, “WebCompanion: A friendly client-side Web prefetching agent,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 11, no. 4, 1999\.
  * [20] V. Ruamviboonsuk, R. Netravali, M. Uluyol, and H. V. Madhyastha, “Vroom: Accelerating the Mobile Web with Server-Aided Dependency Resolution,” in _Proc. of the Conference of the ACM Special Interest Group on Data Communication_ , ser. SIGCOMM ’17, 2017.
  * [21] X. S. Wang, A. Krishnamurthy, and D. Wetherall, “Speeding up web page loads with Shandian,” in _Proc. of the 13th USENIX Symposium on Networked Systems Design and Implementation_ , ser. NSDI ’16, 2016.
  * [22] R. Netravali, A. Sivaraman, J. Mickens, and H. Balakrishnan, “WatchTower: Fast, Secure Mobile Page Loads Using Remote Dependency Resolution,” in _Proc. of the 17th Annual International Conference on Mobile Systems, Applications, and Services_ , ser. MobiSys ’19, 2019.