# Forward Compatible Training for Representation Learning

Vivek Ramanujan
University of Washington
Work completed during internship at Apple. Pavan Kumar Anasosalu Vasu
Apple
Ali Farhadi
Apple
Oncel Tuzel
Apple
Hadi Pouransari
Apple
Correspondence to<EMAIL_ADDRESS>
###### Abstract

In visual retrieval systems, updating the embedding model requires recomputing
features for every piece of data. This expensive process is referred to as
backfilling. Recently, the idea of backward compatible training (BCT) was
proposed. To avoid the cost of backfilling, BCT modifies training of the new
model to make its representations compatible with those of the old model.
However, BCT can significantly hinder the performance of the new model. In
this work, we propose a new learning paradigm for representation learning:
forward compatible training (Fct). In Fct, when the old model is trained, we
also prepare for a future unknown version of the model. We propose learning
side-information, an auxiliary feature for each sample which facilitates
future updates of the model. To develop a powerful and flexible framework for
model compatibility, we combine side-information with a forward transformation
from old to new embeddings. Training of the new model is not modified, hence,
its accuracy is not degraded. We demonstrate significant retrieval accuracy
improvement compared to BCT for various datasets: ImageNet-1k (+$18.1\%$),
Places-365 (+$5.4\%$), and VGG-Face2 ($+8.3\%$). Fct obtains model
compatibility when the new and old models are trained across different
datasets, losses, and architectures.

##  1 Introduction

Figure 1: The old task (top) requires discrimination between _colors_ of
objects. The new task (bottom) further requires discrimination between
_shapes_ of objects. A model which learns the minimum description feature
(_color_ top middle) of the old task will be unable to complete the new task.
This demonstrates how side-information (top right) stores information which
can be useful for future tasks but not necessarily the present task.

Modern representation learning systems for vision use deep neural networks to
embed very high-dimensional images into low-dimensional subspaces. These
embeddings are multi-purpose representations that can be used for downstream
tasks such as recognition, retrieval, and detection. In large systems in the
wild, images are constantly being added in a highly distributed manner.
Updating the models in these kinds of systems is challenging. The new model
architecture and training task may be completely disjoint from that of the old
model, and therefore its embeddings are _incompatible_ with downstream tasks
reliant on the old embeddings.

The naive solution to feature incompatibility, recomputing embeddings
(_backfilling_) for every image, is expensive. To avoid the backfilling cost,
[shen2020towards] proposed Backward Compatible Training (BCT), an algorithm
for the new model to ensure its embeddings are compatible with those of the
old model. BCT succeeds in maintaining retrieval accuracy in the no-
backfilling case. However, we show that BCT struggles to achieve the same
performance gain as independent training for the new model (Section 4.3).
Further, it can potentially carry unwanted biases from the old to the new
model [abnar2020transferring].

Figure 2: Independent training with back-filling (left), backward compatible
training (middle), and forward compatible training (Fct) (right). In Fct, we
store side-information along with gallery features to prepare for a future
embedding model.

Perfect direct compatibility between new ($\phi_{new}$) and old ($\phi_{old}$)
embedding models runs counter to the goal of learning better features.
Therefore, instead of trying to make $\phi_{new}$ directly compatible with
$\phi_{old}$, as in [shen2020towards], we learn a transformation to map from
old to new embeddings similar to [wang2020unified, meng2021learning]. If we
are able to find a perfect transformation, the compatibility problem is solved
because we can convert old to new embeddings without requiring access to the
original images (no-backfilling). However, $\phi_{old}$ possibly discards
information about the image that $\phi_{new}$ does not. This information
mismatch between the embeddings makes it difficult to learn a perfect
transformation.

In this context, we introduce the idea of _forward-compatible training_ (Fct).
This notion is borrowed from software-engineering, where software is written
with the assumption that it will be updated, and therefore is made easy to
update. Here, we adopt a similar strategy for neural networks. When training
the old model we know there will be an update in future, therefore we
_prepare_ to be compatible with some unknown successor model. This model can
vary in training dataset, objective, and architecture. We propose the concept
of __side-information__ : auxiliary information learned at the same time as
the old model which facilitates updating embeddings in the future.
Intuitively, _side-information_ captures features of the data which are not
necessarily important for the task of training the old model but are
potentially important for the next version of the model.

In Figure 1 we show a toy example to demonstrate the concept. The old task is
to classify red objects versus blue ones. The old model therefore learns
_color_ as the feature for each object, as this is the minimum discriminative
attribute for this task [voita2020information]. Later, additional data is
added (triangles), and the new task is to classify objects based on both their
shape and color. Using only what the old embeddings encode (_color_), we
cannot distinguish between different shapes which have the same color, e.g.
blue circles and blue squares. In Fct, we store _side-information_ with the
old embeddings to aid future updates. Here, _shape_ is the perfect _side-
information_ to store. Note that the shape _side-information_ in this scenario
does not help with the old-task, but is useful for future. Learning good
_side-information_ is a challenging and open-ended problem since we are not
aware of the future model or task. We present our results on different
possibilities for side information in Section 5.1.

To use _side-information_ for model compatibility, we construct a
transformation function $h$, which maps from pairs of old embeddings and
_side-information_ to new embeddings. $h$ is trained to mix the auxiliary
information provided in _side-information_ with old embeddings to reproduce
the new embedding. See Figure 2 for a schematic of our setup. Using Fct we
show significant improvements on backward-compatibility metrics (Section 4).
Since $\phi_{new}$ and $\phi_{old}$ are trained independently, $\phi_{new}$
performance remains unaffected by enforced compatibility. This is in contrast
to prior works [shen2020towards, wang2020unified], which have focused on
making $\phi_{new}$ directly compatible with $\phi_{old}$.

We note that Fct requires transforming all old embeddings, which BCT avoids.
This increased computational cost could be seen as a drawback of the method.
However, embeddings in general have much lower dimensionality than images. In
Figure 3 we show the trade-off between computational cost (per example) and
accuracy for different strategies in the ImageNet setup (with image size
$3\times 224\times 224$) as described in Section 4. Fct does not affect new
model performance, and obtains significantly higher backward accuracy for a
small additional computation and storage. Note that backfilling cost (both
computation and storage) scales with image resolution, but Fct transformation
cost only depends on the embedding dimension. Fct is particularly effective in
the paradigm where computations take place privately on-device, trivializing
its cost. We elaborate this point further in Section 3.5.

Our contributions are as follows:

  1. 1.

We propose a new learning paradigm, forward compatible training (Fct), where
we explicitly prepare for future model updates.

  2. 2.

In the context of Fct for representation learning we propose __side-
information__ : an auxiliary features learned at the same time as $\phi_{old}$
which aids transfer to $\phi_{new}$. Intuitively, _side-information_ captures
task-independent features of the data which are not necessarily important for
the task $\phi_{old}$ is trained on but are likely to be important for
$\phi_{new}$.

  3. 3.

We demonstrate substantial retrieval accuracy improvement compared to BCT for
various datasets: ImageNet-1k (+$18.1\%$), Places-365 (+$5.4\%$), and VGG-
Face2 ($+8.3\%$). We show Fct outperforms the BCT paradigm by a large margin
when new and old models are trained across different datasets, losses, and
architectures. Unlike prior works [shen2020towards, meng2021learning,
wang2020unified], models in Fct are trained independently, and hence their
accuracies are not compromised for the purpose of compatibility.

Figure 3: A comparison of compatibility methods’ accuracies and associated
costs. We show (left) Number of multiply–accumulate operations (MACs) and
(right) data storage requirement in KBytes. Transformation models of different
capacity produce different Fct costs and accuracies. We added 0 to the
logarithmic $x$-axis (left) for visual comparison.

##  2 Related Works

##### Model Compatibility

Our closest point of comparison is [shen2020towards], which presents the
problem of “backwards-compatibility”. They posit that future embeddings models
should be compatible with old models when used in a retrieval setting. They
present BCT, an algorithm that allows new embedding models to be compatible
with old models through a joint training procedure involving a distillation
loss. Other works [wang2020unified, meng2021learning, chen2019r3] attempt to
construct a unified representation space on which models are compatible. These
procedures also modify training of individual models to ensure that they are
easy to transform to this unified embedding space. This is in contrast to our
work, which assumes the old and new model are trained independently.

[li2015convergent] studies the relationship and degree of compatibility
between two models trained on the same dataset. [yosinski2014transferable]
discusses the transferability of features for the same model. These are both
empirical works primarily studying deep learning phenomena.

Side-information The idea of side-information is commonly discussed in the
zero-shot learning literature with a different context and purpose
[chen2020simple]. In zero-shot learning, side-information is acquired separate
from pretraining datasets, and is used to provide information about unseen
classes [mensink2014costa, rohrbach2010helps, elhoseiny2013write]. The recent
CLIP [radford2021learning] model and others [elhoseiny2013write] for example
use a language model as side-information to initialize their classifiers for
zero-shot inference of unseen classes. Our usage of _side-information_ differs
quite significantly, in that it is used per-example and our transfer to future
tasks is not zero-shot. Further, our retrieval setup precludes the usage of
side-information to define new categories.

Transfer and cross-domain learning Transfer learning as a field is quite
varied. Methods are variously classified under few-shot[snell2017prototypical,
li2021universal], continual learning [parisi2019continual,
wortsman2020supermasks], and life-long learning [mallya2018packnet,
rebuffi2017icarl]. In general, transfer learning methods seek to use knowledge
learned from one domain in another to improve performance[pan2009survey]. The
assumption here is that transfer learning domains are related: one can exploit
knowledge from one domain to aid in another domain. We have no such
expectation when training old and new models. The goal of our feature
transformation is not solve a new domain, but make two existing embedding
models compatible, which is not a goal of transfer learning. Current practice
for transfer learning largely centers around various fine-tuning schemes
[li2020rethinking], which makes an assumption that the architecture used in
the old domain and new domain are the same. We have no constraint on old and
new architectures.

##  3 Method

###  3.1 Problem Setup

A gallery set $\mathcal{G}$ is a collection of images, which are grouped into
different clusters, $\\{y_{1}\ldots y_{n}\\}$. In visual retrieval, given a
query image of a particular class, the goal is to retrieve images from
$\mathcal{G}$ with the same class. A set of such query images is called the
query set $\mathcal{Q}$.

In embedding based retrieval, we have an embedding model
$\phi:\mathbb{R}^{D}\to\mathbb{R}^{d}$ where $d<<D$ and $D$ is the
dimensionality of the input image, trained offline on some dataset
$\mathcal{D}$ which embeds the query and gallery images. $\mathcal{D}$ is
disjoint from $\mathcal{Q}$ and $\mathcal{G}$. Then, using some distance
function $\mathfrak{D}:\mathbb{R}^{d}\times\mathbb{R}^{d}\to\mathbb{R}_{\geq
0}$, we get the closest possible image to query image $x_{q}\in\mathcal{Q}$ as
$\operatorname*{arg\,min}_{x\in\mathcal{G}}\mathfrak{D}(\phi(x_{q}),\phi(x))$.
We use L2-distance for $\mathfrak{D}$ in our case.

In our particular setup, we have two embedding models: the old model
$\phi_{old}:\mathbb{R}^{D}\to\mathbb{R}^{d_{old}}$ and the new model
$\phi_{new}:\mathbb{R}^{D}\to\mathbb{R}^{d_{new}}$, where $d_{old},d_{new}<<D$
are the embedding dimensions of old and new models, respectively. $\phi_{old}$
and $\phi_{new}$ are trained with datasets $\mathcal{D}_{old}$ and
$\mathcal{D}_{new}$, respectively, using some supervised loss function.

We assume $\phi_{old}$ is applied to every $x\in\mathcal{G}$, generating a
collection of gallery embeddings, and $\phi_{new}$ is applied to every
$x_{q}\in\mathcal{Q}$, generating a set of query embeddings. Our goal is to
design a method for performing retrieval between these two sets of embeddings
without directly using the images in $\mathcal{G}$. We quantify the model
compatibility performance as the accuracy of this retrieval.

###  3.2 Forward Compatibility Setup

In Fct, we define a function $\psi:\mathbb{R}^{D}\to\mathbb{R}^{d_{side}}$,
which takes in an input from the $\mathcal{G}$ and produces our _side-
information_. Along with every embedding $\phi_{old}(x)$ for $x\in\mathcal{G}$
we also store the corresponding _side-information_ $\psi(x)$. Finally, we have
a transformation model
$h:\mathbb{R}^{d_{old}}\times\mathbb{R}^{d_{side}}\to\mathbb{R}^{d_{new}}$
which maps from $\phi_{old}$ and $\psi$ to $\phi_{new}$. In the Fct setup, we
assume $\psi$ and $h$ are trained using $\mathcal{D}_{old}$ and
$\mathcal{D}_{new}$ training sets, respectively. To perform retrieval using
$\psi$ and $h$, we take
$\operatorname*{arg\,min}_{x\in\mathcal{G}}\mathfrak{D}(h(\phi_{old}(x),\psi(x)),\phi_{new}(x))$.

Fct has a few key properties and constraints:

1. We do not modify the training of the new model, so it gets the highest possible accuracy.

2. We make the old model representation compatible to that of the new model through a learned transformation.

3. As we discussed in the example of Figure 1, direct transformation from old embeddings to new may not be possible. When we train the old model, we _prepare_ for a future update by storing _side-information_ for each example in $\mathcal{G}$.

Figure 4: Fct transformation model architecture and training. We use an MLP
with BatchNorm [ioffe2015batch] for the projection layers and the mixer. The
exact layer sizes are in the Appendix.

###  3.3 Training Fct Transformation

For each sample $x\in\mathcal{D}_{new}$, we compute the transformation
$h(\phi_{old}(x),\psi(x))$. Our objective to minimize is
$\ell(x)=\|h(\phi_{old}(x),\psi(x))-\phi_{new}\|_{2}^{2}$. In our case, we
write $h$ as $h_{\theta}$, a neural network parameterized by $\theta$. Our
final optimization problem can be written as
$\theta^{*}=\operatorname*{arg\,min}_{\theta}\sum_{x\in\mathcal{D}_{new}}\ell(x)$.
The transformation model consists of two projection layers corresponding to
old embedding and side-information branches followed by a mixing layer to
reconstruct the new embedding (Figure 4). We tested several alternative
objectives, including KL divergence as in [hinton2015distilling], but found L2
loss to perform the best, in contrast to [shen2020towards], which finds that
using L2 loss does not allow for backwards compatibility. Recently,
[budnik2021asymmetric] also observed superior performance using L2 loss
between features for knowledge distillation. We provide training details in
the Appendix.

###  3.4 Training Fct Side-Information

The ideal side-information encodes compressed features for each example with
which we can reconstruct new features given the old ones. This is a
challenging task since we do not know about the future task/model when
learning the _side-information_. The old model mainly learns features that are
useful for its own training objective, and is (ideally) invariant to
extraneous information. Side-information, on the other hand, should be task
independent, and capture features complementary to the old embeddings, that
are useful for possible future updates.

Learning features has been extensively studied in the context of unsupervised
and contrastive learning [chen2020simple, bachman2019learning,
gidaris2018unsupervised, noroozi2016unsupervised, zhang2016colorful,
doersch2015unsupervised] when labels are not available. Here, our objective is
to learn task-independent features that are not necessarily relevant for the
old task, and hence the labels. This makes unsupervised learning method a
particularly interesting choice to learn _side-information_ for Fct. We use
SimCLR contrastive learning [chen2020simple] to train $\psi$. As a constraint
of our setup, we can only train $\psi$ with data available at old training. We
study other choices of side-information for Fct in Section 5. We provide
details of the improved SimCLR training in the appendix.

###  3.5 Decentralized Design

Compared to BCT, Fct comes with two additional costs: (1) when a new image is
being added to the gallery-set we need to compute and store side-information,
and (2) when the embedding model is updated we need to perform a
transformation. (1) is a one-time computation for every new image added to the
system (an incremental cost) and the side-information is small relative to the
size of the image. For (2) the transformation requires access to old
embeddings and side-information. This is a small computation compared to full
backfilling (i.e., running the new model on all images in the gallery set) as
demonstrated in Figure 3.

For many applications Fct can be implemented in a decentralized fashion
(Figure 5). In this setup, we have edge devices, each with their own gallery
sets. When a new image is added, its embedding and side-information are
computed and stored on device, and the raw image is encrypted and transferred
to a remote server for storage. To update the model from old to new, the Fct
transformation runs on device. This design has three benefits: 1) The raw data
always remains encrypted outside of the device without need to download it
every time the model is updated. 2) Embedding and _side-information_
computations are privately performed on-device one-time for every new image.
3) The small Fct transformation computation is massively distributed on edge-
devices when updating the model.

Figure 5: (left) A centralized system where data and compute reside on a
server. Backfilling cost for such a system is mainly due to high compute.
(right) A decentralized system where data is stored encrypted on a server, and
Fct computations take place privately on edge devices. Storing all data on the
user side is infeasible due to capacity constraints, so only side-info and
embeddings are stored on device. The primary backfilling cost in this scenario
is transferring images from the server to every edge device.

.

##  4 Experiments

In our experimentation, we consider different model update setups by changing
training dataset, gallery set, architecture, and loss. We show Fct
consistently results in high model compatibility while not affecting accuracy
of the new model.

###  4.1 Evaluation Metrics

##### Cumulative Matching Characteristics (CMC)

corresponds to top-$k$ accuracy. A similarity ordering is produced using the
query embedding and every embedding in the gallery set, sorted by lowest L2
distance. If an image with the same class or identity (for face retrieval)
appears in the top-$k$ retrievals, this is recorded as correct. We report CMC
top-1 and 5 percentages for all models.

Mean Average Precision (mAP) is a standard metric which summarizes precision
and recall metrics by taking the area under the precision-recall curve. We
compute<EMAIL_ADDRESS>which is the average precision over recall values in [0.0,
1.0].

Notation Following [shen2020towards], we use the notation Query Embedding /
Gallery Embedding to denote which model we are using for query and gallery
embeddings, respectively. For example $\phi_{new}/\phi_{old}$ refers to using
the new model for the query embedding and the old model for the gallery
embedding. All Fct results are backwards-compatible with the definition
provided in [shen2020towards] on these two metrics.

###  4.2 Datasets

##### ImageNet-1k

[imagenet] is a large-scale image recognition dataset used in the ILSVRC 2012
challenge. It has 1000 image classes. Its train set is approximately class
balanced with approximately $\sim$1.2k images per class, $\sim$1.2 million
images total and its validation set is exactly class balanced with 500 images
per class. When used for retrieval, we utilize its validation set for both the
query set and gallery set. When we query with an image, we remove it from the
validation set to fairly compute retrieval accuracy. The old train set is
constructed using the first 500 classes of ImageNet. This is a biased split,
the second 500 classes are generally more difficult (e.g. they contain all the
dog breeds). We call this old train set ImageNet-500.

VGGFace2 [cao2018vggface2] is a large-scale face recognition dataset of 3.31
million images of 9131 subjects. It is split into a train set of 8631 subjects
and a validation set of 500 (disjoint) subjects. On average, there are 362.6
images per subject in the train set and exactly 500 images per subject in the
validation set. The old train set is constructed from the first $10\%$ of
subjects. We call this train set VGGFace2-863 and the full train set as
VGGFace2-8631. For the validation, we generate a (fixed) random subsample of
50 images per subject in the validation set to perform retrieval on, as in
[cao2018vggface2].

Places-365 [places365] is a large-scale scene recognition dataset with 1803460
images of 365 scene categories with between 3068 and 5000 images per category.
The validation set for Places-365 contains 36500 images, 100 images per scene
category. We use the first 182 classes as the old train set, which we refer to
as Places-182. As with ImageNet, we use the validation set for both the query
and gallery set.

###  4.3 Old to New Update Experiments

By convention, we use the notation (Architecture)-(Embedding Dimension). We
choose the embedding dimension with the highest accuracy on the new model. Our
ablation on this is provided in the Appendix. Note that
$(\phi_{new}/\phi_{new})$ is the upper bound and $(\phi_{old}/\phi_{old})$ is
the lower bound for model compatibility.

ImageNet-500 to ImageNet-1k In this experiment, we train $\phi_{old}$
(ResNet50-128) on ImageNet-500 $\phi_{new}$ (ResNet50-128) on ImageNet-1k
using the Softmax cross-entropy loss. The side-information model $\psi$ for
Fct is also a ResNet50-128 trained on the ImageNet-500 dataset with SimCLR
[chen2020simple]. Results are provided in Table 1.

Fct obtains substantial improvement over BCT. First, Fct, by construction,
does not affect accuracy of the new model ($68.1\%$ top-1, which is the same
as independent training). This is opposed to $5.7\%$ top-1 accuracy drop for
BCT. When comparing the CMC top-1 in the compatibility setup Fct outperforms
BCT even more significantly: $46.9\%$ to $65.0\%$, $+18.1\%$ improvement in
CMC top-1. In fact, Fcts’ top-1 compatibility accuracy ($65.0\%$) is more than
the upper bound accuracy in BCT ($62.4\%$) by $2.6\%$. For reference, in the
case where we use no side-info, where $\psi(x)=0$, we get a drop of top-1
compatibility accuracy to 61.8%, showing the importance of side-information.
See Section 5.1 for a more detailed ablation on the affects of side-
information.

We also report accuracy of the transformed old model,
$h(\phi_{old},\psi)/h(\phi_{old},\psi)$. We see significant improvement over
the independently trained old model: $46.5\%$ to $59.3\%$. This shows that a
model update with Fct, not only results in model compatibility, but also
significantly improves quality of the old features through the transformation.
This has significant consequences for practical use-cases such as embedding
based clustering.

Places-182 to Places-365 This experiment uses the same setup as ImageNet-500
to ImageNet-1k, except with training the old model (ResNet50-512) on
Places-182 and the new model (ResNet50-512) on Places-365. The SimCLR side-
information used for these experiments is trained on ImageNet-1k, showing that
side-information can be transferred between tasks. These results are shown in
Table 2. Our compatibility performance $(\phi_{new}/h(\phi_{old},\psi))$
yields a 5.4% improvement in CMC top-1 over BCT, even outdoing its upper bound
by 0.9% top-1.

VGGFace2-863 to VGGFace2-8631 In this experiment, we train the old model
(ResNet18-128) on VGGFace2-863 and the new model (ResNet50-128) on
VGGFace2-8631 using the ArcFace objective [deng2019arcface]. The exact
hyperparameters for these experiments is provided in the Appendix. In this
experiment we use the alternate old model side-information, another training
run of the old model, differing only in SGD randomness and initialization
(Section 5.1). ImageNet-1k SimCLR side-information provided marginal
improvement over no side information (91.6% to 92.0% top-1). This makes sense
because ImageNet as a domain is very distant from cropped faces. These results
are shown in Table 3 and are consistent with the results on other datasets.
Our compatibility performance yields a 8.3% improvement in CMC top-1 over BCT.
This shows our algorithm generalizes to models trained with objectives other
than softmax cross-entropy.

ImageNet-500 to ImageNet-1k w/ changing architecture This experiment has the
same setup as ImageNet-500 to ImageNet-1k, except we consider model
architecture is also changed during the update. Changing architecture when
updating a model is frequent practice. We show Fct results in Table 4 on
architectures of varying depths, structure, and embedding dimension compared
to the new model. We see that as old model performance drops, compatibility
performance also drops. However, in all cases compatibility performance
remains quite high, still outperforming the BCT upper bound on ResNet50.

Method | Case |  | CMC
---
top-1—top-5 %
<EMAIL_ADDRESS>
Independent | $\phi_{old}/\phi_{old}$ | 46.5 — 64.6 | 29.9
$\phi_{new}/\phi_{old}$ | 0.1 — 0.5 | 0.003
$\phi_{new}/\phi_{new}$ | 68.1 — 84.4 | 45.0
BCT[shen2020towards] | $\phi^{BCT}_{new}/\phi_{old}$ | 46.9 — 65.4 | 30.1
$\phi^{BCT}_{new}/\phi^{BCT}_{new}$ | 62.4 — 81.9 | 41.1
Fct (ours) | $h(\phi_{old},\psi)/h(\phi_{old},\psi)$ | 59.3 — 76.4 | 41.3
$\phi_{new}/h(\phi_{old},\psi)$ | 65.0 — 82.3 | 43.6
$\phi_{new}/\phi_{new}$ | 68.1 — 84.4 | 45.0

Table 1: Comparison of different compatible training methods. The old model is
trained on the ImageNet-500, and new model on the ImageNet-1k. The ImageNet
validation set is used as gallery and query sets. Both models are
ResNet50-128.

Method | Case |  | CMC
---
top-1—top-5 %
<EMAIL_ADDRESS>
Independent | $\phi_{old}/\phi_{old}$ | 29.6 — 58.2 | 11.6
$\phi_{new}/\phi_{old}$ | 0.3 — 1.5 | 0.12
$\phi_{new}/\phi_{new}$ | 37.0 — 65.1 | 17.0
BCT[shen2020towards] | $\phi^{BCT}_{new}/\phi_{old}$ | 30.4 — 58.7 | 12.6
$\phi^{BCT}_{new}/\phi^{BCT}_{new}$ | 34.9 — 64.2 | 16.0
Fct (ours) | $h(\phi_{old},\psi)/h(\phi_{old},\psi)$ | 34.0 — 62.3 | 17.3
$\phi_{new}/h(\phi_{old},\psi)$ | 35.8 — 64.5 | 18.1
$\phi_{new}/\phi_{new}$ | 37.0 — 65.1 | 17.0

Table 2: Comparison of different compatible training methods. The old model is
trained on Places-182, and new model on Places-365. The Places-365 validation
set is used as gallery and query sets. Both models are ResNet50-512.

Method | Case |  | CMC
---
top-1—top-5 %
<EMAIL_ADDRESS>
Independent | $\phi_{old}/\phi_{old}$ | 84.0 — 93.5 | 44.6
$\phi_{new}/\phi_{old}$ | 0.2 — 1.0 | 0.2
$\phi_{new}/\phi_{new}$ | 96.6 — 98.4 | 77.8
BCT[shen2020towards] | $\phi^{BCT}_{new}/\phi_{old}$ | 84.2 — 93.9 | 45.0
$\phi^{BCT}_{new}/\phi^{BCT}_{new}$ | 95.1 — 98.3 | 74.3
Fct (ours) | $h(\phi_{old},\psi)/h(\phi_{old},\psi)$ | 87.2 — 94.9 | 53.4
$\phi_{new}/h(\phi_{old},\psi)$ | 92.5 — 97.5 | 62.6
$\phi_{new}/\phi_{new}$ | 96.6 — 98.4 | 77.8

Table 3: Comparison of different compatible training methods on VGGFace2. The
old model is trained on VGGFace2-863, and the new model is trained on
VGGFace2-8631 with the ArcFace training loss. The architecture is ResNet18-128
for the old model and ResNet50-128 for the new model.

Old Architecture | Case |  | CMC
---
top-1—top-5 %
<EMAIL_ADDRESS>
N/A | $\phi_{new}/\phi_{new}$ | 68.1—84.4 | 45.0
ResNet18-128 [he2016deep] | $\phi_{old}/\phi_{old}$ | 40.1—60.1 | 19.9
$h(\phi_{old},\psi)/h(\phi_{old},\psi)$ | 57.0—75.0 | 39.4
$\phi_{new}/h(\phi_{old},\psi)$ | 64.2—82.1 | 42.6
MobileNet-128 [howard2017mobilenets] | $\phi_{old}/\phi_{old}$ | 42.3—62.7 | 20.3
$h(\phi_{old},\psi)/h(\phi_{old},\psi)$ | 57.6—75.7 | 39.8
$\phi_{new}/h(\phi_{old},\psi)$ | 64.3—82.1 | 42.9
ResNet50-64 [he2016deep] | $\phi_{old}/\phi_{old}$ | 44.7—62.7 | 30.6
$h(\phi_{old},\psi)/h(\phi_{old},\psi)$ | 56.6—74.1 | 39.0
$\phi_{new}/h(\phi_{old},\psi)$ | 63.6—81.9 | 42.0

Table 4: Fct compatibility results when old and new architectures are
different. For all cases, new model is ResNet50-128. The old model is trained
on the ImageNet-500, and new model on the ImageNet-1k. The ImageNet validation
set is used as gallery and query sets.

ImageNet-500 to ImageNet-1k evaluated on Places-365 In this experiment we
evaluate the same embedding models as in the using Places-365 validation set
as gallery and query sets. This is a challenging case since Places365 is a
very different domain from ImageNet. This is clear from the drop in old and
new model retrieval performance from ImageNet to Places365 in this setting:
46.3% to 14.5% and 68.1% to 21.9% top-1, respectively. With Fct we obtain
20.3% top-1 compatibility performance, $\phi_{new}/h(\phi_{old},\psi)$, only
1.6% below the accuracy of the new model. This shows generalization of the
side-information and transformation models to out-of-domain data. We report
more results with this setup in the Appendix.

###  4.4 Sequence of Model Updates

We consider a sequence of model updates:
$\text{v}_{1}\rightarrow\text{v}_{2}\rightarrow\ldots\rightarrow\text{v}_{n}$.
The first model, v1, directly computes features and side-information from the
gallery set. The vi embedding and side-information models, denoted by
$\phi_{i}$ and $\psi_{i}$, respectively, are trained independently on the vi
training set. When updating from vi to vi+1, we apply Fct transformation on
both features and side-information of vi to make them compatible with those of
vi+1:

$\phi_{i+1}\xleftrightarrow{\text{compatible}}h_{i+1}(\phi_{i},\psi_{i}),~{}\psi_{i+1}\xleftrightarrow{\text{compatible}}g_{i+1}(\phi_{i},\psi_{i})$

We use the same architectures for $h_{i+1}$ and $g_{i+1}$ transformations, as
shown in Figure 4. Both transformations are trained using the MSE loss over
the vi+1 training set, the same process as in Section 3.3.

Figure 6: Fct for a sequence of model updates. Here, we transform both side-
information and embedding to be compatible with their next versions.

Here, we experiment with a sequence of three model updates: the v1 model is
trained on the ImageNet-250 training set (a quarter of the full ImageNet
training set). v2, and v3 models have the same setup as v1 but with larger
training sets: ImageNet-500, and ImageNet-1000, respectively. The vi side-
information model is a ResNet50-128 backbone trained on vi training set using
SimCLR. We compare two Fct scenarios: (i) v1 is first updated to v2 and then
v2 is updated to v3 as shown in Figure 6, and (ii) v1 is directly updated to
v3 using $h_{1\to 3}$ transformation.

We show compatibility results in Table 5. Last two rows correspond to scenario
(i) and (ii), respectively. Fct demonstrates a great performance ($57.4\%$
top-1) even in the two-hop update, Scenario (i). Scenario (ii) is slightly
more accurate ($+1.5\%$), but in practice requires extra storage of v1’s
feature and side-information (even after updating to vi with $i>1$). In
Scenario (i) all vi features and side-information are replaced when updating
to vi+1 as shown in Figure 6.

If we do not store side-information, we observe significant drop in
compatibility performance for both scenarios: _57.4%_ to _44.9%_ in Scenario
(i) and _59.9%_ to _53.9%_ in Scenario (ii). This shows storing side-
information is crucial for compatibility performance in a sequence of model
updates.

Case | CMC top-1 — top-5 % |<EMAIL_ADDRESS>
---|---|---
$\phi_{1}/\phi_{1}$ | 29.6 — 44.1 | 15.5
$\phi_{2}/\phi_{2}$ | 46.5 — 65.1 | 28.7
$\phi_{3}/\phi_{3}$ | 68.1 — 84.4 | 45.0
$\phi_{2}/h_{2}(\phi_{1},\psi_{1})$ | 40.5 — 58.8 | 26.4
$\phi_{3}/h_{3}(\phi_{2},\psi_{2})$ | 65.0 — 82.3 | 43.6
$\phi_{3}/h_{3}(h_{2}(\phi_{1},\psi_{1}),g_{2}(\phi_{1},\psi_{1}))$ | 57.4 — 78.0 | 33.5
$\phi_{3}/h_{1\to 3}(\phi_{1},\psi_{1})$ | 59.9 — 79.3 | 37.0

Table 5: Model compatibility in sequence of updates. We see only a small
degradation in performance in performing a sequence of updates v${}_{1}\to$
v${}_{2}\to$ v3 versus going directly from v${}_{1}\to$ v3, corresponding to
the last two rows.

##  5 Ablation studies

In this section, we ablate several design and setup choices presented in
Section 4.

###  5.1 Types of Side-Information

No side-information We use no side-information as a simple baseline. In terms
of implementation, we pass the zero-vector into the projection layer in Figure
4.

Autoencoder We train a simple autoencoder with L2 reconstruction loss. The
encoder and decoder architectures are convolutional, and based on MobileNetv2
[sandler2018mobilenetv2]. The exact architectures are in the Appendix.

Alternate Old Model With this method, we train another version of
$\phi_{old}$, with the only difference being randomness from data order and
model initialization. This is very similar to ensembling [fort2019deep]. The
intuition is that each model captures a different facet of the data. We denote
this model as $\phi_{old}^{alt}$.

Alternate Model + Mixup We perform the same training as for $\phi_{old}$ but
with Mixup augmentation [zhang2017mixup] applied only on images (for labels we
use one-hot vectors). This encourages learning features of the data different
from the old model, which will aid in transformation as they capture different
invariances of the data.

Contrastive Model Here we train a SimCLR [chen2020simple] self-supervised
contrastive learning model to use as side-info. Taking the previous ideas to
their natural extreme, a self-supervised contrastive approach directly
captures invariances in the data, which will be useful for transfer even if it
is not as useful for retrieval.

Side-Information $(\psi)$ |  | $\phi_{new}/h(\phi_{old},\psi)$
---
CMC top-1 — top-5 %
<EMAIL_ADDRESS>| CKA
None | 61.8 — 80.5 | 39.9 | 0.0
Autoencoder | 62.0 — 80.9 | 40.5 | 0.04
$\phi_{old}^{alt}$ | 63.5 — 81.6 | 42.1 | 0.70
$\phi_{old}$ w/ Mixup | 63.8 — 81.9 | 42.4 | 0.66
SimCLR-128-ImageNet-500 | 65.0 — 82.3 | 43.6 | 0.27
SimCLR-128-ImageNet-1k | 66.5 — 83.3 | 46.6 | 0.25

Table 6: Comparison of different kinds of side-info. Both old and new
architectures are ResNet-50 with feature size 128. The old model is trained on
ImageNet-500 and the new model is trained on ImageNet-1k

Side-information ablation results First note that no side-information results
in 61.8% top-1 retrieval accuracy, which is 14.9% greater than BCT in this
setting. This demonstrates the strength of our transformation setup even
without side-information. The autoencoder provides a slight improvement over
no side-information. This make sense considering much of the information of
the autoencoder is encoding is contextually dependent on its decoder, and is
therefore hard to extract a priori.

The $\phi_{old}^{alt}$ and the $\phi_{old}^{alt}$ with Mixup both provide
substantial improvements over no side-information. It is well-known in deep
learning literature that different runs of SGD for the same model produce
diverse predictions [mania2019model, fort2019deep], which is particularly
useful for ensembling. Here we show that we can leverage this fact for side-
information. It’s important to note that simply applying a transformation from
ResNet50-256 with no side-info trained on ImageNet-500 to the new model only
yields 62.0% CMC top-1, which is less than the improvement we get from
training a second old model. Mixup regularization gets a further 0.3%
improvement over $\phi_{old}^{alt}$, even though it does not provide a better
model for retrieval, with 46.4% $(\phi_{old}/\phi_{old})$ CMC top-1.

The SimCLR model with embedding dimension of $128$ trained on half of the
ImageNet training set, denoted by SimCLR-128-ImageNet-500 outperforms other
side-information choices considered (+1.2% over Mixup). This is despite the
fact that CMC top-1 with just the SimCLR model is only 35.41%, far worse than
even $\phi_{old}$ at 46.5%. What’s even more interesting is that when we train
our transfer model with only SimCLR side-information and no $\phi_{old}$ we
get 63.2% top-1 retrieval accuracy, better than no side-info. This
demonstrates the efficacy of contrastive features for forward transformation.
Even given the large gap in retrieval performance between $\phi_{old}$ and
SimCLR, transfer performance for SimCLR is still superior, validating our
intuition that it is important for side-information to capture all the
invariances of the target data source.

We also consider a case that for the old model training where we have access
to the next version training set (ImageNet-1k), yet without labels. We use the
unlabeled dataset to train the side-information model, shown by
SimCLR-128-ImageNet-1k. This side-information results in $66.5\%$ backward
compatibility accuracy, $+1.5\%$ boost compared to the case when
SimCLR-128-ImageNet-500 is used as the side-information model.

Centered kernel alignment (CKA) [kornblith2019similarity] is a similarity
index identifying correspondences between representations. We show CKA with
linear kernel between the old model features and side-information over the
gallery set (the ImageNet validation set) in Table 6. We use CKA as a measure
of complementarity of the side-information to the old model. Evidently, using
mixing to train an alternate old model results in lower CKA (0.7 to 0.66), and
hence higher compatibility accuracy when used as a side-information. More
interestingly, the SimCLR side-information model has significantly less CKA
with the old model (0.27). This demonstrates features learned by SimCLR are
_complementary_ to those of the old model. Hence, combining them in Fct
transformation results in excellent accuracy (65.0). Note that no side-info
(the zero vector) and the Autoencoder have very low CKA but poor performance,
demonstrating that this is not the only factor important to selecting a good
model for side-information.

##  6 Conclusion

In this paper we study the compatibility problem in representation learning.
We presented a new learning paradigm: forward compatible training, where we
prepare for the next version of the model when training the current model. Fct
enables efficient embedding updates to ensure compatibility with future models
in large systems. We have shown that it is important to learn side-information
to transform old features to new embedding models. Side-information captures
features of the data which might not be useful for the old task, but are
potentially important in the future. We demonstrate that contrastive learning
methods works well to train side-information models. Through experimentation
across multiple datasets, architectures, and losses, we share insights into
necessary components important to the design of model-to-model transformation
using side-information. Finally, by ensuring that training of the old and new
embedding models are completely independent, we do not degrade model
performance or retain biases from old model training, a problem in prior
compatibility literature.

##  7 Limitations

Fct provides a mechanism to make old embeddings compatible with their future
version. As a result, after transforming gallery embeddings, only downstream
models which are non-parametric will not need to be updated (e.g. nearest
neighbor retrieval in our setup). This is not a requirement with BCT since new
embeddings are directly comparable with old embeddings. However, the
convenience of not updating the downstream task model comes with limited
accuracy improvement when updating the embedding model. We have not evaluated
on cross-domain compatibility: when the old and new model are trained on
widely different domains. In practice, it is unclear how likely this scenario
is, but we suspect it will make learning a transformation and useful side-
information more difficult.

$\mathcal{D}_{old}$ | $\mathcal{D}_{new}$ | Embedding Dimension | $\phi_{new}/h(\phi_{old})$ | $h(\phi_{old})/h(\phi_{old})$
---|---|---|---|---
CMC top 1—5 % | mAP % | CMC top 1—5 % | mAP %
ImageNet-500 | ImageNet-1k | 128 | 61.8 — 80.5 | 39.9 | 51.9 — 69.8 | 36.1
ImageNet-500 | ImageNet-1k | 256 | 62.0 — 80.9 | 38.7 | 53.8 — 71.9 | 35.8
Places-182 | Places-365 | 512 | 34.8 — 63.6 | 17.5 | 31.9 — 60.4 | 16.4
VGGFace2-863 | VGGFace2-8631 | 128 | 91.5 — 97.3 | 60.1 | 84.0 — 93.4 | 47.7

Table 7: Detailed results in the “no side-information” case. We see a
substantial improvement in retrieval performance across all datasets.

Our appendix is organized as follows:

  1. 1.

Appendix A contains detailed experimental results in the “no-side information”
case. We show that adding side-information outperforms no side-information in
all cases, including when the embedding for “no-side information” has
dimension equal to that of (side-information + embedding).

  2. 2.

Appendix B shows performance of transformation with side-information on out-
of-distribution datasets.

  3. 3.

Appendix C shows a more detailed analysis of performance degradation of the
Backwards Compatible Training (BCT) algorithm when applied to $\phi_{new}$ on
ImageNet.

  4. 4.

Appendix D reports hyperparameters and training details across all
experiments.

  5. 5.

Appendix E reports the results of our ablation for the transformation function
on both capacity and loss function.

  6. 6.

Appendix F reports details about embedding dimension ablations for
$\phi_{new}$ and $\phi_{old}$ across datasets.

  7. 7.

Appendix G shows our results with highly compressed side-information.

  8. 8.

Appendix H reports the licenses of all the resources used.

  9. 9.

Appendix I contains a simplified one-file version of our code to apply to
ImageNet.

##  Appendix A Comparison with No Side-Information

###  A.1 Standard Setup

Table 7 contains detailed results for “no side-information” in the standard
setup. This corresponds to Tables 1, 2, and 3. In terms of implementation, we
use $\psi(x)=0$ as for our side-information function. We see that side-
information improves model performance across all datasets. In particular, we
see a CMC top-1 improvement of +3.2% on ImageNet, +1.0% on Places, and +1.0%
on VGGFace2. The magnitude of improvement on
$h(\phi_{old}/\psi)/h(\phi_{old},\psi)$, a measure of how much our
transformation improves the old features, is even greater, with CMC top-1
improvements of +7.4% on ImageNet, +2.1% on Places, and +3.2% on VGGFace2.

Finally, an argument could be made that the overall feature dimension is
simply larger, when we consider side-information feature size together with
the embedding dimension feature size. To show the benefit of our side-
information over simply increasing the embedding dimension size, we show that
a 256-dimensional feature vector for ImageNet only results in a 0.2%
improvement in CMC top-1, far from the improvement which comes from having
good side-information (+3.2%).

###  A.2 Sequence of Model Updates

Case | CMC top-1 — top-5 % |<EMAIL_ADDRESS>
---|---|---
$\phi_{1}/\phi_{1}$ | 29.6 — 44.1 | 15.5
$\phi_{2}/\phi_{2}$ | 46.5 — 65.1 | 28.7
$\phi_{3}/\phi_{3}$ | 68.1 — 84.4 | 45.0
$\phi_{2}/h_{2}(\phi_{1})$ | 36.5 — 53.5 | 23.7
$\phi_{3}/h_{3}(\phi_{2})$ | 61.8 — 80.5 | 39.9
$\phi_{3}/h_{3}(h_{2}(\phi_{1}),g_{2}(\phi_{1}))$ | 44.9 — 68.0 | 26.4
$\phi_{3}/h_{1\to 3}(\phi_{1})$ | 53.9 — 74.2 | 30.6

Table 8: Model compatibility in sequence of updates with no side-information
(see Table 5 for results with side-information). We see a large performance
degradation v${}_{1}\to$ v3 versus with side-information (-6.0% CMC top-1) and
an even larger degradation in the sequential case v${}_{1}\to$ v${}_{2}\to$ v3
(-12.5% CMC top-1), showing that side-information is crucial for sequential
updates to prevent feature drift.

Table 8 contains results corresponding to the sequence of model updates case
(see Section 4.4) with no side-information. As in the previous section, this
is implemented by setting $\psi(x)=0$ for all inputs $x$. We denote this case
$h(\phi)$ instead of $h(\phi,\psi)$ for embedding model $\phi$ and
transformation model $h$. All other notation is consistent with Section 4.4.

##  Appendix B Out-of-distribution Side-information performance

Side-Info |  | $(\phi_{new}/h(\phi_{old},\psi))$
---
CMC top-1 — top-5 %
<EMAIL_ADDRESS>
None ($\phi_{old}$) | 14.5 — 35.2 | 3.7
None | 16.9 — 40.7 | 5.6
Autoencoder | 17.4 — 41.6 | 5.8
$\phi_{old}^{alt}$ | 18.4 — 43.1 | 6.3
SimCLR | 20.3 — 45.5 | 7.1
None ($\phi_{new}$) | 21.9 — 46.8 | 7.1

Table 9: Using the same ImageNet embedding models $\phi_{old}$ and
$\phi_{new}$ we analyze the same side-information strategies as Table 6 but
instead evaluating on out of distribution data: the Places-365 validation set.
We show that the same trends hold even out of distribution. Note that the
retrieval performance is quite poor since ImageNet and Places-365 are very
different domains.

Here we provide a more detailed ablation of the out-of-distribution retrieval
performance for various side-information transformation methods originally
presented in Section 4.3. We present these results in Table 9. We see the same
trends from Table 6 repeated on this very different domain.

##  Appendix C BCT New Model Biased Performance Degradation

Figure 7: CMC Top-1 performance on the validation set of ImageNet for various
cases by class range. Note that $\phi_{old}$ has not seen any data from
classes [501-1000]. Note how $\phi_{new}^{BCT}$ performance is degraded
primarily for the classes on which $\phi_{old}$ also performs poorly. This
shows that the old model biases training of the new model for BCT.

In Figure 7 we show how BCT training of the new model biases it towards the
old model. Since the old model has not seen any data from ImageNet classes
[501-1000], it naturally performs quite poorly on these classes. However,
performance on classes [1-500] is quite similar across $\phi_{new}$ and
$\phi_{old}$. $\phi_{new}^{BCT}$ sees some improvement on classes [501-1000],
however it performs 6.4% worse than $\phi_{new}$ on these classes. Given that
its accuracy is similar to that of $\phi_{old}$ and $\phi_{new}$ on classes
[1-500], we can conclude that the effect of BCT training is to bias
$\phi_{new}^{BCT}$ to perform more similarly to $\phi_{old}$, i.e. poorly on
[501-1000]. Transferring unwanted biases from old to new model is a crucial
drawback of the BCT method.

##  Appendix D Architectures, Hyperparameters, and Training

###  D.1 Architecture Details

Embedding Models For our embedding model architectures, we use the standard
ResNet [he2016deep] and MobileNetV1 [howard2017mobilenets] with penultimate
layers modified to output 128 dimension embeddings (for VGGFace2 and ImageNet)
and 512 dimension embeddings (for Places-365). We find that this architecture
modification performs either the same or better than their higher dimension
counterparts. We also observed that directly modifying the output dimension of
the penultimate layer performs equivalently to projecting the output of the
penultimate layer to a lower dimension, as in [shen2020towards].

Transformation Models See Figure 4 for the basic transformation structure. The
projection units in this diagram are MLPs with architecture:
$\text{Linear}(d\to
256)\to\text{BatchNorm}\to\text{ReLU}\to\text{Linear}(256\to
256)\to\text{BatchNorm}\to\text{ReLU}$, where $d$ is the dimension for either
the embedding ($d_{old}$) or side-information ($d_{side}$). The linear layers
use bias, however preliminary experiments have shown this is not necessary.
The outputs of both projection branches are concatenated together before being
passed to the “Mixer”, which has architecture $\text{Linear}(512\to
2048)\to\text{BatchNorm}\to\text{ReLU}\to\text{Linear}(2048\to
2048)\to\text{BatchNorm}\to\text{ReLU}\to\text{Linear}(2048\to d_{new})$.

See flat_2xrn50_to_rn50_with_bn_projection in the provided code (Appendix I)
for exact implementation.

###  D.2 Model Selection Details

We did not perform hyperparameter tuning on transformation architecture
training. We reused these ImageNet hyperparameters on the other datasets and
embedding dimension sizes. The hyperparameters for ResNet and MobileNet
training were taken from ResNetV1.5 [repo], an improved training setup for
ResNet. Hyperparameter details for this setup are provided in the in the
individual dataset sections.

###  D.3 Hardware Details

We trained all of our models on 8 Nvidia V100 GPUs with batch size 1024. We
have found that it is possible to decrease the batch size proportionally with
the learning rate to use with fewer resources (e.g. batch size 1024 with
learning rate 1.024 corresponds to batch size 256 with learning rate 0.256)
without a drop in performance.

Embedding Size | ImageNet-{500, 1k} | Places-{182, 365} | VGGFace2-{836, 8631}
---|---|---|---
$\phi_{old}/\phi_{old}$ | $\phi_{new}/\phi_{new}$ | $\phi_{old}/\phi_{old}$ | $\phi_{new}/\phi_{new}$ | $\phi_{old}/\phi_{old}$ | $\phi_{new}/\phi_{new}$
128 | 46.5 | 68.0 | 29.5 | 21.9 | 84.0 | 96.6
256 | 48.0 | 67.8 | 29.7 | 23.2 | 83.8 | 95.7
512 | 49.1 | 67.4 | 29.6 | 37.0 | 83.6 | 96.4
1024 | 49.0 | 67.1 | 29.5 | 36.7 | 84.1 | 95.9
2048 | 48.5 | 67.9 | 29.8 | 37.0 | 83.9 | 95.1

Table 10: Embedding dimension size ablation on ImageNet, Places-365, and
VGGFace2 for CMC top-1. The $\phi_{old}$ architectures are ResNet50 for Places
and ImageNet and ResNet18 for VGGFace2. The $\phi_{new}$ architectures are
ResNet50 for all datasets. We train $\phi_{old}$ on ImageNet-500, Places-182,
and VGGFace2-863 and $\phi_{new}$ on ImageNet-1k, Places-365, and
VGGFace2-8631, respectively. We chose to use the dimension corresponding to
the best performing $\phi_{new}$ for each dataset (bolded). In the case of a
tie, we chose the lower dimensional model.

###  D.4 Side-Information

####  D.4.1 SimCLR

Architecture We use a standard ResNet50 [he2016deep] architecture with feature
output directly modified to 64 or 128 for all of our SimCLR results. We add an
extra BatchNorm layer to the end of the model, as suggested in [chen2020big].

Training Most of our training procedure is taken directly from the original
SimCLR paper [chen2020simple]. We use the multi-crop augmentation procedure
originating from [caron2020unsupervised], specifically implemented as in
[caron2021emerging].

####  D.4.2 Autoencoder

Architecture See Appendix J for the code corresponding to our AutoEncoder
architecture.

Training We train our AutoEncoder with the Adam optimizer using learning rate
$3\times 10^{-4}$ and weight decay 0.0 for 100 epochs with cosine learning
rate decay and 5 epochs of linear warmup with batch size 512.

###  D.5 ImageNet-1k Training

Transformation We train the transformation for 80 epochs with the Adam
[kingma2014adam] optimizer. We use learning rate $5\times 10^{-4}$, weight
decay $3.0517578125\times 10^{-5}$, cosine annealing learning rate schedule
with one cycle [cosinelr] with linear warmup [warmup] for 5 epochs, taken from
ResNetV1.5 [repo]. At epoch 40 we freeze the BatchNorm statistics. We find
empirically that this makes training more stable for smaller embedding sizes.
We suspect that there is some configuration of hyperparameters and learning
rates where this is not necessary, however we were not able to find it. For
normalization methods with no batch statistics (e.g. LayerNorm [ba2016layer]),
we find that this is not necessary, but we get slightly worse performance
(61.2% vs 61.6% CMC top-1 in the no side-information case with LayerNorm
instead of BatchNorm).

Old and New Embedding Models We train the old and new embedding models
(standard ResNet50 with 128 dimension embedding) with ResNetV1.5
hyperparameters [repo]. We train with batch size 1024, learning rate 1.024,
weight decay $3.0517578125\times 10^{-5}$, momentum 0.875, and cosine learning
rate decay with 5 epochs of linear warmup for 100 total epochs.

###  D.6 Places-365 Training

Transformation We train the Transformation for the same duration and with the
same hyperparameters as for ImageNet.

Old and New Embedding Models We train the old and new embedding models for the
same duration and with the same hyperparameters as for ImageNet. We use
embedding dimension 512 for our ResNet50 models. We explain this choice in
Appendix F.

###  D.7 VGGFace2 Training

Transformation We train the Transformation for the same duration and with the
same hyperparameters as for ImageNet. Since the embeddings are normalized in
this instance, we normalize the outputs of the Transformation during both
training and inference.

Old and New Embedding Models We train VGGFace2 with the ArcFace
[deng2019arcface] loss function. Following [deng2019arcface], we use a margin
of 0.5 and scale of 64, with embedding dimension 128 (we find this to perform
equally to embedding dimension 512, which was used in [deng2019arcface]). We
also resize to 3x224x224 as we find this does better than 3x112x112, which is
a standard for face retrieval applications. We train the old and new embedding
models for the same duration and with the same hyperparameters as for
ImageNet.

###  D.8 BCT Modifications

Old and New Embedding Models We use the author provided code [repoBCT] with a
few modifications. In particular, we add the ResNetV1.5 parameters (stated
previously) to properly compare with our method and modify the output
embedding dimension to 128 (for ImageNet and VGGFace2) and 512 (for Places-182
and Places-365). We find that simply projecting the output layer, as we do in
the provided code snippet in Appendix I, performs equally well. These
modifications result in significant improvement over the original code
provided. In particular, $\phi_{new}^{BCT}/\phi_{new}^{BCT}$ performance goes
from 60.3% CMC top-1 before our modifications to 62.4% CMC top-1 after our
modifications.

##  Appendix E Transformation Size Ablation

# of params (M) |  | $(\phi_{new}/h(\phi_{old},\psi))$
---
CMC top-1 — top-5 %
0.79 | 62.9 — 81.0
1.9 | 64.1 — 81.8
5.7 | 65.0 — 82.3
19.6 | 65.0 — 82.5

Table 11: Effect of transformation function capacity on accuracy. Accuracy
seems to saturate at a relatively small number of parameters. Note that there
is not a one-to-one comparison between number of parameters and FLOPS, as
convolutional layers tend to have a higher FLOPS to parameter count ratio. See
Figure 3 for a direct comparison of these attributes.

Loss |  | $(\phi_{new}/h(\phi_{old},\psi))$
---
CMC top-1 — top-5 %
MSE | 65.0 — 82.3
KL | 60.7 — 78.3
KL-Reversed | 55.7 — 76.1

Table 12: Effect of loss function on training of Fct transformation.

###  E.1 Transformation Capacity and Training

The transformation function $h$ should have small memory foot-print and
computational cost. In Table 11, for the same setup as ImageNet experiment as
in Section 4, we show accuracy of the transformed features for a different
transformation model architectures with growing width. In Table 12 we compare
effect of loss function. For KL-divergence, we apply both $\phi_{new}$ and
$h(\phi_{old},\psi)$ to the linear classifier trained with the new model
(which is frozen), get log-probabilities, and then apply the Softmax function.
We also considered reversed KL-Divergence. Empirically, MSE with target
feature outperforms other choices. This has also been observed recently in
[budnik2021asymmetric].

##  Appendix F Embedding Dimension ablation

In this section, we report the effect of embedding dimension in performance of
embedding models for ImageNet, Places-365, and VGGFace2. The old and new
embedding models’ top-1 retrieval performance is shown for different embedding
dimensions in Table 10. In all cases we directly modify the feature layer
output size, rather than projecting the original higher dimension output
(e.g., 2048-dimensionsonal features of ResNet50) to a lower dimension, as in
[shen2020towards]. Empirically, for ImageNet-1k and VGGFace2-8631 embedding
dimension of 128 obtains highest accuracy, while for Places-365, an embedding
dimension 512 performs the best. Interestingly, old model performance is
superior to new model performance for Places-365 at embedding dimensions 128
and 256, validating the notion that we need a higher embedding dimension for
that particular dataset.

##  Appendix G Compressed SimCLR results

[ramapuram2021stochastic] presents a method which results in a highly
compressed contrastive representation based on SimCLR with similar
performance. We used the method in [ramapuram2021stochastic] to train a SimCLR
model with feature dimension $\in\mathbb{Z}_{2}^{1}28$ on ImageNet-500 to be
used as side-information. This is a similar setup as in Table 1, but with a 32
times more compressed side-information feature vector. We report the numbers
for this case in Table 13. We see slightly worse performance than with
standard SimCLR (-0.9%), however this shows that side-information
representations can be compressed while still maintaining favorable
transformation properties.

Case | CMC top-1—5 (%) |<EMAIL_ADDRESS>
---|---|---
$h(\phi_{old},\psi)/h(\phi_{old},\psi)$ | 57.2 — 74.7 | 40.0
$\phi_{new}/h(\phi_{old},\psi)$ | 64.1 — 82.0 | 42.7

Table 13: Results for the ImageNet retrieval setup (see Section 4) with a
highly compressed SimCLR representation [ramapuram2021stochastic].

##  Appendix H Licenses

###  H.1 Software

PyTorch [paszke2019pytorch] is under the BSD license.

Python is under a Python foundation license (PSF)

###  H.2 Datasets

ImageNet [imagenet] has no license attached.

Places-365 [places365] is under the Creative Commons (CC BY) license.

VGGFace2 [cao2018vggface2] has no license attached.

_(ImageNet transformation training code on the next page)_

##  Appendix I ImageNet training code

Here we provide the code to train the ImageNet old to new transformation
function. Note that this requires model files for new, old, and side-
information to run. We cannot provide this at present due to size limits on
supplementary material. We will provide the full code upon complete release of
the paper.

⬇

import os

import torch

import torch.nn as nn

import torch.nn.functional as F

import torchvision

import argparse

\par\pardef main():

parser = argparse.ArgumentParser()

parser.add_argument(

”–side-info-model”,

default=None,

type=str,

help=”Path to side-info-model (torchscript)”,

)

parser.add_argument(”–old-model”, default=None, type=str, help=”Path to old
model”)

parser.add_argument(”–new-model”, default=None, type=str, help=”Path to new
model”)

parser.add_argument(

”–data”,

default=None,

type=str,

help=”Root directory of data, contains train and val imagenet folders”,

)

parser.add_argument(”–ngpus”, default=8, type=int, help=”How many GPUs to
use”)

args = parser.parse_args()

\parnew_dataset = load_imagenet_new_dataset(root_dir=args.data)

\parnew_model = torch.jit.load(args.new_model)

new_model = nn.DataParallel(new_model,
device_ids=list(range(args.ngpus))).to(0)

\parold_model = torch.jit.load(args.old_model)

old_model = nn.DataParallel(old_model,
device_ids=list(range(args.ngpus))).to(0)

\partransform_model = flat_2xrn50_to_rn50_with_bn_projection(

embedding_dim=128, norm_feature=False

)

transform_model = nn.DataParallel(

transform_model, device_ids=list(range(args.ngpus))

).to(0)

\parside_info_model = torch.jit.load(args.side_info_model)

side_info_model = nn.DataParallel(

side_info_model, device_ids=list(range(args.ngpus))

).to(0)

\paroptimizer = torch.optim.Adam(

parameters=transform_model.parameters(), lr=5e-4,
weight_decay=0.000030517578125

)

\parold_model.eval()

new_model.eval()

side_info_model.eval()

transform_model.train()

\parfor e in range(80):

if e > 40:

transform_model.eval()

\partrain_transform_fn(

new_dataset.train_loader,

transform_model=transform_model,

old_model=old_model,

new_model=new_model,

side_info_model=side_info_model,

optimizer=optimizer,

epoch=e,

)

\partorch.save(”transform_model.pth.tar”)

\par\pardef train_transform_fn(

train_loader,

transform_model,

old_model,

new_model,

side_info_model,

optimizer,

epoch,

):

criterion = nn.MSELoss()

for i, (batch, _) in enumerate(train_loader):

batch = batch.to(0)

with torch.no_grad():

transform = transform_model(old_model(batch), side_info_model(batch))

ground_truth = new_model(batch)

\parloss = criterion(transform, ground_truth)

\paroptimizer.zero_grad()

loss.backward()

optimizer.step()

\par\pardef load_imagenet_old_dataset(root_dir):

return ImageNet(root_dir, is_old=True)

\par\pardef load_imagenet_new_dataset(root_dir):

return ImageNet(root_dir, is_old=False)

\par\pardef conv1x1(in_features, out_features, bias=True):

return nn.Conv2d(

in_channels=in_features, out_channels=out_features, kernel_size=1, bias=bias

)

\par\pardef conv3x3(in_features, out_features, bias=True):

return nn.Conv2d(

in_channels=in_features, out_channels=out_features, kernel_size=3, bias=bias

)

\par\parclass flat_2xrn50_to_rn50_with_bn_projection(nn.Module):

def __init__(self, embedding_dim, norm_feature=True):

super().__init__()

\parself.p1 = nn.Sequential(

conv1x1(embedding_dim, 256, bias=True),

nn.BatchNorm2d(256),

nn.ReLU(),

conv1x1(256, 256, bias=True),

nn.BatchNorm2d(256),

nn.ReLU(),

)

\parself.p2 = nn.Sequential(

conv1x1(embedding_dim, 256, bias=True),

nn.BatchNorm2d(256),

nn.ReLU(),

conv1x1(256, 256, bias=True),

nn.BatchNorm2d(256),

nn.ReLU(),

)

\parself.mixer = nn.Sequential(

conv1x1(512, 2048, bias=True),

nn.BatchNorm2d(2048),

nn.ReLU(),

conv1x1(2048, 2048, bias=True),

nn.BatchNorm2d(2048),

nn.ReLU(),

conv1x1(2048, embedding_dim, bias=True),

)

\pardef forward(self, x, side_info):

x1 = self.p1(x)

x2 = self.p2(side_info)

\parout = self.mixer(torch.cat([x1, x2], dim=1))

\parif self.norm_feature:

return F.normalize(out, dim=1)

else:

return out

\par\parclass ImageNet:

def __init__(self, root, is_old=False, batch_size=1024, workers=24):

super(ImageNet, self).__init__()

\partrain_dataset, val_dataset = self.get_datasets(root)

self.train_dataset, self.val_dataset = train_dataset, val_dataset

\parif is_old:

self.train_dataset.imgs = [

(path, cls_num)

for path, cls_num in self.train_dataset.imgs

if cls_num < 500

]

self.train_dataset.samples = self.train_dataset.imgs

\parself.train_loader = torch.utils.data.DataLoader(

train_dataset,

batch_size=batch_size,

shuffle=True,

pin_memory=True,

num_workers=workers,

drop_last=True,

)

\parself.val_loader = torch.utils.data.DataLoader(

val_dataset,

batch_size=batch_size,

shuffle=False,

pin_memory=True,

num_workers=workers,

)

\pardef get_datasets(self, data_root):

# Data loading code

traindir = os.path.join(data_root, ”train”)

valdir = os.path.join(data_root, ”val”)

\parnormalize = torchvision.transforms.Normalize(

mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]

)

\partrainset = torchvision.datasets.ImageFolder(

traindir,

transform=torchvision.transforms.Compose(

[

torchvision.transforms.RandomResizedCrop(224),

torchvision.transforms.RandomHorizontalFlip(),

torchvision.transforms.ToTensor(),

normalize,

]

),

)

\parvalset = torchvision.datasets.ImageFolder(

valdir,

transform=torchvision.transforms.Compose(

[

torchvision.transforms.Resize(256),

torchvision.transforms.CenterCrop(224),

torchvision.ransforms.ToTensor(),

normalize,

]

),

)

\parreturn trainset, valset

\par

##  Appendix J AutoEncoder Architecture

⬇

import torch.nn as nn

\parclass SimpleAutoencoder(nn.Module):

def __init__(self, embedding_dim):

super().__init__()

\parbase_dim = 512

self.encoder = nn.Sequential(

Encoder(3, base_dim, 2, 32, stride=4),

Encoder(base_dim, base_dim, 2, 32, stride=2),

Encoder(base_dim, base_dim, 1, 32, stride=2),

Encoder(base_dim, embedding_dim, 1, 32, stride=2),

nn.AvgPool2d((7, 7)),

)

\parself.decoder = nn.Sequential(

nn.ConvTranspose2dembedding_dim, base_dim, 3, stride=1,

padding=0),

nn.ConvTranspose2d(base_dim, base_dim, 5, stride=1, padding=0),

Decoder(

in_channel=base_dim,

out_channel=base_dim,

channel=base_dim,

n_res_block=2,

n_res_channel=32,

stride=4,

),

Decoder(

in_channel=base_dim,

out_channel=base_dim,

channel=base_dim,

n_res_block=2,

n_res_channel=32,

stride=4,

),

Decoder(

in_channel=base_dim,

out_channel=3,

channel=base_dim,

n_res_block=1,

n_res_channel=32,

stride=2,

),

)

\pardef forward(self, x, feature_out=False):

encoding = self.encoder(x)

decoding = self.decoder(encoding)

\parif feature_out:

return decoding, encoding

else:

return decoding

\par\parclass ResBlock(nn.Module):

def __init__(self, in_channel, channel):

super().__init__()

\parself.conv = nn.Sequential(

nn.ReLU(),

nn.Conv2d(in_channel, channel, 3, padding=1),

nn.ReLU(inplace=True),

nn.Conv2d(channel, in_channel, 1),

)

\pardef forward(self, input):

out = self.conv(input)

out += input

\parreturn out

\par\parclass Encoder(nn.Module):

def __init__(self, in_channel, channel, n_res_block, n_res_channel,

stride):

super().__init__()

\parif stride == 4:

blocks = [

nn.Conv2d(in_channel, channel // 2, 4, stride=2, padding=1),

nn.ReLU(inplace=True),

nn.Conv2d(channel // 2, channel, 4, stride=2, padding=1),

nn.ReLU(inplace=True),

nn.Conv2d(channel, channel, 3, padding=1),

]

\parelif stride == 2:

blocks = [

nn.Conv2d(in_channel, channel // 2, 4, stride=2, padding=1),

nn.ReLU(inplace=True),

nn.Conv2d(channel // 2, channel, 3, padding=1),

]

\parfor i in range(n_res_block):

blocks.append(ResBlock(channel, n_res_channel))

\parblocks.append(nn.ReLU(inplace=True))

\parself.blocks = nn.Sequential(*blocks)

\pardef forward(self, input):

return self.blocks(input)

\par\parclass Decoder(nn.Module):

def __init__(

self, in_channel, out_channel, channel, n_res_block, n_res_channel,

stride

):

super().__init__()

\parblocks = [nn.Conv2d(in_channel, channel, 3, padding=1)]

\parfor i in range(n_res_block):

blocks.append(ResBlock(channel, n_res_channel))

\parblocks.append(nn.ReLU(inplace=True))

\parif stride == 4:

blocks.extend(

[

nn.ConvTranspose2d(channel, channel // 2, 4, stride=2,

padding=1),

nn.ReLU(inplace=True),

nn.ConvTranspose2d(

channel // 2, out_channel, 4, stride=2, padding=1

),

]

)

\parelif stride == 2:

blocks.append(

nn.ConvTranspose2d(channel, out_channel, 4, stride=2,

padding=1)

)

\parself.blocks = nn.Sequential(*blocks)

\pardef forward(self, input):

return self.blocks(input)