# Tailor-designed models for the turbulent velocity gradient through
normalizing flow

M. Carbone1,2<EMAIL_ADDRESS>V. J. Peterhans3,2 A. S.
Ecker4,2 M. Wilczek1,2<EMAIL_ADDRESS>1Theoretical Physics I,
University of Bayreuth, Germany
2Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany
3Faculty of Physics, University of Göttingen, Germany
4Institute of Computer Science and Campus Institute Data Science, University
of Göttingen, Germany

###### Abstract

Small-scale turbulence can be comprehensively described in terms of velocity
gradients, which makes them an appealing starting point for low-dimensional
modeling. Typical models consist of stochastic equations based on closures for
non-local pressure and viscous contributions. The fidelity of the resulting
models depends on the accuracy of the underlying modeling assumptions. Here,
we discuss an alternative data-driven approach leveraging machine learning to
derive a velocity gradient model which captures its statistics by
construction. We use a normalizing flow to learn the velocity gradient
probability density function (PDF) from direct numerical simulation (DNS) of
incompressible turbulence. Then, by using the equation for the single-time PDF
of the velocity gradient, we construct a deterministic, yet chaotic, dynamical
system featuring the learned steady-state PDF by design. Finally, utilizing
gauge terms for the velocity gradient single-time statistics, we optimize the
time correlations as obtained from our model against the DNS data. As a
result, the model time realizations statistically closely resemble the time
series from DNS.

Turbulence

Velocity gradients encode comprehensive information on the smallest scales of
turbulent flows Meneveau (2011); Johnson and Wilczek (2024), their statistics
unveiling hallmarks of turbulence including cascades Alexakis and Biferale
(2018); Carbone and Bragg (2020); Johnson (2020); Johnson and Wilczek (2024),
time irreversibility Jucha  _et al._ (2014); Cheminet  _et al._ (2022) and
intermittency She  _et al._ (1990); Buaria and Pumir (2021); Fuchs  _et al._
(2022). On a more applied level, velocity gradients also govern the small-
scale turbulent advection, stretching, and folding of material elements
Girimaji and Pope (1990a); Bentkamp  _et al._ (2022); Qi  _et al._ (2023), and
the collisional statistics of inertial particles transported by the flow
Balkovsky  _et al._ (2001); Bec  _et al._ (2007); Ireland  _et al._ (2016),
relevant e.g. for droplet growth in turbulent warm clouds Bodenschatz  _et
al._ (2010); Grabowski and Wang (2013). Reduced-order modeling of the velocity
gradient is thus appealing for a better theoretical understanding of
turbulence and for practical purposes.

A widely-employed and successful approach consists of modeling the Lagrangian
dynamics of the velocity gradient $\bm{A}\equiv\bm{\nabla u}$ (where
$\bm{u}(\bm{x},t)$ is a three-dimensional turbulent velocity field), following
fluid particle trajectories (e.g. Girimaji and Pope, 1990b; Chertkov  _et al._
, 1999; Naso and Pumir, 2005; Chevillard and Meneveau, 2006; Wilczek and
Meneveau, 2014; Johnson and Meneveau, 2016, 2017; Leppin and Wilczek, 2020).
However, this single-time/single-point modeling strategy has to deal with the
closure problem: Due to the intrinsic non-locality of turbulence, the
dynamical evolution of the velocity gradient at a point is not uniquely
determined solely by the velocity gradient configuration at that point. This
issue is evident from the equations governing the velocity gradient dynamics

$\displaystyle\textrm{d}_{t}\bm{A}=-\bm{A}^{2}-\bm{H}+\nu\nabla^{2}\bm{A}+\bm{\nabla
F},$ $\displaystyle\textrm{Tr}(\bm{A})$ $\displaystyle=0,$ (1)

where $\textrm{d}_{t}$ indicates the Lagrangian derivative taken following the
fluid particle trajectory, $\bm{\nabla F}$ represents an external forcing, Tr
indicates the matrix trace and standard matrix product is implied. The
pressure Hessian $\bm{H}$ and the viscous Laplacian $\nu\nabla^{2}\bm{A}$ are
nonlocal functionals of the velocity gradient field (e.g. Ohkitani and
Kishiba, 1995; Majda and Bertozzi, 2001). These unclosed terms cannot be
computed by knowing the local velocity gradient only, and they require
modeling when addressed from a single-time/single-point perspective.

Due to the intrinsic randomness of equations (1), reduced-order modeling of
the velocity gradient aims to predict the statistics of $\bm{A}(t)$, rather
than the individual time realizations. This is achieved by integrating a model
Langevin equation Girimaji and Pope (1990b); Chertkov  _et al._ (1999);
Chevillard and Meneveau (2006); Wilczek and Meneveau (2014); Johnson and
Meneveau (2016); Leppin and Wilczek (2020)

$\displaystyle\textrm{d}_{t}\bm{\mathcal{A}}=\bm{N}\\!\left(\bm{\mathcal{A}}\right)+\bm{\Gamma},$
$\displaystyle\textrm{Tr}(\bm{\mathcal{A}})=0,$ (2)

for an ensemble of initial conditions, where $\bm{N}$ contains a closure model
for the pressure Hessian and viscous term, and $\bm{\Gamma}$ is a Gaussian
noise, such that the steady-state statistics of the $3\times 3$ matrix
$\bm{\mathcal{A}}$ approximate the statistics of the turbulent velocity
gradient ensemble $\bm{A}$. The modeling step from Eq. (1) to (2) implies that
the right-hand side of (1) depends on the whole velocity field
$\bm{u}(\bm{x},t)$, while the right-hand side of (2) depends only on the
eight-dimensional state variable $\bm{\mathcal{A}}$. This drastic reduction of
degrees of freedom generates the aforementioned closure problem.

Many phenomenological models have been successful in qualitatively predicting
the statistical dynamics of the velocity gradient in isotropic turbulence
(e.g. Girimaji and Pope, 1990b; Chertkov  _et al._ , 1999; Naso and Pumir,
2005; Chevillard and Meneveau, 2006; Wilczek and Meneveau, 2014; Johnson and
Meneveau, 2016, 2017; Pereira  _et al._ , 2018; Leppin and Wilczek, 2020;
Johnson and Meneveau, 2018). More recently, data-driven approaches based on
machine learning Tian  _et al._ (2021); Buaria and Sreenivasan (2023) or
lookup tables Das and Girimaji (2023) have reached accuracy levels comparable
to the most advanced phenomenological closures. These recent approaches for
data-driven models Tian  _et al._ (2021); Buaria and Sreenivasan (2023); Das
and Girimaji (2023) approximate/learn the unclosed terms arising from (1).
Statistically, this amounts to estimating the average time derivative of the
velocity gradient conditional on the configuration of the velocity gradient
itself, i.e. $\left\langle\textrm{d}_{t}\bm{A}(t)|\bm{A}\right\rangle$, and
then integrating it in time to generate velocity gradient time realizations. A
major issue in this procedure is that an accurate a-priori estimation of the
conditional time derivatives does not guarantee that the a-posteriori
generated velocity gradient time series are realistic. This a-priori vs.
a-posteriori issue, also known from Large Eddy Simulations Jiménez and Moser
(2000), leads not only to inaccuracies in the predictions but may even imply
the blow-up of the gradient realizations over time. Diverging trajectories are
either neglected Buaria and Sreenivasan (2023) or damped via ad-hoc terms
Leppin and Wilczek (2020) to reach a non-trivial statistically steady state.
Another open problem of the current phenomenological and data-driven models is
that they can faithfully approximate single-time statistics, performing well
on certain marginal PDFs of the relatively high-dimensional velocity gradient
PDF, while leaving room for improvement on other marginal distributions and
two-time statistics (e.g. Chevillard and Meneveau (2006); Wilczek and Meneveau
(2014); Johnson and Meneveau (2016); Buaria and Sreenivasan (2023)). On a more
general level, recent alternatives that have the potential of improving the
classical reduced-order modeling include Lagrangian Large Eddy Simulations
Tian  _et al._ (2023) and time series generation by diffusive models Ho  _et
al._ (2020); Li  _et al._ (2023) or Generative Adversarial Networks Goodfellow
_et al._ (2014); Arjovsky  _et al._ (2017).

In this work, we propose an approach to construct velocity gradient models
that yield the correct velocity gradient statistics of homogeneous isotropic
turbulence by design. Earlier attempts to formulate such constrained models
consisted of imposing certain statistics such as the log-normality of the
velocity gradient magnitude Girimaji and Pope (1990b); Pereira  _et al._
(2018). Instead, here we constrain the model to yield the full distribution of
the velocity gradient: We train a neural network to predict the PDF of the
velocity gradient, $f(\bm{A})$, from a DNS dataset employing a normalizing
flow Tabak and Vanden-Eijnden (2010); Kobyzev  _et al._ (2021). Then we
construct a dynamical system in the form of Eq. (2) that has the learned PDF
as the steady-state probability density distribution. In the following, we
focus on a deterministic reduced-order model capable of maintaining a chaotic
statistically steady state (i.e. Eq. (2) without noise). This eliminates the
need for any ad-hoc stochastic forcing, which is required in existing models
to keep non-trivial steady-state statistics (e.g. Chevillard and Meneveau,
2006; Wilczek and Meneveau, 2014; Johnson and Meneveau, 2016; Leppin and
Wilczek, 2020; Tian  _et al._ , 2021; Das and Girimaji, 2023; Buaria and
Sreenivasan, 2023). Our deterministic model is associated with a Liouville
equation for the conservation of probability in the phase space

$\displaystyle\partial_{t}f+\partial_{ij}\left(N_{ij}f\right)=0,$ (3)

where $\partial_{ij}$ denotes tensor derivative with respect to
$\mathcal{A}_{ij}$, and summation over repeated indices is implied. Since we
know the intended steady-state solution $f(\bm{\mathcal{A}})$, we can derive
an expression for the vector field $\bm{N}$ such that Eq. (3) yields that
solution by construction, namely

$\displaystyle N_{ij}(\bm{\mathcal{A}};\\{\psi\\})$
$\displaystyle=\partial_{pq}T_{ijpq}+T_{ijpq}\partial_{pq}\log f,$ (4)

where $\bm{T}(\bm{\mathcal{A}};\\{\psi\\})$ is any smooth tensor function of
the velocity gradient such that $T_{ijpq}=-T_{pqij}$, and $\\{\psi\\}$ is a
set of model parameters. For stochastic generalizations of our approach, the
same can be done by replacing $\bm{T}$ with $\bm{T}+\bm{D}/2$ in Eq. (4),
where $\bm{D}(\bm{\mathcal{A}})$ is the correlation of the noise $\bm{\Gamma}$
in Eq. (2).

A drift of the form of Eq. (4) ensures that $f$ is a steady-state distribution
of our model, and this holds true for any tensor $\bm{T}$ that is anti-
symmetric in its index pairs. Therefore, imposing the single-time steady PDF
leaves the gauge term $\bm{T}$ undetermined, that is, there exists a class of
models with the same single-time distribution but different multi-time
statistics. Within this class, we can select the model that produces
realizations of the velocity gradient which are statistically closest to the
realizations in the DNS dataset. To this end, we optimize the conditional time
derivatives together with the time correlations of the velocity gradient. For
statistically isotropic turbulence, we can exploit additional gauge terms that
affect velocity gradient multi-time correlations while leaving the single-time
statistics unchanged for kinematic reasons Carbone  _et al._ (2020); Leppin
and Wilczek (2020). Those gauge terms stem from the particular functional form
of an isotropic velocity gradient PDF, which depends only on the five
invariants formed through the strain rate $\bm{S}=(\bm{A}+\bm{A}^{\top})/2$
and the vorticity $\bm{\omega}=\nabla\times\bm{u}$ Pennisi and Trovato (1987);
Lund and Novikov (1992)

$\displaystyle\mathcal{I}_{1}$
$\displaystyle=\textrm{Tr}\big{(}\bm{S}^{2}\big{)}$
$\displaystyle\mathcal{I}_{2}=\|\bm{\omega}\|^{2}/2$
$\displaystyle\mathcal{I}_{3}=\textrm{Tr}\big{(}\bm{S}^{3}\big{)}$
$\displaystyle\mathcal{I}_{4}$ $\displaystyle=\bm{\omega}\cdot\bm{S\omega}$
$\displaystyle\mathcal{I}_{5}=\bm{\omega}\cdot\bm{S}^{2}\bm{\omega}.$ (5)

The tensors $\widetilde{\bm{T}}$ which leave the dynamics of the invariants
(5) unchanged satisfy the kinematic condition

$\displaystyle\widetilde{T}_{ijpq}\partial_{pq}\mathcal{I}_{k}=0,$ (6)

and will help us optimize the velocity gradient multi-time correlations while
preserving its single-time statistics.

\begin{overpic}[width=433.62pt]{single_point.pdf} \put(32.0,47.0){(a)}
\put(32.0,23.0){(b)} \put(68.0,47.0){(c)} \put(67.0,23.0){(e)}
\put(93.0,47.0){(d)} \put(94.0,23.0){(f)} \end{overpic} Figure 1: Single-time
velocity gradient distributions from the DNS (colormaps and colored solid
lines) and from our model (black dashed lines). Panel (a) shows the joint PDF
of principal invariants of the velocity gradient,
$Q=-\textrm{Tr}(\bm{A}^{2})/2$ and $R=-\textrm{Tr}(\bm{A}^{3})/3$, panel (b)
shows the joint PDF of the strain invariants
$\mathcal{I}_{1}=\textrm{Tr}\big{(}\bm{S}^{2}\big{)}$ and
$\mathcal{I}_{3}=\textrm{Tr}\big{(}\bm{S}^{3}\big{)}$. Panel (c) shows the PDF
of the strain-rate and vorticity magnitudes; the PDFs of the longitudinal and
transverse velocity gradient components are shown in panel (d). Panels (e) and
(f) display the PDFs of the normalized vorticity components in the strain
eigenframe, $\hat{\omega}_{i}=\bm{v}_{i}\cdot\bm{\omega}/\|\bm{\omega}\|$,
where $\bm{v}_{i}$ are the strain-rate eigenvectors (index 1 denoting the most
extensional direction and 3 the most compressional). All dimensional
quantities are nondimensionalized by the Kolmogorov time scale,
$\tau_{\eta}=1/\sqrt{\left\langle 2\mathcal{I}_{1}\right\rangle}$.

Now that we have established the basic theory behind the method, let us
discuss its practical implementation. The proposed approach relies on a
parameterization of the velocity gradient PDF, a technical step that we
accomplish through a normalizing flow Tabak and Vanden-Eijnden (2010); Kobyzev
_et al._ (2021). To facilitate the implementation of the normalizing flow, we
unroll the components of the $3\times 3$ traceless matrix $\mathcal{A}_{ij}$
into an eight-dimensional vector $\bm{a}$. Then, starting from an ensemble of
eight-dimensional Gaussian vectors $\bm{a}^{(0)}$, we perform a sequence of
real, non-volume-preserving, quasi-linear transformations Dinh  _et al._
(2017). In each transformation layer, we update one variable through an affine
transformation, whose scaling $\Lambda^{(K)}$ and bias $b^{(K)}$ are computed
from the remaining seven variables. Then we repeat the sequence of
transformations four times for each variable

$\displaystyle
a^{(K)}_{i}=\begin{cases}\exp\left(\Lambda^{(K)}\right)a^{(K-1)}_{i}+b^{(K)}\textrm{
if }i=K\,(\bmod\,8)\\\ a^{(K-1)}_{i}\textrm{ otherwise},\end{cases}$ (7)

from transformation layer $K=1$ to $K=32$. The scaling functions
$\Lambda^{(K)}\left(\bm{a}^{(K-1)};\\{\theta^{(K)}\\}\right)$ and the bias
functions $b^{(K)}\left(\bm{a}^{(K-1)};\\{\theta^{(K)}\\}\right)$ in Eq. (7)
depend on all the gradient components except the one updated across that
layer, so that $\partial a_{i}^{(K)}/\partial
a_{j}^{(K-1)}=\exp(\Lambda^{(K)})$ for $i=K\,(\bmod\,8)$ and $i=j$. The
scaling and bias functions consist of two neural networks for each layer,
featuring two hidden linear layers with 64 neurons, Softplus activation
function and parameters $\\{\theta^{(K)}\\}$. The transformed eight-
dimensional vector $\bm{a}^{(32)}$ is mapped back to a $3\times 3$ traceless
matrix, $\bm{a}^{(32)}\Leftrightarrow\bm{\mathcal{A}}$, and the ensemble of
matrices $\bm{\mathcal{A}}$ aims to be statistically similar to the ensemble
of turbulent velocity gradients $\bm{A}$, through a maximum likelihood
criterion. Owing the quasi-linearity of the transformation (7), the log-
likelihood $\log f^{(K)}$ of the ensemble of matrices $\bm{\mathcal{A}}^{(K)}$
changes across a layer as

$\displaystyle\log f^{(K)}\big{(}\bm{\mathcal{A}}^{(K)}\big{)}=\log
f^{(K-1)}\big{(}\bm{\mathcal{A}}^{(K-1)}\big{)}+\Lambda^{(K)}.$ (8)

We find the optimal parameters in the transformations (7) and the
corresponding parameterization of the velocity gradient PDF by maximizing the
log-likelihood of the turbulent velocity gradient ensemble from the DNS

$\displaystyle\\{\bar{\theta}\\}=\operatorname*{argmax}_{\\{\theta\\}}\left\langle\log
f\left(\bm{A};\\{\theta\\}\right)\right\rangle,$ (9)

where $f\equiv f^{(32)}$ is the target velocity gradient PDF, $\bm{A}$ is an
element of the turbulent velocity gradient ensemble from the DNS, the angle
brackets denote ensemble average computed over the DNS dataset, and
$\\{\theta\\}$ stands for all the parameters featured in the transformations
(7).

The DNS dataset consists of the velocity gradient realizations along $2^{18}$
fluid particle Lagrangian trajectories from a DNS of three-dimensional,
incompressible, statistically steady and isotropic turbulence. The velocity
gradient time realizations span $64\tau_{\eta}$, and are sampled with a time
step $\tau_{\eta}/60$ (where $\tau_{\eta}$ is the Kolmogorov time scale). The
DNS resolves $1024^{3}$ Fourier modes with a maximum resolved wavenumber
$k_{\textrm{max}}=3/\eta$ (where $\eta$ is the Kolmogorov scale) and Reynolds
number based on the Taylor microscale $\textrm{Re}_{\lambda}=220$. The
normalizing flow implementation relies on Pytorch Paszke  _et al._ (2019). We
employ the Adam optimizer Kingma and Ba (2017) with a learning rate of
$5\times 10^{-5}$ and running average coefficients $\beta_{1}=0.9$,
$\beta_{2}=0.999$. We train the model on 40 GPUs for $5\times 10^{4}$ training
steps, on batches of 1024 velocity gradient time realizations.

\begin{overpic}[width=433.62pt]{trajectories.pdf} \put(46.0,34.1){(a)}
\put(46.0,7.2){(b)} \put(72.0,35.0){(c)} \put(72.0,18.0){(d)} \end{overpic}
Figure 2: Comparison of DNS and model trajectories, time correlations and
model convergence. Panels (a) and (b) show sample realizations of the
longitudinal velocity gradient component along fluid particle trajectories
from the DNS and our model. Panel (c) displays the normalized time
correlations of the strain rate and vorticity from the DNS (colored solid
lines) and our model (black dashed lines). Panel (d) shows the convergence of
our model (transparent black dashed lines) from Gaussian initial conditions to
turbulent-like statistics, as quantified by the log-likelihood of the velocity
gradient ensemble and the average velocity gradient invariants, as functions
of time. The red line indicates the DNS reference, with the transparent region
denoting a 10% accuracy interval. All dimensional quantities are
nondimensionalized by the Kolmogorov time scale,
$\tau_{\eta}=1/\sqrt{\left\langle 2\mathcal{I}_{1}\right\rangle}$.

The normalizing flow yields an accurate parameterization of the turbulent
velocity gradient PDF, as well as an invertible transformation (7) capable of
mapping an ensemble of Gaussian random matrices onto an ensemble resembling
the turbulent velocity gradients. This is in the same spirit of Gaussian
mapping closures in fluid dynamics Chen  _et al._ (1989); Kraichnan (1990). We
can now sample the learned steady-state PDF by using the transformation (7) or
by integrating our model (2) with vector field (4), and the results coincide.
Figure 1 shows a comparison between velocity gradient statistics from DNS data
and our model. The model captures several hallmark statistics of small-scale
turbulence with unprecedented accuracy, including: The PDF of the velocity
gradient (Fig. 1a) and strain-rate principal invariants (Fig. 1b), the
distribution of the strain-rate and vorticity magnitude (Fig. 1c) and the
velocity gradient Cartesian components (Fig. 1d), as well as the statistical
alignments between the vorticity and the strain-rate eigenframe (Fig. 1e-f).
The quantitative agreement between the model and the DNS data on several
marginal distributions from the velocity gradient PDF shows that we have
gained access to the full velocity gradient PDF.

Our data-driven approach decouples the single-time statistics modeling from
the optimization of the multi-time statistics. The anti-symmetric tensor
$\bm{T}(\bm{\mathcal{A}};\\{\psi\\})$ governs the velocity gradient multi-time
statistics, and we represent it by a neural network featuring three hidden
linear layers with 128 neurons, Softplus activation function and parameters
$\\{\psi\\}$. To optimize the parameters of $\bm{T}$, we numerically integrate
the model equations (2) for an ensemble of initial conditions sampled from the
DNS data, $\bm{\mathcal{A}}(0)\equiv\bm{A}(0)$, and require that the time
realizations
$\bm{\mathcal{A}}(t)=\bm{\mathcal{A}}(0)+\int_{0}^{t}\textrm{d}s\,\bm{N}(\bm{\mathcal{A}}(s);\\{\psi\\})$
have time correlations and conditional time derivatives similar to the DNS,

$\displaystyle\\{\bar{\psi}\\}=\operatorname*{argmin}_{\\{\psi\\}}\Bigg{(}\int_{0}^{T}\textrm{d}t\left\|\left\langle\mathcal{A}_{ij}(0)\mathcal{A}_{pq}(t)-A_{ij}(0)A_{pq}(t)\right\rangle\right\|^{2}$
$\displaystyle+\beta\sum_{k=1}^{5}\left\langle\left\|\partial_{ij}\mathcal{I}_{k}\left(\bm{A}\right)\left(\textrm{d}_{t}A_{ij}-N_{ij}\left(\bm{A};\\{\psi\\}\right)\right)\right\|^{2}\right\rangle\Bigg{)}$
(10)

where $\beta$ is a constant weighting parameter. For the minimization (10), we
employ the Adam optimizer with the parameters reported above for Eq. (9), a
time span $T=8\tau_{\eta}$ and weighting factor $\beta=10^{-5}$. The training
converges within $10^{3}$ train steps.

The optimization (10) results in a network $\bm{T}$, thus a tensor function
$\bm{N}$ (see Eq. (4)), such that the time correlations from the model
approximate the time correlations from the DNS data while preserving the
previously learned single-time PDF $f(\bm{\mathcal{A}})$ by construction.
Integrating the model (2) with the drift (4) optimized according to the
criterion (10) can be understood as a particular Monte Carlo sampling of the
PDF $f$ aided by normalizing flow Bialas  _et al._ (2023), in which the
sampling trajectories $\bm{\mathcal{A}}(t)$ have the meaning of actual time
realizations. Although the model was not designed to capture individual
velocity gradient time realizations, the time series generated by the model
look qualitatively similar to those from DNS, as displayed in Fig. 2(a-b). By
visual inspection, the DNS trajectories $A_{11}(t)$ and model trajectories
$\mathcal{A}_{11}(t)$ have similar amplitudes with large and negative
excursions occurring more frequently than positive ones, and the duration of
such excursions is comparable. Also quantitatively, the autocorrelation
functions of strain $C_{\bm{S}}=\left\langle S_{ij}(0)S_{ij}(t)\right\rangle$
and vorticity
$C_{\bm{\omega}}=\left\langle\omega_{i}(0)\omega_{i}(t)\right\rangle$ from the
model match well those of the DNS data, as Fig. 2(c) shows.

Since our model is deterministic, it is a priori unclear whether it approaches
a unique statistically steady state independent of initial conditions. To
answer this question, we performed numerical tests in which we observed
convergence from an ensemble of Gaussian initial conditions to the prescribed
steady-state PDF. We quantify the convergence by comparing the log-likelihood
of the model PDF $\left\langle\log f(\bm{\mathcal{A}}(t))\right\rangle$ to the
log-likelihood of the DNS data estimated via normalizing flow. Additionally,
we compare the average invariants from the model realizations and the DNS
data. As Fig. 2(d) shows, all quantities approach the expected steady-state
values for times longer than ${\cal O}(100\tau_{\eta})$. The convergence of
the deterministic system to a statistically steady state, independent of the
initial conditions, is a typical feature of a chaotic system Strogatz (2000).
Accordingly, we find the largest Lyapunov exponent for our model to be
positive, and the dynamics to be mildly dissipative, with an estimated Kaplan-
Yorke dimension of $7.96$. Those observations, combined with the convergence
to stationary statistics, indicate deterministic chaos.

Finally, the convergence to a target distribution showcases that it is
possible to construct generative machine-learning models which do not rely on
mapping stochastic signals into a distribution of target objects (e.g. Ho  _et
al._ , 2020), but rather use the chaotic nature of the underlying high-
dimensional mapping to mimic creativity.

Summarizing, we have developed a reduced-order deterministic velocity gradient
model that captures the statistical properties of the full eight-dimensional
velocity gradient PDF by design. This is achieved in a two-step procedure.
First, the full PDF is learned from DNS data using a normalizing flow Tabak
and Vanden-Eijnden (2010); Dinh  _et al._ (2017). Based on that, we construct
a Liouville equation which has the learned PDF as a stationary solution. The
Liouville equation is associated with a deterministic reduced-order model,
which allows to generate time realizations of the process. We then use gauge
freedom for the single-time statistics to optimize the time correlations and
conditional time derivatives of the model to emulate actual realizations of
Lagrangian velocity gradients in turbulence. The resulting deterministic
reduced-order model shows signs of chaotic dynamics. For example, it features
a positive Lyapunov exponent and converges to the expected statistically
stationary state from Gaussian initial conditions. A more detailed analysis of
the model dynamical system’s properties is an interesting subject for future
work.

###### Acknowledgements.

This project has received funding from the European Research Council (ERC)
under the European Union’s Horizon 2020 research and innovation programme
(Grant agreement No. 101001081). The authors gratefully acknowledge the
scientific support and HPC resources provided by the Erlangen National High
Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität
Erlangen-Nürnberg (FAU) under the NHR project b159cb EnSimTurb. Computational
resources from the Max Planck Computing and Data Facility and support by the
Max Planck Society are gratefully acknowledged.

## References

  * Meneveau (2011) C. Meneveau, Annu. Rev. Fluid Mech. 43, 219 (2011).
  * Johnson and Wilczek (2024) P. L. Johnson and M. Wilczek, Annu. Rev. Fluid Mech. 56, 463 (2024).
  * Alexakis and Biferale (2018) A. Alexakis and L. Biferale, Phys. Rep. 767–769, 1 (2018).
  * Carbone and Bragg (2020) M. Carbone and A. D. Bragg, J. Fluid Mech. 883, R2 (2020).
  * Johnson (2020) P. L. Johnson, Phys. Rev. Lett. 124, 104501 (2020).
  * Jucha  _et al._ (2014) J. Jucha, H. Xu, A. Pumir, and E. Bodenschatz, Phys. Rev. Lett. 113, 054501 (2014).
  * Cheminet  _et al._ (2022) A. Cheminet, D. Geneste, A. Barlet, Y. Ostovan, T. Chaabo, V. Valori, P. Debue, C. Cuvier, F. Daviaud, J.-M. Foucaut, J.-P. Laval, V. Padilla, C. Wiertel-Gasquet, and B. Dubrulle, Phys. Rev. Lett. 129, 124501 (2022).
  * She  _et al._ (1990) Z.-S. She, E. Jackson, and S. A. Orszag, Nature 344, 226 (1990).
  * Buaria and Pumir (2021) D. Buaria and A. Pumir, Phys. Rev. Res. 3, L042020 (2021).
  * Fuchs  _et al._ (2022) A. Fuchs, C. Herbert, J. Rolland, M. Wächter, F. Bouchet, and J. Peinke, Phys. Rev. Lett. 129, 034502 (2022).
  * Girimaji and Pope (1990a) S. S. Girimaji and S. B. Pope, J. Fluid Mech. 220, 427–458 (1990a).
  * Bentkamp  _et al._ (2022) L. Bentkamp, T. D. Drivas, C. C. Lalescu, and M. Wilczek, Nat. Commun. 13, 2088 (2022).
  * Qi  _et al._ (2023) Y. Qi, C. Meneveau, G. A. Voth, and R. Ni, Phys. Rev. Lett. 130, 154001 (2023).
  * Balkovsky  _et al._ (2001) E. Balkovsky, G. Falkovich, and A. Fouxon, Phys. Rev. Lett. 86, 2790 (2001).
  * Bec  _et al._ (2007) J. Bec, L. Biferale, M. Cencini, A. Lanotte, S. Musacchio, and F. Toschi, Phys. Rev. Lett. 98, 084502 (2007).
  * Ireland  _et al._ (2016) P. J. Ireland, A. D. Bragg, and L. R. Collins, J. Fluid Mech. 796, 617 (2016).
  * Bodenschatz  _et al._ (2010) E. Bodenschatz, S. P. Malinowski, R. A. Shaw, and F. Stratmann, Science 327, 970 (2010).
  * Grabowski and Wang (2013) W. W. Grabowski and L.-P. Wang, Annu. Rev. Fluid Mech. 45, 293 (2013).
  * Girimaji and Pope (1990b) S. S. Girimaji and S. B. Pope, Phys. Fluids 2, 242 (1990b).
  * Chertkov  _et al._ (1999) M. Chertkov, A. Pumir, and B. I. Shraiman, Phys. Fluids 11, 2394 (1999).
  * Naso and Pumir (2005) A. Naso and A. Pumir, Phys. Rev. E 72, 056318 (2005).
  * Chevillard and Meneveau (2006) L. Chevillard and C. Meneveau, Phys. Rev. Lett. 97, 174501 (2006).
  * Wilczek and Meneveau (2014) M. Wilczek and C. Meneveau, J. Fluid Mech. 756, 191 (2014).
  * Johnson and Meneveau (2016) P. L. Johnson and C. Meneveau, J. Fluid Mech. 804, 387 (2016).
  * Johnson and Meneveau (2017) P. L. Johnson and C. Meneveau, Phys. Rev. Fluids 2, 072601(R) (2017).
  * Leppin and Wilczek (2020) L. A. Leppin and M. Wilczek, Phys. Rev. Lett. 125, 224501 (2020).
  * Ohkitani and Kishiba (1995) K. Ohkitani and S. Kishiba, Phys. Fluids 7, 411 (1995).
  * Majda and Bertozzi (2001) A. J. Majda and A. L. Bertozzi,  _Vorticity and Incompressible Flow_ (2001).
  * Pereira  _et al._ (2018) R. M. Pereira, L. Moriconi, and L. Chevillard, J. Fluid Mech. 839, 430 (2018).
  * Johnson and Meneveau (2018) P. L. Johnson and C. Meneveau, J. Fluid Mech. 837, 80–114 (2018).
  * Tian  _et al._ (2021) Y. Tian, D. Livescu, and M. Chertkov, Phys. Rev. Fluids 6, 094607 (2021).
  * Buaria and Sreenivasan (2023) D. Buaria and K. R. Sreenivasan, Proc. Natl. Acad. Sci. U.S.A. 120 (2023).
  * Das and Girimaji (2023) R. Das and S. S. Girimaji, (2023), arXiv:2304.14529 .
  * Jiménez and Moser (2000) J. Jiménez and R. D. Moser, AIAA J. 38, 605 (2000).
  * Tian  _et al._ (2023) Y. Tian, M. Woodward, M. Stepanov, C. Fryer, C. Hyett, D. Livescu, and M. Chertkov, Proc. Natl. Acad. Sci. U.S.A. 120, e2213638120 (2023).
  * Ho  _et al._ (2020) J. Ho, A. Jain, and P. Abbeel, in  _Advances in Neural Information Processing Systems_, Vol. 33 (2020) pp. 6840–6851.
  * Li  _et al._ (2023) T. Li, L. Biferale, F. Bonaccorso, M. A. Scarpolini, and M. Buzzicotti, (2023), arXiv:2307.08529 .
  * Goodfellow  _et al._ (2014) I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, in  _Advances in Neural Information Processing Systems_, Vol. 27 (2014).
  * Arjovsky  _et al._ (2017) M. Arjovsky, S. Chintala, and L. Bottou, in  _Proceedings of the 34th International Conference on Machine Learning_, Vol. 70 (2017) pp. 214–223.
  * Tabak and Vanden-Eijnden (2010) E. G. Tabak and E. Vanden-Eijnden, Commun. Math. Sci. 8, 217 (2010).
  * Kobyzev  _et al._ (2021) I. Kobyzev, S. J. Prince, and M. A. Brubaker, IEEE Trans. Pattern Anal. Mach. Intell. 43, 3964 (2021).
  * Carbone  _et al._ (2020) M. Carbone, M. Iovieno, and A. D. Bragg, J. Fluid Mech. 900, A38 (2020).
  * Pennisi and Trovato (1987) S. Pennisi and M. Trovato, Int. J. Eng. Sci. 25, 1059 (1987).
  * Lund and Novikov (1992) T. Lund and E. A. Novikov, Center for Turbulence Research (Stanford University and NASA) (1992).
  * Dinh  _et al._ (2017) L. Dinh, J. Sohl-Dickstein, and S. Bengio, in  _International Conference on Learning Representations_ (2017).
  * Paszke  _et al._ (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, (2019), arXiv:912.01703 .
  * Kingma and Ba (2017) D. P. Kingma and J. Ba, (2017), arXiv:1412.6980 .
  * Chen  _et al._ (1989) H. Chen, S. Chen, and R. H. Kraichnan, Phys. Rev. Lett. 63, 2657 (1989).
  * Kraichnan (1990) R. H. Kraichnan, Phys. Rev. Lett. 65, 575 (1990).
  * Bialas  _et al._ (2023) P. Bialas, P. Korcyl, and T. Stebel, (2023), arXiv:2308.13294 .
  * Strogatz (2000) S. H. Strogatz,  _Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering_ (Westview Press, 2000).


# Generalization to translation shifts: a study in architectures and
augmentations

Suriya Gunasekar
Microsoft Research
<EMAIL_ADDRESS>
###### Abstract

We study how effective data augmentation is at capturing the inductive bias of
carefully designed network architectures for spatial translation invariance.
We evaluate various image classification architectures (antialiased,
convolutional, vision transformer, and fully connected MLP networks) and data
augmentation techniques towards generalization to large translation shifts. We
observe that: (a) without data augmentation, all architectures, including
convolutional networks with antialiased modification suffer some degradation
in performance when evaluated on translated test distributions.
Understandably, both the in-distribution accuracy and degradation to shifts is
significantly worse for non-convolutional models. (b) The robustness of
performance is improved by even a minimal augmentation of $4$ pixel random
crop across all architectures. In some instances, even $1-2$ pixel random crop
is sufficient. This suggests that there is a form of meta generalization from
augmentation. For non-convolutional architectures, while the absolute accuracy
is still low with this basic augmentation, we see substantial improvements in
robustness to translation shifts. (c) With a sufficiently advanced
augmentation pipeline ($4$ pixel crop+RandAugmentation+Erasing+MixUp), all
architectures can be trained to have competitive performance in terms of in-
distribution accuracy as well as generalization to large translation shifts.

##  1 Introduction

Convolutional neural networks (ConvNets) are a natural architectural choice
for a variety of computer vision tasks. The built-in structure from
localization and translation equivariance of the convolutional layers is
intrinsically useful in many image processing scenarios (Krizhevsky et al.,
2012; LeCun et al., 1989; Fukushima and Miyake, 1982). For over a decade
ConvNets were the backbone of computer vision and continue to be one of the
most important class of models. At the same time, advances in large datasets
and data augmentation techniques have made it possible to train general
purpose architectures to be competitive with ConvNets despite lacking any
image specific priors. Most popular among these are the Vision Transformers
(ViTs) and their variants (Vaswani et al., 2017; Dosovitskiy et al., 2020;
Touvron et al., 2021c, b). When pretrained on ultra-large datasets like
ImageNet-21k (14 million images) or JFT-300/3B (300 million/3 billion weakly
labeled images, respectively), ViTs can outperform similarly pretrained
ConvNets on diverse vision tasks. While the scale of the data was original
thought to be crucial, follow up work show that competitive accuracies can
also be achieved in small-to-medium data regimes using advanced data
augmentation (Touvron et al., 2021b) or optimization techniques (Chen et al.,
2021). Detailed experimentation on competitive benchmarks by Steiner et al.
(2021) showed that ViTs trained with extensive data augmentation can recover
the performance gains from $\sim 10$x larger independently annotated dataset.
In such large data or extensive augmentation regimes, even simpler fully
connected multi-layer perceptron (MLP) architectures (Tolstikhin et al., 2021;
Touvron et al., 2021a) can achieve competitive performance. In this work we
focus on the role of data augmentation in learning from general purpose
architectures.

Can data augmentation capture the inductive biases of carefully designed
architectures? Beyond accuracy, the design of architectures is often motivated
by domain knowledge of desired invariances. One of the fundamental image
priors is the invariance of object labels to spatial shift or translation of
its position. Indeed, the success and motivation of ConvNets is often
attributed to their component convolutional operators being definitionally
translation equivariant. In this work, we study to what extent the shift-
equivariant convolutional layers make ConvNets robust to translation shifts
compared to ViTs and MLPs? How effective are data augmentation techniques in
encouraging similar behaviors?

There is a rich literature on understanding the translation invariance
properties of ConvNets (see Section 1.1). It has been shown that despite the
equivariance of the convolutional operator, other architectural components
like non-linearities and strides can cause the networks lose their invariance
to translations (Zhang, 2019; Azulay and Weiss, 2019; Chaman and Dokmanic,
2021; Engstrom et al., 2019; Xiao et al., 2018; Alsallakh et al., 2020).
Despite these findings, it is reasonable to believe that ConvNets will still
have substantial edge in robustness to translation when compared to ViT or MLP
models. At the same time, the competitive performance of latter architectures
suggests that good image priors can also be learned from rich training data.
The goal of this work is to quantify the relative effectiveness of
architectures design and data augmentation towards robustness to translation
shifts. Algorithmically, these are complementary tools for incorporating
domain knowledge. On one hand, it is conceptually simpler to generate data
with desired invariant transformation rather than hard code it in the
architecture. On the other hand, data augmentation only provides a weak
supervision about the invariant properties and is limited by the biases in the
training samples.

#### Generalization vs invariance.

In our experiments, rather than chase the gold standard of strict invariance
on all inputs, we work with a data-centric measure of generalization to
translation shifts in the test distribution. We evaluate our models for
accuracy on specific out-of-distribution test datasets where the object
locations are systematically shifted without creating additional distortions
or domain gaps from training distribution. See Section 2 for full setup.

In the strict sense, a classifier $f:X\to Y$ on input space $X$ is invariant
to set of spatial translations $T$ if for all $t\in T$ and all inputs $x\in
X$, $f(t(x))\approx f(x)$. Often we are not concerned with performance on all
inputs but rather on typical samples from a task distribution, say $(x,y)\sim
D$. We thus evaluate our models a structured out-of-distribution accuracy,
wherein we train on samples from $D$, but test on translation shifted inputs
$t(x)$, i.e., generalized accuracy to shift $t$ is $\mathbb{E}_{(x,y)\sim
D}\mathbf{1}[f(t(x))=y]$. This evaluation also differs from adversarial
robustness to translation shifts as studied in Engstrom et al. (2019); Xiao et
al. (2018). In our notation, the adversarial accuracy metric would be
$\mathbb{E}_{(x,y)\sim D}\mathbf{1}[f(t_{x}(x))\neq y]$, where $t_{x}\in T$ is
adversarially chosen per-sample $(x,y)$. Finally, we clarify that data
augmentation does change the training distribution $D$. If inputs $x$ were to
be augmented with their respective transformations $\\{t(x):t\in T\\}$ (and no
other augmentation), then there is no distribution shift. However, our
experiments are never in the no-distribution shift regime. Importantly, all
our augmentation pipelines uses random crop of at most $4$ pixels (some are
further restricted to $1$ or $2$ pixels), but we evaluate our models on much
larger translation shifts of up to $8$ pixels on CIFAR and $16$ pixels on
TinyImageNet.

###  1.1 Related work

#### Translation invariance in ConvNets.

ConvNets have been extensively studied on various approximate measures of
invariance to translation shifts including, but not limited to, Goodfellow et
al. (2009); Zeiler and Fergus (2014); Fawzi and Frossard (2015); Kanbak et al.
(2018); Azulay and Weiss (2019); Zhang (2019); Engstrom et al. (2019); Kayhan
and Gemert (2020); Chaman and Dokmanic (2021). For example, Zeiler and Fergus
(2014) provide visualizations of the hidden layer filters in early ConvNets
that show their sensitivity to small changes in translation, scale, and
rotation. Similar visual evaluation was also more recently used in Alsallakh
et al. (2020). Other work Fawzi and Frossard (2015); Kanbak et al. (2018);
Azulay and Weiss (2019); Zhang (2019); Engstrom et al. (2019) focus on more
quantitative measures of invariance such as mean change in top-$1$ prediction
or class probabilities between images within a distortion range. These works
collectively establish that despite the built-in inductive bias, ConvNets are
not “truly" invariant even to small translation. Some recent work, importantly
(Zhang, 2019; Chaman and Dokmanic, 2021), address this shortcoming of ConvNets
by designing new architectural modifications to ConvNets. These prior works
show that strict spatial invariance is a strong measure that even ConvNets
with their built-in priors do not satisfy. We choose a data-centric measure in
our evaluation that is more practical for general architectures. Our
experimental and evaluation protocol has some key differences from these prior
works studying invariance: (a) Our evaluation is designed to introduce
translations shifts in the test data without with any additional confounding
domain gaps between train and test distributions. For example, in the
experiments of Azulay and Weiss (2019); Zhang (2019) the test samples are
padded and resized during evaluation, while the models were trained on regular
dataset without such padding or resizing. This leads the test dataset to
differ from the training data in ways that are not just translation of the
objects. (b) Our evaluation is non-adversarial in that it measures degradation
in average test loss and not in the worst-case drop in performance on any
single image. Further, by comparing the drop in performance from translation
shift to the unperturbed in-distribution accuracy, we inherently down-weight
the non-robustness on hard-to-learn inputs on which the classifier had
inaccurate prediction to begin with. (c) Finally, our evaluation does not
penalize models from learning position dependent features as observed by
Kayhan and Gemert (2020). For example, a network has the flexibility to use
its large representation power to create a separate model for an object (say a
cat) at each pixel location — while such a model will be inefficient, it would
still do a good job at detecting cats in translated test distribution. These
differences are nuanced but significant, which makes our results complementary
to prior work in this space.

#### Robustness under adversarial perturbations and other OOD benchmarks.

In a work closest to ours, Engstrom et al. (2019) study test accuracy
degradation from adversarially chosen translations and rotations on test
images. While much of the work in adversarial robustness focus on
$\ell_{\infty}$ or $\ell_{2}$ norm bounded perturbations to the inputs,
Engstrom et al. (2019) show that ConvNets can be effectively “attacked" even
when the perturbed inputs are merely small rotations and/or translations of
the input. Similar adversarial attack on ConvNets based on more complex
spatial transformations was previously studied in (Xiao et al., 2018). Our
evaluation is closer to “random perturbation" evaluation, which is only
briefly explored in Engstrom et al. (2019). In comparison to these studies,
our distribution shift is picked non-adversarially and independent of input
samples.

Complementing the work on adversarial robustness, there has also been lot of
interest in evaluating models on other out-of-distribution robustness.
Generalization to out-of-distribution test datasets is a broad umbrella topic,
and aside from adversarial robustness, many work also focus on robustness
performance on benchmarks for benign “natural” perturbations Recht et al.
(2019); Hendrycks and Dietterich (2018); Koh et al. (2021); Djolonga et al.
(2021). For ConvNets, inspired by the neural networks scaling laws line of
work, Djolonga et al. (2021) probe for relation between OOD robustness and
learning choices like data size, model size, optimization algorithm, and model
choices like model sizes, and normalization (they do not consider data
augmentation in detail though). They also propose a synthetic benchmark SI-
SCORE for controlled image invariance evaluation. Expanding on this line of
work, Yung et al. (2021); Bhojanapalli et al. (2021); Bai et al. (2021);
Mahmood et al. (2021); Shao et al. (2021); Paul and Chen (2021); Pinto et al.
(2022), compare ConvNets and ViTs on adversarial robustness and/or out-of-
distribution robustness benchmarks.

Aside from the methodological differences, much of the work in this space has
focused on the relative merits, demerits, and robustness of different model
choices and the role of training data size. The effects of data augmentation
is only minimally considered, if at all. In contrast, our goal in this work is
to specifically quantify how much different data augmentation pipelines can
capture the inductive biases in a carefully designed architecture.

##  2 Experimental setup

All our experiments are conducted on the CIFAR-10, CIFAR-100, and TinyImageNet
datasets. Since our study involves training from scratch and testing on large
models in numerous configurations, it is beyond the scope of the paper to
extend such a detailed study to larger benchmarks like full ImageNet.
Moreover, we emphasize that our goal here is not to get the state-of-the-art
accuracy/robustness on benchmarks, but rather to understand how much data
augmentation captures the benefits of the convolutional architecture.
Arguably, it is also the small data regime where the inductive biases from
architecture and/or augmentations play more important roles. In ultra-large
scale datasets, accuracy/robustness might naturally come from dataset size
itself rather than model priors.

In the main paper we focus on results from CIFAR-10 and CIFAR-100 datasets
which consists of $32\times 32$ pixel RGB images balanced across $10$ and
$100$ classes, respectively. We defer the discussion on TinyImageNet, which a
subset of the more diverse ImageNet benchmark, to Appendix C. To study large
translation shifts without introducing domain gaps, we modify the dataset by
symmetrically padding all the CIFAR images with $8$ pixels ($1/4$ of image
size) on each side. The padded pixels contain the mean channel values of the
entire training dataset, which ensures that the channel-wise means and
standard deviations across training dataset remains the same as the original
un-padded dataset (see illustration in Figure 1). This padded dataset allows
us to evaluate large translation shifts of up to $16$ pixels (Hamming
distance) in the test dataset without creating additional confounding factors.
Importantly, in all the shifted test sets, there is no cropping or loss of the
image content and the entire image is available to the network at the same
scale as seen during training. After padding with a mean-valued canvas, we
resize the resulting $48\times 48\times 3$ images to $224\times 224\times 3$
(the standard input size for ImageNet) using bilinear interpolation. This up-
sampling step helps avoid extensive hyperparameter tuning of the models,
especially, the ViT and MLP models.

We briefly discuss the alternative evaluation methodologies. The more natural
random cropping of images to evaluate robustness to translation is inherently
limited by the number of pixels we can faithfully forgo without losing
information and hence cannot capture large translation shifts. In prior work,
Azulay and Weiss (2019) also used similar padded images to investigate
translation shifts. A key difference in our methodology is that we have our
entire training and testing pipeline on the preprocessed images (with
padding), while the latter paper evaluated models pretrained on standard
ImageNet without any padding – this creates an uncontrolled distribution
shifts. Another technical difference is that Azulay and Weiss (2019)
downsampled the images, which leads to loss in resolution, while our
preprocessing is non-lossy. Finally, synthetic benchmarks such as SI-Score
proposed in Djolonga et al. (2021) are a good alternative to our setup.
However, for our simple controlled setting, the conceptual advantage of
padding is that it does not change the natural distribution of foreground and
background which would be lost in the cut-and-paste protocol of Djolonga et
al. (2021). Furthermore, the segmentation process in Djolonga et al. (2021)
appears to be not perfect which might create additional confounders. To
further ensure that our “synthetic" padding is benign, we verified that the
test accuracy of our models trained on the padded dataset is comparable to the
standard train-test pipeline on $32\times 32\times 3$ CIFAR inputs without any
padding (see Table 1).

(a) Original

(b) Mean padding

(c) Resize to $224\times 224$

(d) Shifted test sample

Figure 1: Sample images from the prepossessing steps

#### Training

All the models are trained on the mean-padded datasets for $1600$ epochs on
$8\times V100$ GPUs. We use implementations (with suitable modifications) of
the models from various open source repositories, most notably Wightman (2019)
and Liu (2017). We performed basic hyperparameter tuning in a small grid
around the parameters reported in the respective papers. The exact value of
hyperparameters used in experiments along with code are provided in the
supplementary material. All the evaluation metrics reported in this paper are
median performance over $3$ runs.

###  2.1 Architectures

Our goal is to compare fundamentally different architectures on generalization
to translation shifts. After initial experiments with different variants, we
choose the following models in our evaluation.

  * •

resnet18_bn (11M parameters): We use ResNets He et al. (2016) as our
representative ConvNet. In our initial experiments, larger ResNets and the
other ConvNets like RegNet Radosavovic et al. (2020) did not yield qualitative
difference in performance on our datasets. We use the standard ResNet-18
architecture with one modification that we set stride of the first $7\times 7$
base convolution to $1$ rather than the standard of $2$. We do this as it gave
better results with the resized datasets, specially with the group norm
variation discussed below.

  * •

resnet18_gn (11M parameters): It has been observed that batch normalization
often leads to poor performance in transfer learning as the batch statistics
from source task could be widely off for target task (see e.g., Kolesnikov et
al. (2020); Wu and Johnson (2021)). The same reasoning also applies when
dealing with distribution shifts, where batch statistics could become
irrelevant even when the test distribution shifts systematically Djolonga et
al. (2021). To overcome this, we consider a variant of resnet18 with group
normalization and weight standardization Wu and He (2018); Qiao et al. (2019)
in place of batch normalization. This variation indeed leads to more robust
performance in our experiments, specially in the absence of data augmentation.

  * •

antialiased_resnet18 (11M parameters): Azulay and Weiss (2019); Zhang (2019)
show that the downsampling layers (pooling/strides) make ConvNets non-shift
invariant. To remedy this, Zhang (2019) proposed a a specialized modification
to ConvNets to improve their invariance to spacial shifts by introducing a
BlurPool layer as an antialiasing-filter before downsampling. This constitute
a model with more specialized priors about translation invariance built into
its design.

  * •

cait_xxs36 (17M parameters): CaiT architecture Touvron et al. (2021c) is a
variant of the basic vision transformer (ViT) Dosovitskiy et al. (2020) that
leads to more efficient training of deeper models. We use CaiT as our
representative transformer model as it had the best performance in initial
experiments. Other ViT variants, including larger models and the distilled
variant DeiT Touvron et al. (2021b) did not provide significant performance
boost on our small scale datasets.

  * •

resmlp_12 (18M parameter): Among the MLP models for image classification, we
tried MLP-mixer Tolstikhin et al. (2021) and ResMLP Touvron et al. (2021a) in
our initial experiments. We stick with ResMLP for detailed experimentation as
it had slightly better performance.

Although the above model configurations are not the state-of-the-art on larger
benchmarks, on smaller scale CIFAR and TinyImageNet datasets, they have
competitive performance as their larger or more complex counterparts. Since
our goal is to evaluate the relative degradation in performance with
translation shifts, we do not overly optimize for top-accuracy.

###  2.2 Augmentations

We first consider four data augmentation pipelines while training the models
described above. In the appendix, we look at more minimal augmentations to
elaborate on our findings.

  * •

No Augmentation (NoAug): We use this setting as a baseline for purely
evaluating the merits of an architecture in generalization to translation
shifts.

  * •

Basic augmentation (BA): The basic augmentation consists of a random flip and
a random crop with up to $4\times 4$ pixel padding. This minimal augmentation
has been a de-facto standard in many vision tasks, and it already gives over
$5\%$ boost in accuracy even without considering its effects on distribution
shifts. Note that unlike in standard training pipeline, with our padding of
training images, the random crop does not lose any original image pixels.

  * •

Advanced augmentation (AA): The current slate of image data augmentation
techniques are more varied and less intuitive compared to the simple
transformations described above. In our version of advanced augmentation (AA),
we use the following pipeline: first we apply (a) the basic augmentation (BA)
described above, then (b) RandAugment Cubuk et al. (2019), then (c) random
erasing Zhong et al. (2020), and finally use (d) MixUp Inoue (2018).
RandAugment uses a randomly chosen composition of transformations from a
predefined list. We use the standard RandAugment list from Wightman (2019) but
without the TranslationX and TranslationY as these are covered with more
control within basic augmentation.

  * •

AA without translation (AA(no tr)): We also consider a variant of advanced
augmentation (AA) where we remove any augmentations that are explicitly
related translation shifts. Specifically, we remove random crop from basic
augmentation (BA) and in the RandAugment transformations list, along with
previously removed TranslationX and TranslationY, we also remove ShearX and
ShearY. In this case, any improvements over NoAug arise only indirectly.

###  2.3 In-distribution test accuracy

Table 1 gives the test accuracies on the in-distribution test dataset, which
is preprocessed with the same padding configuration as the train dataset
(i.e., with $8$ pixels padded symmetrically on all sides). We use this as a
reference performance without any distribution shifts (a.k.a. the in-
distribution  accuracy). Ideally, we would expect a classifier that that
learns good image priors to maintain their reference performance even when the
test dataset is shifted by object invariant properties.

|  |  resent18_bn |  resnet18_gn |  antialiased _resnet18 |  cait_xxs36 |  resmlp_12
---|---|---|---|---|---|---
CIFAR-10 |  NoAug |  $90.85_{\pm 0.19}$ |  $91.48_{\pm 0.09}$ |  $92.55_{\pm 0.21}$ |  $77.58_{\pm 0.11}$ |  $79.99_{\pm 0.15}$
BA |  $96.10_{\pm 0.05}$ |  $95.96_{\pm 0.06}$ |  $95.63_{\pm 0.15}$ |  $87.69_{\pm 0.50}$ |  $87.73_{\pm 0.07}$
AA(no-tr) |  $96.35_{\pm 0.06}$ |  $96.06_{\pm 0.07}$ |  $96.98_{\pm 0.07}$ |  $95.09_{\pm 0.19}$ |  $91.90_{\pm 0.10}$
AA |  $97.74_{\pm 0.03}$ |  $98.03_{\pm 0.06}$ |  $97.77_{\pm 0.10}$ |  $97.25_{\pm 0.01}$ |  $96.03_{\pm 0.09}$
CIFAR-100 |  NoAug |  $67.62_{\pm 0.65}$ |  $64.62_{\pm 0.29}$ |  $70.37_{\pm 0.09}$ |  $43.43_{\pm 0.12}$ |  $52.79_{\pm 0.21}$
BA |  $78.68_{\pm 0.18}$ |  $78.42_{\pm 0.08}$ |  $78.20_{\pm 0.16}$ |  $57.62_{\pm 0.78}$ |  $60.52_{\pm 0.60}$
AA(no-tr) |  $74.56_{\pm 0.46}$ |  $74.09_{\pm 0.22}$ |  $77.62_{\pm 0.25}$ |  $77.74_{\pm 1.40}$ |  $65.43_{\pm 0.30}$
AA |  $82.98_{\pm 0.14}$ |  $82.09_{\pm 0.27}$ |  $81.54_{\pm 0.17}$ |  $82.46_{\pm 0.27}$ |  $78.63_{\pm 0.28}$
Table 1: Accuracy on in-distribution test dataset (i.e., test and train images
have same padding).

##  3 Generalization to translation shifts

With the flexibility of padded pixels in our preprocessed training data, we
can now create test datasets with up to $16$ pixel translations (in Hamming
distance) from the training image distribution by moving the test images
anywhere within the $48\times 48$ frame. This allows us to evaluate large
translation while not distorting the contents of the image itself. In the
extreme locations (see, e.g., corners of grid in Figure 2) there only $25\%$
overlap with the training distribution.

For each trained model, we can look at a grid of $17\times 17$ test
evaluations on modified test datasets. Each cell in the evaluation grid
corresponds to the position of the $32\times 32$ test images within the
$48\times 48$ frame (see illustration in Figure 2 for a resnet18_bn network).
The center cell of the grid acts as the reference performance and corresponds
to the no distribution shift, i.e., the test images are centered on the frame,
same as the train images. As we move away from the center, we analogously
translate the position of the object image in the test dataset. The model is
then evaluated for classification accuracy on the shifted test dataset. Thus,
the generalization or robustness of trained models to translation shifts can
be comprehensively summarized by such a grid.

Figure 2: Generalization to translation shifts of a resnet18_bn trained
without data augmentation (NoAug) for $1600$ epochs on CIFAR-10 dataset: Each
cell in the grid corresponds to the model performance on a test dataset with
specific positioning of the object image. The center cell corresponds to no
translation shift from training; and the distance of the cells from the center
corresponds to respective position shift between train and test dataset. The
values in each cell is the accuracy on the shifted test dataset. The color
corresponds to “relative" drop in performance from the in-distribution
performance: yellow the maximum accuracy in the grid (typically the center
cell), while the dark blue is saturated at $90\%$ of the max-value in the
grid, i.e., $10\%$ drop in accuracy. For resnet18_bn +NoAug on CIFAR-10, max
accuracy on the grid is $90.85\%$ while min accuracy is $81.82\%$.

###  3.1 Case study: Convolutional networks

We first describe some observations from our initial experimentation
exclusively on ConvNets trained on CIFAR-10. Our evaluations on other
architectures are provided in Section 4. In Figure 2 along with the
illustration of translation shift grid, we show the evaluation of resnet18_bn
(with batch normalization) on generalization to translation shifts. This model
was trained without any augmentation to demonstrate the baseline performance
of convolutional architecture. We observe that, _despite the built-in image
priors in ConvNets and despite using a weaker notion than translation
invariance, there is a drop in performance on even small translations. For
example, in the non-yellow cells close to center in Figure 2, we notice that a
two pixel hamming distance translation can lead to $>5\%$ drop in
performance._ It is worth forward referencing at this point that when compared
to other architectures ConvNets are relatively lot more resilient to
translation shifts in the absence of data augmentation (see Section 4)—the
worst case drop in performance is $\sim 10\%$ for ConvNets, while for other
architectures, it could be as large as $30-50\%$ (see Figure 4).

(a) resnet18_gn+NoAug: max = $91.48$, min = $87.97$.

(b) resnet18_gn+BA: max = $95.96$, min = $95.63$

Figure 3: Generalization to translation shifts of resnet variants using the
same evaluation as in Figure 2. (a) resnet18_gn network group norm and weight
standardization again trained without any augmentation. (b) resnet18_gn
trained with basic augmentation (BA) consisting of random horizontal flip and
random crop at most $4$ pixels. For quick reference, the sub-captions mention
the max and min accuracy of the models over the grid.

#### Batchnorm vs groupnorm+weight standardization.

Prior work most notably Azulay and Weiss (2019); Zhang (2019); Chaman and
Dokmanic (2021) attribute the lack of invariance to strides and ReLU non-
linearities in the standard networks. We believe these factors also affect the
weaker notion of generalization to translation shifts that we study. Our
experiments suggests that, in addition to strides and pooling, batch
normalization is yet another factor that might contribute to lack of
translation invariance. Figure 3(a) shows the performance of resnet18_gn model
trained with the same configuration as resnet18_bn in Figure 2, but with a
modification that all the batch normalization layers replaced by group
normalization and weight standardization Wu and He (2018); Qiao et al. (2019).
We see that this simple modification already improves the robustness of ResNet
to translation shifts. In hindsight, it is understandable that batch
normalization would have detrimental effects when the test distribution shifts
from the training distribution as the batch statistics obtained from
exponential moving average of training statistics no longer remains accurate
Wu and Johnson (2021).

#### Training with basic augmentation (BA).

Figure 3(b) illistrates our evaluation of resnet18_gn with basic augmentation
(BA) which includes random crops of up to $4$ pixels and horizontal flips.
This simple augmentation not only improve in-distribution accuracy by over
$5\%$, but also makes the models near-perfectly robust to up to $8$ pixel
translations in test distribution – indicating a form of meta-generalization
from augmentations. In the appendix we provide further evidence of such meta-
generalization: (a) In Appendix B, we look at even more minimal BA-lite and
BA-liter augmentations with smaller range of random crops of at most $2$ and
$1$ pixels, respectively. There we see that even $1$ pixel augmentation
improved robustness of ConvNets, while other architectures still benefit from
larger random crops. (b) In Appendix C our results on TinyImageNet dataset
show robustness to a larger range of translation shifts of up to $16$ pixel
shifts, even though BA still uses only $4$ pixel shifts.

##  4 Architectures and augmentations for generalization to translation shifts

The summary grid view of evaluations on translation shifts (as in Figure 2-3)
is more comprehensive and we will revisit them in the appendix. However, it is
not ideal for comparing different configurations of architectures and
augmentations. In this section, we use an alternative visualization and plot
the test accuracies as a function of Hamming distance between the position of
images in the test and training datasets. The performance of all our models
and augmentations are summarized in Figure 4.

|  CIFAR-10 |  CIFAR-100
---|---|---
|  |
AA |  |
BA |  |
| AA
---
(no-tr)
|
NoAug |  |
|  |
|  |
Figure 4: Generalization to translation shifts. The $x$-axis in each plot is
the Hamming distance of the location of test images (within the $48\times 48$
frame) relative to the location in the training images (center of the frame).
The larger the $x$-axis value, the larger is the shift from train
distribution. For a given value of $x$, there might be multiple test
configurations that are $x$-Hamming distance away (like shift of $1$ pixel to
top, bottom, right, or left when $x=1$). The median of these values is plotted
as line plot, while the shaded region covers the min and max values of the
list. The left and right plots correspond to accuracies on the CIFAR-10 and
the more challenging CIFAR-100 dataset, respectively. The $y$-axis for each
dataset (column) is normalized to be on the same scale.

###  4.1 Key takeaways

We make the following observations that are supported by our experiments. We
provide additional experiments and discussions from our study including
evaluation on TinyImageNet in the appendix.

  1. 1.

Without data augmentation, even ConvNets with designed architecture has a
noticeable drop in performance on spatially shifted test dataset. From Figures
2-3, one can see that merely switching batchnorm with groupnorm appears to
mitigate some performance gap. Using a specialized antialiased modification by
Zhang (2019) further improves the performance, albeit not to perfect
invariance as the non-linearities could be a source of breaking spacial
invariance Chaman and Dokmanic (2021). Nonetheless, the architectural
inductive biases in these ConvNets are still useful, if not perfect, as we can
predictably see that the drop in performance is much more dramatic for the
non-convolutional architectures without any image priors. In CaiT and ResMLP,
without augmentation, even $1-2$ pixel translation shift can lead to dramatic
drop in performance.

  2. 2.

In the other extreme, with an Advanced Augmentation (AA) pipeline, all the
architectures are remarkably robust even to large translation shifts in test
distribution. Note that even with AA, the maximum translation augmentation we
provide (in the form of random crop) is at most $4$ pixels ($8$ pixels in
Hamming distance), but we see robustness to up to $8$ pixel shifts ($16$
pixels in Hamming distance). This supports a notion of meta generalization in
robustness performance. In Appendix C, we see that ResNets continue to be
robust to even larger translation shifts of up to $16$ pixels on each
direction on TinyImageNet dataset. To further support the idea of meta-
generalization, we also show in Appendix B, even more minimal $1$ or $2$ pixel
random crop augmentation already boosts robustness to translation shifts.

  3. 3.

Furthermore, with AA, the performance on in-distribution test error becomes
significantly closer for all architectures. Specially, the performance of
resnet18 and cait_xxs36 are statistically identical in this setting even
though we trained on the small-medium scale CIFAR datasets. The performance of
resmlp_12 is however relatively suboptimal even with AA pipeline. First, there
is non-trivial gap in the in-distribution accuracies. Secondly, the robustness
to translation shifts is not nearly as good as with ResNet or CaiT. Despite
these differences, even for ResMLP, the augmentations dramatically boost the
generalization to translation shifts and the differences in relative drop
start to appear only after $10$-$12$ pixel hamming distance shifts in test
distribution. These experiments suggest that with sufficient augmentation, the
relative benefits or shortcomings of the architectures are effectively
diminished.

  4. 4.

Even with a minimal Basic Augmentation (BA), we see significant improvement in
robustness to translations. In fact, for ConvNets, BA is sufficient to achieve
the near perfect generalization on our canvas. This further highlights the
benefit of the built-in inductive biases in ConvNet. For non-convolutional
architectures, this simple augmentation is not sufficient to achieve optimal
absolute test performance, but the relative robustness is still uniformly
improved.

  5. 5.

Finally, an intriguing phenomenon is observed when training with advance
augmentation but without translation related augmentations (AA(no-tr)). Here
the absolute test accuracy of all the models improves (presumably from
learning some useful priors). For ConvNets on CIFAR-10, even such indirect
augmentation is effective in making the models robust to translation shifts –
but this does not appear to uniformly hold across datasets, so the conclusion
might be spurious. For transformer and MLP architectures, the robustness does
not improve significantly, even though the in-distribution accuracies are
significantly higher.

  6. 6.

Somewhat tangentially, our experiment slightly supports the position that data
augmentation can recover some of the benefits of large datasets even when
learning with general architectures like ViT and MLP are beneficial. In
ImageNet scale datasets this was previously observed in Touvron et al.
(2021c); Steiner et al. (2021) for ViTs and Touvron et al. (2021a) for MLPs.
Our experiments show similar validation even on small CIFAR-10 datasets.

In summary, even though convolutional networks are not invariant or robust to
translations in absolute sense, they clearly fare much better compared to
other general-purpose architectures. Specially, ConvNets learn robust models
with minimal augmentation, while it appears the transformer and MLP
architectures require more sophisticated augmentations. At the same time, our
experiments suggest that data augmentation can enforce learning of the right
inductive bias with comparable or more effectiveness than the network
architecture.

## References

  * Alsallakh et al. [2020] Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Jun Yuan, and Orion Reblitz-Richardson.  Mind the pad–cnns can develop blind spots.  _arXiv preprint arXiv:2010.02178_ , 2020.
  * Azulay and Weiss [2019] Aharon Azulay and Yair Weiss.  Why do deep convolutional networks generalize so poorly to small image transformations?  _Journal of Machine Learning Research_ , 20:1–25, 2019\.
  * Bai et al. [2021] Yutong Bai, Jieru Mei, Alan L Yuille, and Cihang Xie.  Are transformers more robust than cnns?  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Bhojanapalli et al. [2021] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit.  Understanding robustness of transformers for image classification.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10231–10241, October 2021.
  * Chaman and Dokmanic [2021] Anadi Chaman and Ivan Dokmanic.  Truly shift-invariant convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3773–3783, 2021.
  * Chen et al. [2021] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong.  When vision transformers outperform resnets without pretraining or strong data augmentations.  _arXiv preprint arXiv:2106.01548_ , 2021.
  * Cubuk et al. [2019] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le.  Autoaugment: Learning augmentation strategies from data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 113–123, 2019.
  * Djolonga et al. [2021] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al.  On robustness and transferability of convolutional neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16458–16468, 2021.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Engstrom et al. [2019] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry.  Exploring the landscape of spatial robustness.  In _International Conference on Machine Learning_ , pages 1802–1811. PMLR, 2019.
  * Fawzi and Frossard [2015] Alhussein Fawzi and Pascal Frossard.  Manitest: Are classifiers really invariant?  In _British Machine Vision Conference (BMVC)_ , 2015.
  * Fukushima and Miyake [1982] Kunihiko Fukushima and Sei Miyake.  Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition.  In _Competition and cooperation in neural nets_ , pages 267–285. Springer, 1982.
  * Goodfellow et al. [2009] Ian Goodfellow, Honglak Lee, Quoc Le, Andrew Saxe, and Andrew Ng.  Measuring invariances in deep networks.  _Advances in neural information processing systems_ , 22, 2009.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hendrycks and Dietterich [2018] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In _International Conference on Learning Representations_ , 2018.
  * Hoffer et al. [2020] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry.  Augment your batch: Improving generalization through instance repetition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8129–8138, 2020.
  * Inoue [2018] Hiroshi Inoue.  Data augmentation by pairing samples for images classification.  _arXiv preprint arXiv:1801.02929_ , 2018.
  * Kanbak et al. [2018] Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard.  Geometric robustness of deep networks: analysis and improvement.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4441–4449, 2018.
  * Kayhan and Gemert [2020] Osman Semih Kayhan and Jan C van Gemert.  On translation invariance in cnns: Convolutional layers can exploit absolute spatial location.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14274–14285, 2020.
  * Koh et al. [2021] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In _International Conference on Machine Learning_ , pages 5637–5664. PMLR, 2021.
  * Kolesnikov et al. [2020] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 491–507. Springer, 2020.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Le and Yang [2015] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7):3, 2015.
  * LeCun et al. [1989] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural computation_ , 1(4):541–551, 1989.
  * Liu [2017] Kuang Liu.  Train cifar10 with pytorch.  https://github.com/kuangliu/pytorch-cifar, 2017.
  * Mahmood et al. [2021] Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk.  On the robustness of vision transformers to adversarial examples.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7838–7847, 2021.
  * Paul and Chen [2021] Sayak Paul and Pin-Yu Chen.  Vision transformers are robust learners.  _arXiv preprint arXiv:2105.07581_ , 2(3), 2021.
  * Pinto et al. [2022] Francesco Pinto, Philip HS Torr, and Puneet K Dokania.  An impartial take to the cnn vs transformer robustness contest.  _arXiv preprint arXiv:2207.11347_ , 2022.
  * Qiao et al. [2019] Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille.  Weight standardization.  2019\.
  * Radosavovic et al. [2020] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár.  Designing network design spaces.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10428–10436, 2020.
  * Recht et al. [2019] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In _International Conference on Machine Learning_ , pages 5389–5400. PMLR, 2019.
  * Shao et al. [2021] Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh.  On the adversarial robustness of visual transformers.  _arXiv e-prints_ , pages arXiv–2103, 2021.
  * Steiner et al. [2021] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  _arXiv preprint arXiv:2106.10270_ , 2021.
  * Tolstikhin et al. [2021] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al.  Mlp-mixer: An all-mlp architecture for vision.  _arXiv preprint arXiv:2105.01601_ , 2021.
  * Touvron et al. [2021a] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al.  Resmlp: Feedforward networks for image classification with data-efficient training.  _arXiv preprint arXiv:2105.03404_ , 2021a.
  * Touvron et al. [2021b] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , volume 139, pages 10347–10357, July 2021b.
  * Touvron et al. [2021c] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with image transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 32–42, October 2021c.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wightman [2019] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * Wu and He [2018] Yuxin Wu and Kaiming He.  Group normalization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 3–19, 2018.
  * Wu and Johnson [2021] Yuxin Wu and Justin Johnson.  Rethinking" batch" in batchnorm.  _arXiv preprint arXiv:2105.07576_ , 2021.
  * Xiao et al. [2018] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song.  Spatially transformed adversarial examples.  In _International Conference on Learning Representations_ , 2018.
  * Yung et al. [2021] Jessica Yung, Rob Romijnders, Alexander Kolesnikov, Lucas Beyer, Josip Djolonga, Neil Houlsby, Sylvain Gelly, Mario Lucic, and Xiaohua Zhai.  Si-score: An image dataset for fine-grained analysis of robustness to object location, rotation and size.  _arXiv preprint arXiv:2104.04191_ , 2021.
  * Zeiler and Fergus [2014] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , pages 818–833. Springer, 2014.
  * Zhang [2019] Richard Zhang.  Making convolutional networks shift-invariant again.  In _International conference on machine learning_ , pages 7324–7334. PMLR, 2019.
  * Zhong et al. [2020] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang.  Random erasing data augmentation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 13001–13008, 2020.

##  Appendix A Additional experimental setup, training, and hyperparameter
details

CIFAR-10 and CIFAR-100 datasets used in the main paper are standard small-
medium scale benchmarks for image classification consisting of 50K training
images and 10K test images evenly distributed across $10$ and $100$ classes,
respectively. We recall from the main paper that the CIFAR images of size
$32\times 32\times 3$ are preprocessed using a mean padding canvas of $8$
pixels on each size (leading to $48\times 48\times 3$ images), and resizing to
standard ImageNet training size of $224\times 224\times 3$ using bilinear
interpolation (see Figure 1).

In addition to the experiments on CIFAR datasets, we also extend our study to
TinyImageNet [Le and Yang, 2015] dataset in the Section C. TinyImageNet is a
smaller subset of the standard ImageNet dataset consisting of 100K training
images and 10K validation images evenly distributed among 200 classes.
Further, the images are resized to $64\times 64\times 3$ from the standard
ImageNet size. This dataset curated for research exploration, with the goal of
having the complexity of ImageNet objects, while keeping the computation
manageable. However, since the size of TinyImageNet is an order of magnitude
smaller than ImageNet, the state-of-the-art in-distribution test accuracies on
this dataset are not as good as those on ImageNet. On the other hand, the
images have more diverse than CIFAR datasets. Thus, this dataset is a good
intermediate testbed where the computation is reasonable at the same time the
objects are more diverse are not focused tightly in the center as is usual in
CIFAR.

We pass the TinyImageNet dataset through a similar preprocessing pipeline as
CIFAR dataset, except now we use an even larger padding canvas of $16$ pixels
($1/4$ of image size) on each side leading to padded images of size $96\times
96\times 3$ – the larger padding maintains the ratio of size of translation
shifts we evaluate to $1/4$ of image dimension in each direction as in CIFAR
dataset. Once again to avoid extensive hyperparameter tuning on transformer
and MLP models, we resize the padded images to $224\times 224\times 3$ using
bilinear interpolation.

The optimization algorithms and hyperparameter used for training the models in
the main paper are summarized in Table 2. Additionally, we use the following
advanced augmentation parameters for models trained with AA and AA(no tr),
where all the implementations are from Wightman [2019]: For RandomAugment we
use $M=5,N=2$. Random erasing is used with probability $=0.25$. Mixup is used
in batch mode with probability $1$, where switch between standard mixup and
cutmix with probability $0.5$ each time; mixup $\alpha=0.8$ and mixup label
smoothing $=0.1$. We also use the repeated augmentation Hoffer et al. [2020]
when training with AA or AA(no tr) in distributed mode.

| resnet18(_bn,_gn) | cait_xxs36 | resmlp_12
---|---|---|---
opt. alg. | SGD+momentum | AdamW | LAMB
batchsize | $128\times 8$ | $64\times 8$ | $128\times 8$
starting lr | $0.1\times\frac{\text{batchsize}}{512}$ | $0.001\times\frac{\text{batchsize}}{512}$ | $0.005\times\frac{\text{batchsize}}{512}$
weight decay | $5e^{-4}$ | $0.05$ | $0.05$
drop path | none | 0.1 | 0.1
dropouy | 0 | 0 | 0
gradient clip | yes | no | no
Table 2: Summary of training algorithm and hyperparameters. All models are
trained distributed on $8\times$V100 GPUs and we use the automatic mixed
precision implementation through the torch.cuda.amp library in pytorch for
ResNet and CaiT training. We train all models for $1600$ epochs (except for
one experiment in Section B) and we use cosine annealing with
$T_{\text{max}}=\\#epochs$ and warmup for $20$ epochs to the linearly scaled
starting learning rate. When using gradient clipping, we clip to norm $2$ for
CIFAR datasets and $5$ for TinyImageNet.

##  Appendix B Additional experiments exploring meta generalization

In the main paper, we observed that even though all our augmentation pipelines
are restricted to at most $4$ pixel random crops, the learned models display
robustness to twice larger translation shifts of up to $8$ pixels. We termed
this phenomenon as meta generalization, where in small magnitude augmentations
of an invariance property leads to generalization to much larger shifts in the
test distribution. To investigate this phenomenon more, we introduce $4$
additional augmentation protocols that severely restrict the amount of direct
augmentation we get about translation invariance.

  1. 1.

BA-lite We use basic augmentation (BA) but with random crop of only up to $2$
pixels on each side along with random horizontal flip.

  2. 2.

BA-liter A more extreme BA where we restrict the random crop to only $1$ pixel
pad.

  3. 3.

AA-lite Here we keep the entire advance augmentation (AA) pipeline but
analogous to BA-lite, restrict the random crop to at most $2$ pixel padding.

  4. 4.

AA-liter This is the same as AA-lite but with at most $1$ pixel random crop
augmentation.

We summarize the results of the above mentioned minimal augmentation protocols
in Figure 5 and Figure 6, respectively, for the BA and AA pipelines.

|  CIFAR-10 |  CIFAR-100
---|---|---
|  |
BA |  |
| BA
---
lite
|
| BA
---
liter
|
NoAug |  |
|  |
|  |
Figure 5: Minimal augmentation pipelines: Basic Augmentation (BA) variants. The figure is plotted with the same protocol as the summary in Figure 4. BA-lite and BA-liter correspond to AA pipeline with at most $2$ pixel and $1$ pixel random crop respectively. |  CIFAR-10 |  CIFAR-100
---|---|---
|  |
AA |  |
| AA
---
lite
|
| AA
---
liter
|
NoAug |  |
|  |
|  |
Figure 6: Minimal augmentation pipelines: Advanced Augmentation (AA) variants.
The figure is plotted with the same protocol as the summary in Figure 4. AA-
lite and AA-liter correspond to AA pipeline with at most $2$ pixel and $1$
pixel random crop respectively.

In Figure 5 we see that ConvNets when trained with even an extremely minimal
augmentation of BA-liter (at most $1$ pixel random crop) show remarkably
performance gains. They not only achieve near-perfect generalization to
translation shifts of up to $8$ pixels, but also increase their absolute
accuracy and recover most of the performance gains from standard basic
augmentation (BA), wwhich uses up to $4$ pixel random crop.

Even for non-convolutional architectures, the gains from minimal basic
augmentations (BA-lite and BA-liter) compared to no augmentation are
substantial. More impressive gains are obtained from AA-lite and AA-liter
pipelines. However, unlike ConvNets the robustness is not perfect and these
architectures continue to benefit from more aggressive augmentation. We do
emphasize that there is still evidence of meta generalization in these non-
convolutional models: recall that even in the full advanced augmentation
pipeline (AA) has at most $4$ random crop. In the following section we further
see general purpose models, specially CaiT, trained with AA continues to be
robust even larger shifts of up to $16$ pixels on TinyImagenet dataset.

##  Appendix C Generalization to translation shifts: experiments on
TinyImageNet

In this section, we repeat the experiments in Section 4 on TinyImageNet
dataset. In comparison to CIFAR datasets, TinyImageNet more diverse images
where the objects are often not as centered. A priori, thus we expect less
sensitivity to small translations and shifts. Another difference from our
CIFAR evaluation is that in order to maintain the relative ratio of padding to
original image, we pad the $64\times 64$ images with a larger canvas of $16$
pixels rather than the $8$ pixels used in CIFAR datasets (both are padded with
$1/4$th of image size). This allows us to study the effects of even larger
translation shifts. Figure 7 shows results on TinyImageNet that are analogous
to Figure 4 on CIFAR-10 and CIFAR-100, except that we drop AA (no tr)
augmentation as it did not yield meaningful insights even in CIFAR-10.

AA |
---|---
BA |
NoAug |
Figure 7: Generalization to translation shifts results on TinyImageNet. The
plots are of the same format as Figure 4. The $x$-axis is the Hamming distance
between the position of test images (within the canvas of $16$ pixels on each
side) and corresponding position of the training images. For a given value of
$x$, there might be multiple test configurations that are $x$-Hamming distance
away, the median of these values is plotted as line plot, while the shaded
region covers the min and max values of the list.

Many of our observations on CIFAR also carry over to TinyImageNet. We
highlight some key similarities and differences below. In addition, we notice
that in comparison to CIFAR datasets, the drop in performance appears to have
a smaller slope w.r.t the pixel shifts in hamming distance. We attribute this
to the TinyImageNet dataset having more diversity in the positions of objects.

  1. 1.

As with CIFAR datasets, in the absence of data augmentation, we again see a
drop in performance to translation shifts in test distribution – smaller but
still significant on ConvNets and more drastic on non-convolutional
architectures. In this dataset, we also see that antialiased ResNet is
significantly more robust than standard ResNets.

  2. 2.

For ConvNets, we again observe that the basic augmentation suffices to make
the models robust to large translation shifts. Note that even on TinyImageNet
our augmentation pipelines have translation related augmentation of at most
$4$ pixels on each side ($8$ pixels in hamming distance). Thus, it is more
impressive that we see generalization to translation shifts of larger
magnitude than evaluated on CIFAR datasets, i.e., up to $16$ pixels on each
side or up to $32$ pixels in hamming distance. For non-convolutional models,
the behavior is similar to CIFAR datasets – there is improvement in robustness
to translations, but the absolute test accuracies are significantly lower.

  3. 3.

Advanced Augmentation (AA) again dramatically improves performance across all
architectures and make them robust to large translations shifts. For ConvNets
the performance continues to be robust even on this evaluation on a larger
canvas of up $32$ pixel Hamming distance shifts. However, for non-
convolutional architectures we start to see the drop in generalization
performance. Notice that even this case, the models are robust up to much
larger translations than present in the augmentation pipeline. In particular,
cait_xxs36 is robust up to $20$ pixel Hamming distance shifts and resmlp_12 up
to $14$ pixels, even though the maximum magnitude of translation shifts in the
augmentation pipeline is only $8$ pixels in Hamming distance.

In summary, our conclusions from experiments on CIFAR-10 and CIFAR-100
datasets are also supported by our results on TinyImageNet. Additionally, we
see the points at which the meta generalization to larger translation shifts
from translation augmentation of up to $4$ pixel random crop starts to break
for non-convolutional models.

##  Appendix D Limitations

We study to what extent data augmentation can capture the benefits of
carefully designed architectures in learning one of the fundamental priors of
image processing: invariance to translations. This was a natural candidate for
us as it is an instance where there is clearly a “good" architecture, and we
could focus on how augmentations might play a complementary role. To get a
broader picture about learning good image priors, such studies could also be
extended to other natural invariances, including other $2D$ invariances such
as scale and rotations, as well as $3D$ invariances such as occlusion and
lighting. Such studies would indeed be valuable, but due to combinatorial
increase in number of configurations we can test, we restrict our focus to one
key invariance.

We also restrict our study to only one representative architecture from each
family. In our initial experiments, we did not see significant difference in
qualitative trends when using other models within these architectures, but we
did not do a full study to make conclusive statements about how architectural
choices like width and depth affect the robustness to translation shifts.
Finally, from the experiments on ResMLP on CIFAR and further experiments on
TinyImageNet (Section C), we observe that we see that the prior learned from
augmentation are not perfect and begin to observe differences between ConvNets
and and other general purpose architectures under very large translation
shifts. This suggests there might be limitations to the extent of meta-
generalization from the $4$ pixel random crop holds for non-convolutional
models.