# Fast Lithium Ion Diffusion in Brownmillerite
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$

Xin Chen  School of Physical Science and Technology, Soochow University,
Suzhou 215006, China  Xixiang Zhang  Physical Science and Engineering
Division, King Abdullah University of Science and Technology (KAUST), Thuwal
23955-6900, Kingdom of Saudi Arabia  Jie-Xiang Yu<EMAIL_ADDRESS>School of
Physical Science and Technology, Soochow University, Suzhou 215006, China
Department of Physics and Astronomy, University of New Hampshire, Durham, New
Hampshire 03824, USA  Jiadong Zang  Department of Physics and Astronomy,
University of New Hampshire, Durham, New Hampshire 03824, USA

###### Abstract

Ionic conductors have great potential for interesting tunable physical
properties via ionic liquid gating and novel energy storage applications such
as all-solid-state lithium batteries. In particular, low migration barriers
and high hopping attempt frequency are the keys to achieve fast ion diffusion
in solids. Taking advantage of the oxygen-vacancy channel in
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$, we show that migration
barriers of lithium ion are as small as $0.28\sim 0.17\leavevmode\nobreak\
{\,\mathrm{eV}}$ depending on the lithium concentration rates. Our first-
principles calculation also investigated hopping attempt frequency and
concluded the room temperature ionic diffusivity and ion conductivity is high
as ${10}^{-7}\sim{10}^{-6}\leavevmode\nobreak\
\mathrm{{cm}^{2}\leavevmode\nobreak\ s^{-1}}$ and
${10}^{-3}\sim{10}^{-2}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$
respectively, which outperform most of perovskite-type, garnet-type and
sulfide Li-ion solid-state electrolytes. This work proves
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ as a promising solid-state
electrolyte.

##  I Introduction

Ionic diffusion in solids has played a key role in not only manipulating many
interesting physical properties in ion-electron-lattice-coupled systems via
ionic liquid gating (ILG)[1, 2, 3], but also great potential applications in
energy storage such as all-solid-state lithium batteries in which solid
electrolytes with both high Li-ion conductivity and low electron
conductivity[4, 5]. To achieve highly efficient ionic diffusion, high ion
conductivity or high ionic diffusivity is required. According to Nernst-
Einstein equation[6, 7], the relationship between ionic diffusivity $D$ and
ion conductivity $\sigma_{i}$ in solids can be given by

$\sigma_{i}=q^{2}nD\beta$ (1)

where $q$ is the electric charge of conducting ion, $n$ is the ionic carrier
concentration and $\beta=(k_{B}T)^{-1}$ is the the Boltzmann factor, the
inverse of the product of Boltzmann constant and temperature. Meanwhile, under
kinetically ideal conditions, the diffusivity can also be described by ion
hopping through a pathway with a hopping frequency via Vogele-Tammanne-Fulcher
model[8], which is given by

$D=a^{2}\nu^{*}\exp\left(-E_{a}\beta\right)$ (2)

where $a$ is the hopping distance between two neighbor stable sites, $\nu^{*}$
is the hopping attempt frequency and $E_{a}$ is the activation energy of ions
or the migration barriers during the diffusion. To this end, the high hopping
attempt frequency and the low migration barriers is crucial for fast ionic
diffusion.

Perovskite-type oxide systems are promising candidates for all-solid-state
lithium batteries due to their well ordered diffusion channels and the high
Young’s modulus[9]. Furthermore, the desirable combination of the complex
electron-lattice-spin coupling, the strongly correlated $d-$electrons and the
multivalent transition metal ions in transition metal oxides brings about
novel electronic and magnetic properties[10, 11, 12]. Similar to the
perovskite-type oxides, the recent study on brownmillerite
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ demonstrated that ILG could induce tri-
state phase transformation from $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ to
perovskite $\mathrm{SrCoO}_{3}$ and
$\mathrm{H}_{2}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ by the insertion of oxygen
anions and hydrogen cations respectively[2]. Han et al.[13] also reported that
the direction of the oxygen-vacancy channels can be well controlled. The well-
ordered[14] and controllable oxygen-vacancy channels in brownmillerite
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ provide favorable conditions for fast ionic
diffusion and storage of ions, bringing about applications in fuel cells and
rechargeable batteries.

In this paper, we systemically studied the diffusion of $\mathrm{Li}^{+}$ ion
in brownmillerite $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ based on first-principles
calculations. The injected $\mathrm{Li}^{+}$ cations can be stabilized inside
the oxygen-vacancy channels to form
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}\left(x=0.0\sim 1.0\right)$.
After confirming the chemical stability, we obtained the migration barrier of
$\mathrm{Li}^{+}$ cations along the oxygen-vacancy channels as $0.28\sim
0.17{\,\mathrm{eV}}$ $\mathrm{Li}^{+}$ depending on $\mathrm{Li}^{+}$
concentration rates. The low migration barrier is caused by the multi-bonding
property of Li-O bonds, and is lower than most of the Perovskite-type Li-ion
solid electrolytes. The corresponding diffusivity and conductivity at room
temperature are obtained as $10^{-7}\sim 10^{-6}\leavevmode\nobreak\
\mathrm{{cm}^{2}\leavevmode\nobreak\ s^{-1}}$ and $10^{-3}\sim
10^{-2}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$ respectively. Such high
ionic diffusivity and ion conductivity can be considered as a super-ionic
conductor[15].

##  II Methodology

We performed density-functional theory (DFT) based calculations with projector
augmented wave pseudopotentials[16, 17] implemented in the Vienna ab initio
simulation (VASP) package[18, 19]. The generalized gradient approximation in
Perdew, Burke, and Ernzerhof formation[20] was used as the exchange-
correlation energy and the Hubbard $U$ method[21] ($U=5.0{\,\mathrm{eV}}$,
$J=0.9{\,\mathrm{eV}}$) was applied on Co($3d$) orbitals to include strong-
correlation effects. An energy cutoff of 600 eV was used for the plane-wave
expansion throughout the calculations. The $k$-points were sampled on a
$10\times 10\times 4$ Monkhorst-Pack mesh in the Brillouin zone of the unit
cell of $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ containing eight Co, eight Sr and
twenty O atoms. For structural relaxations we relaxed the atoms until the
Hellmann-Feynman forces were less than $1{\,\mathrm{meV}}/$Å.

In order to find the diffusion path and the corresponding migration barriers
for Li diffusion in $\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$, we
performed climbing image nudged elastic band (cl-NEB) calculations[22, 23] to
find the saddle point and the minimum energy path between two stable local
minimum. Eight images were employed and the force convergence is down to
$0.02{\,\mathrm{eV}}/$Å. Phonon modes of the diffused $\mathrm{Li}^{+}$ ions
were obtained by using the finite displacement method[24] implemented in the
Phonopy package[25].

##  III Structural, Electronic and Magnetic Properties of
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$

Structure – The atomic structure of $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ is
shown in Fig.1. It has orthorhombic symmetry with space group $Pma2$ (28).
Compared to a $\sqrt{2}\times\sqrt{2}\times 4$ perovskite supercell of
$\mathrm{SrCoO}_{3}$, the unit cell of $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$
loses four O atoms with two $\mathrm{CoO}$ layers (b in Fig. 1(a)) appearing.
$\mathrm{CoO}$ layers shown in Fig. 1(b) have more spacing than
$\mathrm{CoO}_{2}$ layers Fig. 1(c). Periodic oxygen-vacancy channel
structures are formed along the $\left[\bar{1}10\right]$ direction so that
they can provide the well-ordered diffusion channels marked in Fig. 1(b). The
lattice constant for the orthorhombic unit cell with eight cobalt atoms,
namely $\left(\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}\right)_{4}$ is $a=5.57$ Å,
$b=5.46$ Å, and $c=16.00$ Å.

Figure 1: Crystal structure of $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ in one unit
cell. (a) The side view along $\left[\bar{1}10\right]$ direction. Blue, red
and green balls represent Co, O and Sr atoms respectively. The top view of the
$\mathrm{CoO}$ layer and the $\mathrm{CoO}_{2}$ layer in b and c are shown in
(b) and (c) respectively. Black arrows in (a) shows the G-AFM spin-ordering
and the yellow arrow in (b) indicates the vacancy channel along the
$\left[\bar{1}10\right]$ direction in the $\mathrm{CoO}$ layer.

Electronic and magnetic properties – The total energy results show that
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ has the ground state with G-type
antiferromagnetic (G-AF) or rocksalt-type antiferromagnetism (AFM) spin
ordering, consistent with previous studies[14]. G-AF is 68 meV per Co lower in
energy than C-type AFM (C-AF) or column-type AFM, 101 meV lower than A-type
AFM (A-AF) or layered-type AFM and 142 meV lower than ferromagnetic (FM)
ordering. Therefore, the electronic structure of
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ with G-AF ordering is investigated.
According to the density-of-state (DOS) result shown in Fig. 2, a band gap
about 0.6 eV indicates its insulating. In $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$,
Co cations have valence state $+3$ with six $d$ electrons ($d^{6}$) in
principle, so that the high-spin state of $\mathrm{Co^{3+}}$ is $S=2$ which
should have $4.0\leavevmode\nobreak\ \mu_{B}$ local magnetic moment with five
spin-majority and one spin-minority $d$ electrons. According to PDOS result in
Fig. 2(b)(c), The featured Co($3d$) in both $\mathrm{CoO}_{2}$ and
$\mathrm{CoO}$ layers are all below the Fermi level in the spin-up channel,
indicating Co’s are indeed in the high-spin state. However, the local moment
obtained by DFT is only $2.95\leavevmode\nobreak\ \mu_{B}$. According to the
onsite density matrices and the corresponding occupancy of both two types of
Co, six orbitals - five in spin-majority and one in spin-minority - are fully
occupied with occupancy closed to 1, while other four orbitals have occupancy
$0.12\sim 0.48$, indicating unoccupied $3d$ orbitals but hybridized with
oxygen ligands under the ligand field. Such high Co-O hybridization leads to
$\mathrm{Co^{2+}}\underline{L}$ state where the underline $\underline{L}$
refers to a ligand hole[26, 14]. To this end, $d^{6}$ configuration with $S=2$
high spin state on all $\mathrm{Co^{3+}}$ is still justified.

Figure 2: In the unit cell of
$\left(\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}\right)_{4}$ under G-AF ordering, (a)
the total density-of-state (DOS) and the projected DOS (PDOS) of Co($3d$) in
(b) $\mathrm{CoO}_{2}$ layers and in (c) $\mathrm{CoO}$ layers. Positive and
negative values represent the spin-majority (spin-up) and spin-minority (spin-
down) channels respectively. The Fermi level is set to zero. The local spin
magnetic moment on Co is also listed.

##  IV Lithium-injected $\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$

We now investigated the situations where Li atoms are injected into
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. With one Li placed in the unit cell, the
most stable structure of $\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$
is shown in Fig.3. One Li atom is located in the spacing of a $\mathrm{CoO}$
layer, at a hollow site in the center of the oxygen-vacancy channel and have
bonds with surrounding four O atoms. It is distinct from the protonated
$\mathrm{H}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ where each injected
$\mathrm{H}^{+}$ cation is located next to an oxygen atom with a strong
$\mathrm{O-H}$ bond so that $\mathrm{H}^{+}$ cations are not in the same plane
with $\mathrm{CoO}$ layers[2, 27]. Thus, it is obvious that at most four Li
atoms can be placed into a $\left(\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}\right)_{4}$
unit cell to form $\mathrm{Li}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$, where all
injected $\mathrm{Li}^{+}$ cations are located in the oxygen-vacancy channels,
shown in Fig.3. The lattice constant for
$\mathrm{Li}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ are $a=5.72$ Å, $b=5.50$ Åand
$c=16.75$ Å, only about $8\%$ expansion in volume. The chemical stability can
be confirmed by the formation energy of
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ from bulk
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and Lithium metal, which is defined by

$E_{f}=E_{\mathrm{Li}_{n}(\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}})_{4}}-nE_{\mathrm{Li}}-E_{(\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}})_{4}}$
(3)

where $n=4x$ is the number of Li placed in the
$\left(\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}\right)_{4}$ unit cell and the total
energy for face-centered cubic Li lattice is used for $E_{\mathrm{Li}}$.

The formation energy results for $n=1,3,4$ are $-2.01$, $-5.60$ and
$-7.98\leavevmode\nobreak\ {\,\mathrm{eV}}$ per unit cell respectively. Those
for all three non-equivalent $n=2$ situations are $-2.93$, $-3.20$ and
$-3.74\leavevmode\nobreak\ {\,\mathrm{eV}}$ respectively. They are all
negative, indicating the system reduce energy when
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ formed with the injected Li
atoms. The injected Li atoms can be stabilized in the oxygen-vacancy channels
in all the cases of various concentration rates.

Figure 3: Crystal structures of (a)
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ with one Li atoms (yellow
balls) per unit cell placed and (b)
$\mathrm{Li}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ with four Li atoms per unit cell
placed in side views and top views. The dashed lines in side views are the
$\mathrm{CoO}$ layers for top views.

To investigate how lithium’s injection affect the electronic structure and
magnetic ordering, we therefore focus on
$\mathrm{Li}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ with all four hollow sites in
the unit cell occupied by Li atoms to maximum the effect. Since each
$\mathrm{Li}^{+}$ cation has valence state $+1$, the average valence state of
Co is $+2.5$. In that case, half of Co are $+3$ with $d^{6}$ and half are $+2$
with $d^{7}$. G-AF spin ordering with zero net magnetization still has the
lowest total energy so that four Co ions in the unit cell has spin-up local
spin magnetic moment and other four are spin-down. The corresponding DOS and
PDOS on four spin-up Co ions labeled Co1, Co2, Co3 and Co4 are shown in Fig.4.
The total DOS gives a band gap about 0.8 eV indicating the insulating
properties. The local spin magnetic moments, Co1 and Co2 in $\mathrm{CoO}_{2}$
layers are $1.83\leavevmode\nobreak\ \mu_{B}$ and $2.73\leavevmode\nobreak\
\mu_{B}$ respectively, and those on Co3 and Co4 in $\mathrm{CoO}$ layers are
$1.83\leavevmode\nobreak\ \mu_{B}$ and $2.73\leavevmode\nobreak\ \mu_{B}$
respectively. According to PDOS results of Co, Co1(3$d$) in $\mathrm{CoO}_{2}$
layers (Fig. 4(b)) has unoccupied states above the Fermi level in the spin-up
channel, indicating that Co1 is not in the high spin state. On the other hand,
a sharp peak appears around $-4.5\mathrm{eV}$ below the Fermi level in the
spin-down channel, indicating that Co1 has more occupied state in the spin-
down channel than $\mathrm{Co^{3+}}$ in $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$.
Co2(3$d$) in $\mathrm{CoO}_{2}$ layers (Fig.4(c)) and Co3(3$d$) in
$\mathrm{CoO}$ layers (Fig.4(d)) has fully occupied state in the spin-up
channel while in the spin-down channel, the unoccupied state is located around
$3.0\sim 4.0\mathrm{eV}$ above the Fermi level, $1.0\leavevmode\nobreak\
{\,\mathrm{eV}}$ higher than the unoccupied state of $\mathrm{Co^{3+}}$ in
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. It indicates that $d$ electrons on Co2 and
Co3 in $\mathrm{Li}\mathrm{Sr}_{2}\mathrm{Co}_{2}\mathrm{O}_{5}$ meet stronger
on-site Coulomb interaction with more $d$ electrons than on $\mathrm{Co^{3+}}$
in $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. PDOS of Co4 in $\mathrm{CoO}$ layers
(Fig.4(e)) are similar to that of $\mathrm{Co}$ in
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$.

We still examined the onsite density matrices and the corresponding
occupancies of all the four types of Co ions. For Co1 in $\mathrm{CoO}_{2}$
layers, four orbitals in spin-majority and two in spin-minority are occupied
with occupancies close to 1, and others have occupancies $0.12\sim 0.48$,
indicating unoccupied $3d$ orbitals which only have hybridization bonding-
states with the surrounding oxygen ligands. Therefore, Co1 has +3 valence
state and $d^{6}$ configuration with $S=1$ intermediate spin state. For Co2 in
$\mathrm{CoO}_{2}$ layers and Co3 in $\mathrm{CoO}$ layers, five orbitals in
spin-majority and two in spin-minority are occupied with occupancies close to
1, so that they are both in +2 valence state and $d^{7}$ configuration with
$S=3/2$ high spin state. For Co4 in $\mathrm{CoO}$ layers, five orbitals in
spin-majority and one in spin-minority are occupied with occupancies close to
1, so that Co4 are in +3 valence state and $d^{6}$ configuration with $S=2$
high spin state. The insulating properties and the G-AF spin ordering ground
state are also confirmed in the electronic structures of
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ with $x=0.25,0.5$ and
$0.75$. Due to the multivalent cobalt cations, $\mathrm{Li}^{+}$-injection
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ does not affect the
insulating properties and the antiferromagnetic ordering.

Figure 4: In the unit cell of
$\left(\mathrm{Li}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}\right)_{4}$ in G-AF
ordering, (a) the total DOS and the PDOS of $3d$ orbitals of Co1, Co2, Co3 and
Co4 which is the four Co atoms with positive local spin magnetic moments in
the unit cell.

##  V Diffusion properties of Li in
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$

Figure 5: cl-NEB results of one Li diffusion in the unit cell of
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. The diagram is the total
energies of each state relative to the initial state as a function of hyper-
distances between each intermediate image and the initial state. a-d show the
top view of the intermediate images labeled in the diagram.

To investigate the diffusion of $\mathrm{Li^{+}}$ cations in
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$, we first calculated the
migration barriers of $\mathrm{Li^{+}}$ cations with various $\mathrm{Li}$
concentration $x$ in $\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. With
one Li atom in the unit cell,
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ mimics the situation
where very few $\mathrm{Li}^{+}$ cations are injected in the
$\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. We set two stable structural configuration
as the initial and ending states and other eight intermediate images on the
diffusion pathway are used to find the lowest migration barriers using cl-NEB
calculations. The results are shown in Fig.5. The migration barrier is
$0.28\leavevmode\nobreak\ {\,\mathrm{eV}}$ at the saddle point state with
highest potential energy (c in Fig.5). We also calculated the migration
barrier of a $\mathrm{Li}^{+}$ cation with three Li atoms in the unit cell,
that is $\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. This simulates
the opposite situation where $\mathrm{Li}^{+}$ cations occupy most of the
hollow sites in the oxygen-vacancy channels. The carriers is the ”holes” by
analogy with hole carriers in the p-doping electronic semiconductors. The
results are shown in Fig.6. The migration barrier is 0.17eV where the saddle
point state is c in Fig.6. The two cases cover the two most common situations
for ionic diffusion. According to the side-views of each image, in both two
cases, the diffusion pathways are all along the oxygen-vacancy channels. This
magnitude of migration barrier is similar with or even lower than that of
$\mathrm{H}^{+}$ diffusion in the protonated
$\mathrm{H}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ ($\sim
0.3\leavevmode\nobreak\ {\,\mathrm{eV}}$)[27]. It is also significantly lower
than most of the migration barriers of the perovskite-type Li-ion solid
electrolytes ($0.6\sim 0.2\leavevmode\nobreak\ {\,\mathrm{eV}}$)[9].

Figure 6: NEB results of one Li diffusion in the unit cell of
$\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. a-e show the top views
of the intermediate images labeled in the diagram.

To obtain the origin of the low migration barriers of
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$, we investigated the bonding
properties between Li and O during the diffusion. As mentioned, four Li-O
bonds are established for each $\mathrm{Li}^{+}$ at the initial state. During
the diffusion, two of the four bonds still remains and the other two bonds are
broken to form two new bonds near the ending state. Fig.7 shows the charge
differences between intermediate/ending states and the initial state for
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and
$\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ respectively. The change
of the spatial charge distribution illustrates the change of the chemical
bonds during the diffusion. For both cases, while the old bonds are being
broken, the new bonds have been created in the saddle point state (c-a in
Fig.7(a) and (b)). The newly formed bonds compensate the energy cost of broken
bonds, eventually reducing the migration barriers.

Figure 7: In the side views, the charge differences between initial state
(labeled a) and intermediate/ending states (labeled b-d for
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and b-e for
$\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$) for (a)
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and (b)
$\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. The labels corresponds
to the label in Fig.5 and Fig.6. Orange and cyan isosurfaces correspond to
positive and negative values respectively.

The ionic diffusivity/conductivity in a solid depends not only on the
migration barriers but also on the pre-exponential factor $a^{2}\nu^{*}$ in
Eq.2. According to Vineyard formula[28, 29], the hopping attempt frequency can
be given by

$\nu^{*}=\frac{\prod_{i=1}^{N}\nu_{i}^{0}}{\prod_{j=1}^{N-1}\nu_{j}^{s}}$ (4)

where $\nu_{i}^{0}$ and $\nu_{j}^{s}$ are the normal frequencies at the
$\Gamma$ point of vibration modes of the initial and saddle point state,
respectively. Notice that the number of normal modes in the product of saddle
point is one less than that in initial state. That is because of the presence
of an imaginary frequencies mode at the saddle point state caused by the
instability along the diffusion direction. This mode needs to be removed from
the product. The phonon modes at the $\Gamma$ point for the initial and the
saddle point state of both
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and
$\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ were therefore
calculated. As a result, the hopping attempt frequencies for
$\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and
$\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ are 6.87 THz and 0.95
THz, respectively. The lower migration barrier usually leads to a smoother
potential energy surface for diffusion, causing the lower force constant and
the corresponding vibration frequency. So that the hopping attempt frequency
for $\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ is lower than that
for $\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$. The hopping distance
is regarded as the half of the lattice constant along $\left[\bar{1}10\right]$
direction, which is $2.75\times 10^{-8}$cm for both two phases. To this end,
for $\mathrm{Li}_{0.25}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and
$\mathrm{Li}_{0.75}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$, we can estimate the pre-
exponential factor of diffusivity $a^{2}\nu^{*}$, which is $5.20\times
10^{-3}\leavevmode\nobreak\ \mathrm{{cm}^{2}\leavevmode\nobreak\ s^{-1}}$ and
$0.72\times 10^{-3}\leavevmode\nobreak\ \mathrm{{cm}^{2}\leavevmode\nobreak\
s^{-1}}$ respectively. The hopping attempt frequency can be regarded as the
frequency that the diffused ions attempt to climb over the barrier, so that
$\nu^{*}\exp\left(-E_{a}\beta\right)$ is nothing but the hopping frequency
from one stable site to its neighbor site with a successful hopping. At room
temperature with $T=300\leavevmode\nobreak\ {\,\mathrm{K}}$,
$\beta=k_{B}T=38.7\leavevmode\nobreak\ {\,\mathrm{eV}}^{-1}$, the ionic
diffusivity $D$ based on Vogele-Tammanne-Fulcher model (Eq.2) is $1.02\times
10^{-7}\leavevmode\nobreak\ \mathrm{{cm}^{2}\leavevmode\nobreak\ s^{-1}}$ and
$1.00\times 10^{-6}\leavevmode\nobreak\ \mathrm{{cm}^{2}\leavevmode\nobreak\
s^{-1}}$ respectively. If the ionic concentration $n$ is regarded as one
$\mathrm{Li}^{+}$ cation per unit cell, the corresponding ionic conductivity
$\sigma_{i}=q^{2}nD\beta$ based on Eq.1 are therefore $1.20\times
10^{-3}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$ and $1.17\times
10^{-2}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$ respectively. The room
temperature diffusivity and conductivity are both superior to most of the
perovskite-type Li-ion solid electrolytes ($10^{-4}\sim
10^{-3}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$ for bulk
conductivity)[9], garnet-type solid-state electrolytes ($10^{-4}\sim
10^{-3}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$)[30], and sulfide solid
electrolytes ($10^{-7}\sim 10^{-3}\leavevmode\nobreak\
\mathrm{S\cdot{cm}^{-1}}$)[31].

##  VI Conclusions

In summary, We investigated the structural, electronic and magnetic properties
of brownmillerite $\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ and
confirmed the oxygen vacancy channels, the insulating property and the G-AF
spin ordering ground state. At most four $\mathrm{Li}^{+}$ cations can be
stabilized in the center of the oxygen vacancy channels of an orthorhombic
$\mathrm{\left({Sr}_{2}{Co}_{2}{O}_{5}\right)_{4}}$ unit cell to form
$\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}\left(x=0.0\sim 1.0\right)$.
The G-AF spin ordering and insulating property is still remained with the
lithium’s injection. By employing cl-NEB calculations, we obtained the
migration barriers of $\mathrm{Li}_{x}\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ for
$x=0.25$ and $0.75$ as $0.28\leavevmode\nobreak\ {\,\mathrm{eV}}$ and
$0.17\leavevmode\nobreak\ {\,\mathrm{eV}}$, respectively. After obtaining the
hopping attempt frequencies via Vineyard formula, we therefore obtained the
corresponding diffusivity and conductivity at the room temperature as
$1.02\times 10^{-7}\leavevmode\nobreak\ \mathrm{{cm}^{2}\leavevmode\nobreak\
s^{-1}}$ and $1.20\times 10^{-3}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$
for $x=0.25$ respectively, and $1.00\times 10^{-6}\leavevmode\nobreak\
\mathrm{{cm}^{2}\leavevmode\nobreak\ s^{-1}}$ and $1.17\times
10^{-2}\leavevmode\nobreak\ \mathrm{S\cdot{cm}^{-1}}$ for $x=0.75$
respectively. The high ionic diffusivity and ion conductivity indicate
brownmillerite $\mathrm{{Sr}_{2}{Co}_{2}{O}_{5}}$ as a promising super-ionic
conductor.

###### Acknowledgements.

We are grateful for fruitful discussions with Prof. Pu Yu. This work was
financially supported by the National Natural Science Foundation of China
(12274309). The cl-NEB calculations were performed at Shaheen II in King
Abdullah University of Science and Technology (KAUST).

## References

  * Tan  _et al._ [2018] A. J. Tan, M. Huang, C. O. Avci, F. Büttner, M. Mann, W. Hu, C. Mazzoli, S. Wilkins, H. L. Tuller, and G. S. D. Beach, Nature Materials 18, 35 (2018).
  * Lu  _et al._ [2017] N. Lu, P. Zhang, Q. Zhang, R. Qiao, Q. He, H.-B. Li, Y. Wang, J. Guo, D. Zhang, Z. Duan, Z. Li, M. Wang, S. Yang, M. Yan, E. Arenholz, S. Zhou, W. Yang, L. Gu, C.-W. Nan, J. Wu, Y. Tokura, and P. Yu, Nature 546, 124 (2017).
  * Li  _et al._ [2020] Z. Li, S. Shen, Z. Tian, K. Hwangbo, M. Wang, Y. Wang, F. M. Bartram, L. He, Y. Lyu, Y. Dong, G. Wan, H. Li, N. Lu, J. Zang, H. Zhou, E. Arenholz, Q. He, L. Yang, W. Luo, and P. Yu, Nature Communications 11, 184 (2020).
  * Wang  _et al._ [2015] Y. Wang, B. Liu, Q. Li, S. Cartmell, S. Ferrara, Z. D. Deng, and J. Xiao, Journal of Power Sources 286, 330 (2015).
  * Aksyonov  _et al._ [2023] D. Aksyonov, A. Boev, S. Fedotov, and A. Abakumov, Solid State Ionics 393, 116170 (2023).
  * Kutner [1981] R. Kutner, Physics Letters A 81, 239 (1981).
  * Park  _et al._ [2010] M. Park, X. Zhang, M. Chung, G. B. Less, and A. M. Sastry, Journal of Power Sources 195, 7904 (2010).
  * Ikeda and Aniya [2013] M. Ikeda and M. Aniya, Journal of Non-Crystalline Solids 371-372, 53 (2013).
  * Lu and Li [2021] J. Lu and Y. Li, Journal of Materials Science: Materials in Electronics 32, 9736 (2021).
  * Imada  _et al._ [1998] M. Imada, A. Fujimori, and Y. Tokura, Reviews of Modern Physics 70, 1039 (1998).
  * Dagotto [2005] E. Dagotto, Science 309, 257 (2005).
  * Ngai  _et al._ [2014] J. Ngai, F. Walker, and C. Ahn, Annual Review of Materials Research 44, 1 (2014).
  * Han  _et al._ [2022] H. Han, A. Sharma, H. L. Meyerheim, J. Yoon, H. Deniz, K.-R. Jeon, A. K. Sharma, K. Mohseni, C. Guillemard, M. Valvidares, P. Gargiani, and S. S. P. Parkin, ACS Nano 16, 6206 (2022).
  * Muñoz  _et al._ [2008] A. Muñoz, C. de la Calle, J. A. Alonso, P. M. Botta, V. Pardo, D. Baldomir, and J. Rivas, Physical Review B 78, 054404 (2008).
  * He  _et al._ [2017] X. He, Y. Zhu, and Y. Mo, Nature Communications 8, 15893 (2017).
  * Blöchl [1994] P. E. Blöchl, Physical Review B 50, 17953 (1994).
  * Kresse and Joubert [1999] G. Kresse and D. Joubert, Physical Review B 59, 1758 (1999).
  * Kresse and Furthmüller [1996a] G. Kresse and J. Furthmüller, Computation Materials Science 6, 15 (1996a).
  * Kresse and Furthmüller [1996b] G. Kresse and J. Furthmüller, Physical Review B 54, 11169 (1996b).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Physical Review Letters 77, 3865 (1996).
  * Liechtenstein  _et al._ [1995] A. I. Liechtenstein, V. I. Anisimov, and J. Zaanen, Physical Review B 52, R5467 (1995).
  * Henkelman  _et al._ [2000] G. Henkelman, B. P. Uberuaga, and H. Jónsson, The Journal of Chemical Physics 113, 9901 (2000).
  * Henkelman and Jónsson [2000] G. Henkelman and H. Jónsson, The Journal of Chemical Physics 113, 9978 (2000).
  * Parlinski  _et al._ [1997] K. Parlinski, Z. Q. Li, and Y. Kawazoe, Physical Review Letters 78, 4063 (1997).
  * Togo and Tanaka [2015] A. Togo and I. Tanaka, Scripta Materialia 108, 1 (2015).
  * Korotin  _et al._ [1996] M. A. Korotin, S. Y. Ezhov, I. V. Solovyev, V. I. Anisimov, D. I. Khomskii, and G. A. Sawatzky, Physical Review B 54, 5309 (1996).
  * Lu  _et al._ [2018] N. Lu, Y. Wang, S. Qiao, H.-B. Li, Q. He, Z. Li, M. Wang, J. Zhang, S. C. Tsang, J. Guo, S. Yang, J. Zhang, K. Deng, D. Zhang, J. Ma, Y. Wu, J. Zhu, Y. Tokura, C.-W. Nan, J. Wu, and P. Yu, A protonated brownmillerite electrolyte for superior low-temperature proton conductivity (2018), arXiv:1811.10802 [cond-mat.mtrl-sci] .
  * Vineyard [1957] G. H. Vineyard, Journal of Physics and Chemistry of Solids 3, 121 (1957).
  * Koettgen  _et al._ [2017] J. Koettgen, T. Zacherle, S. Grieshammer, and M. Martin, Physical Chemistry Chemical Physics 19, 9957 (2017).
  * Wang  _et al._ [2020] C. Wang, K. Fu, S. P. Kammampata, D. W. McOwen, A. J. Samson, L. Zhang, G. T. Hitz, A. M. Nolan, E. D. Wachsman, Y. Mo, V. Thangadurai, and L. Hu, Chemical Reviews 120, 4257 (2020).
  * Manthiram  _et al._ [2017] A. Manthiram, X. Yu, and S. Wang, Nature Reviews Materials 2, 16103 (2017).


# Attribute Diversity Determines the Systematicity Gap in VQA

Ian Berlot-Attwell
University of Toronto
Vector Institute
<EMAIL_ADDRESS>A. Michael Carrell
University of Cambridge
<EMAIL_ADDRESS>
Kumar Krishna Agrawal
Department of EECS
UC Berkeley
<EMAIL_ADDRESS>
Yash Sharma
University of Tübingen
<EMAIL_ADDRESS>
Naomi Saphra11footnotemark: 1
The Kempner Institute at Harvard University
<EMAIL_ADDRESS>Joint senior authors to whom correspondence should be
addressed

###### Abstract

The degree to which neural networks can generalize to new combinations of
familiar concepts, and the conditions under which they are able to do so, has
long been an open question. In this work, we study the systematicity gap in
visual question answering: the performance difference between reasoning on
previously seen and unseen combinations of object attributes. To test, we
introduce a novel diagnostic dataset, CLEVR-HOPE. We find that while increased
quantity of training data does not reduce the systematicity gap, increased
training data diversity of the attributes in the unseen combination does. In
all, our experiments suggest that the more distinct attribute type
combinations are seen during training, the more systematic we can expect the
resulting model to be.

Attribute Diversity Determines the Systematicity Gap in VQA

Ian Berlot-Attwell University of Toronto Vector Institute
<EMAIL_ADDRESS>A. Michael Carrell University of Cambridge
<EMAIL_ADDRESS>Kumar Krishna Agrawal Department of EECS UC Berkeley
<EMAIL_ADDRESS>

Yash Sharma††thanks:  Joint senior authors University of Tübingen
<EMAIL_ADDRESS>Naomi Saphra11footnotemark: 1 ††thanks:  to whom
correspondence should be addressed The Kempner Institute at Harvard University
<EMAIL_ADDRESS>

##  1 Introduction

Systematicity, the ability to handle novel combinations of known concepts, is
a type of compositional generalization Hupkes et al. (2020). While
systematicity is crucial to human intelligence Fodor and Pylyshyn (1988),
conventionally trained neural networks often struggle to generalize
systematically (Csordás et al., 2021, 2022a, 2022b).

Inspired by prior work investigating compositionality failures in language
models Press et al. (2022), we study the systematicity gap in visual question
answering (VQA): the drop in model performance when reasoning about a
combination of properties that was held out from both the text and vision
modalities at train time. As an example, let us consider material and shape as
two _attribute types_. If a model was trained without exposure to a particular
combination of attribute values, e.g., rubber sphere, then we say the model
composes systematically if it has high performance at test time on data that
includes a rubber sphere.

Our work empirically demonstrates that systematicity emerges in a neural VQA
model if the model is trained with diverse contexts for the attribute values
in question (i.e., exposed to many material-shape combinations). The intuition
for this hypothesis is simple: given many training examples of distinct
combinations, the model learns how material and shape interact, and thus
systematically generalizes to an unseen combination of material and shape. In
contrast, a model trained on low-diversity data (i.e., only exposed to a few
material-shape combinations) fails to learn rules to recombine them.

Using CLEVR-HOPE, a novel dataset for evaluating systematicity on a variety of
held-out object attribute value pairs in a controlled setting, we measure the
systematic compositionality of multi-modal transformer and neurosymbolic
models. We find that, while systematicity does not improve with more training
data, it does improve with more diverse training data. Specifically, attribute
types that include more diverse combinations during training can be composed
systematically.

##  2 CLEVR-HOPE Diagnostic Dataset

Figure 1: Example image-question pairs for the sub-dataset of CLEVR-HOPE
corresponding to rubber cylinder.The test sets are in gray; rubber cylinder is
omitted visually and textually in the train split and the IID test splits;
rubber cylinder only occurs in the OOD splits; occurrences are emphasized in
this figure. The train and complex sets are of comparable visual and textual
complexity to CLEVR. The minimal sets consist only of existence questions,
checking whether a single object matches a given pair of attribute values.

Our dataset is based on CLEVR Johnson et al. (2017a), a synthetic experimental
setting for testing basic visual reasoning skills. CLEVR comprises English
questions (such as “What is the color of the cube on the right side of the
yellow sphere?") and corresponding 3D-rendered images of colored blocks. Each
block has four attribute types (size, color, material, and shape).

We present the CLEVR Held-Out Pair Evaluation (CLEVR-HOPE) dataset for testing
the systematicity of VQA models. CLEVR-HOPE is a controlled setting to test
whether VQA models generalize to pairs of attribute values that were not seen
during either training or fine-tuning. Within CLEVR-HOPE, we refer to an
unseen pair of attribute values as a Held-Out Pair (HOP). The dataset is
composed of 29 sub-datasets, each for a different HOP (Appx. Tab. 2) . Each
HOP has its own train set and 4 test sets. For rubber cylinder, visualized in
Fig. 1, these datasets are:

train: 560k image-question pairs in the training/finetuning set. The data
distribution is similar to CLEVR, but any images or questions involving rubber
cylinder have been removed.

complex-IID test: Test data sampled from the train distribution (i.e., rubber
cylinder is filtered out).

complex-OOD test: Test data sampled from the CLEVR distribution filtered to
always have (i) at least one object matching rubber cylinder, and (ii) rubber
cylinder in the question.

minimal-IID test: Minimal image-question pairs that check whether a model can
recognize pairs of attribute values, corresponding to rubber cylinder’s
attribute types, that were seen in the train set. E.g., as rubber cylinder is
a material-shape combination, the minimal-IID test set checks combinations
including rubber spheres, metal cylinders, and metal cubes, but not
combinations of different types (e.g., yellow cylinders, or small yellow
objects).

minimal-OOD test: Minimal image-question pairs that check recognition of
rubber cylinder. Always returning false would yield 75% accuracy.

Appendix A shows complete details. Note that CLEVR-HOPE omits validation sets
to prevent hyperparameter tuning for this specific task Teney et al. (2020).
Instead, hyperparameters should be chosen using CLEVR.

##  3 Models & Training

Models: Our analysis focuses on LXMERT Tan and Bansal (2019), a multi-modal
transformer-based Vaswani et al. (2017) architecture. We study two LXMERT
variants, finetuning each on varying datasets and measuring the systematicity
gap. We further run a subset of our experiments on a neurosymbolic model,
Tensor-NMN Johnson et al. (2017b), a neural module network Andreas et al.
(2016) that decomposes the task into a composition of subtask-specific
modules.

Training: For each HOP, we subsample the training set to test the impact the
amount of training data has on performance. For 3 random seeds per HOP, we
finetune pretrained LXMERT (LXMERT-p) and train LXMERT from scratch
(LXMERT-s). We also train Tensor-NMN from scratch, again for three runs,
though only for the first 6 HOPs, combinations of {large, cyan, rubber,
cylinder}.

For hyperparameter selection, we perform a grid search on the original CLEVR
dataset Johnson et al. (2017a). For further details, see Appendix B.

##  4 Results

###  4.1 Evidence of Systematic Behaviour

With sufficient training data, over 93% of the tested model-HOP combinations
exceed 75% accuracy on the minimal-OOD test set, some reaching 100% (see Appx.
Fig. 5). The VQA models have a wide range of accuracies generalizing to
different held out pairs. On all models tested, this accuracy varies by around
25% across different HOPs.

Performance on the complex-OOD test set is also generally increasing with the
amount of training data, and we see that the OOD accuracies across HOPs are
similarly distributed (see Appx. Fig. 7). In all, we can conclude that the
models consistently exhibit at least some degree of systematic behaviour. The
same trends are observed for Tensor-NMN (see Appx. Figs. 10 and 12).

###  4.2 Systematicity Gap

Knowing that our models can exhibit systematic behaviour, a natural question
to ask is whether there is any trend in the difference between in- and out-of-
distribution performance: i.e., as the size of the training set increases (and
thus the model’s performance generally improves), does its performance on
held-out combinations approach its performance on the combinations already
seen at train time? We call this performance difference, between the OOD and
IID combinations, the systematicity gap.

For example, if a model has an IID accuracy of 95%, but only 80% for data that
requires the model to systematically compose rubber and cylinder into the held
out pair rubber cylinder, then the systematicity gap is -15% (i.e., a 15%
drop).

Given that the models are somewhat systematic, and that performance in general
improves with more training data, one might expect that the systematicity gap
would trend to zero. To the contrary, we find that, averaging over all HOPs,
the LXMERT systematicity gap plateaus to a drop of 5-6% (see Appx. Fig. 15).
On the minimal test sets, the systematicity gap again plateaus, to a drop of
6-8% (see Appx. Fig. 16). The same trends are observed in Tensor-NMN (see
Appx. Figs. 17 and 18), though the systematicity gap on minimal examples
widens with additional training data.

With that said, the standard deviation of the observed systematicity gap is
quite high – in the following section we make the case that the nature of the
training data, specifically the attribute diversity seen at train time, is
responsible.

###  4.3 Train-time conceptual diversity impacts systematicity

We define attribute diversity as the number of possible attribute values
corresponding to the unseen combination’s attribute types. For example, if the
unseen combination is rubber cylinders, that corresponds to the material and
shape attribute types. Given there are 2 possible materials and 3 possible
shapes in the training set, there are $2\times 3=6$ possible material-shape
combinations; thus the attribute diversity is 6.

HOP | Attribute Types | Diversity
---|---|---
Large rubber |  Size \+ Material | 4
Rubber cylinder |  Material \+ Shape | 6
Large cylinder |  Size \+ Shape | 6
Rubber cyan |  Material \+ Color | 16
Large cyan |  Size \+ Color | 16
Cyan cylinder |  Color \+ Shape | 24
Table 1: Diversity of the first six held-out pairs (HOPs). Diversity is the
number of possible attribute values corresponding to the HOP’s attribute
types.

Tab. 1 lists the attribute diversity of the first six HOPs in CLEVR-HOPE (see
Appx. Tab. 2 for all 29 HOPs). Since the CLEVR training distribution is
uniform across object attribute values, for a train set of fixed size, as
attribute diversity increases, the number of examples per combination
decreases.

Fig. 2(a) again illustrates the systematicity gap, but now only averages over
HOPs of the same diversity (rather than over all HOPs as in Sec. 4.2). With
this, we see that the systematicity gap is stratified by the diversity of the
combinations seen at train time. Specifically, as the diversity of the
training data increases, the systematicity gap narrows. In fact, the gap is
typically near or within a standard deviation of zero for diversities of 16 or
above. In comparison, diversities of 6 show a a plateauing systematicity gap
stabilizing at 7-14%. As seen in Fig. 2(b), we observe similar results with
the systematicity gap of the minimal test sets.

For Tensor-NMN, we also find stratification by diversity for complex examples
(see Appx. Fig. 19). The trend on minimal examples is noisier, but converges
to the expected ordering (see Appx. Fig. 20).

(a) Complex

(b) Minimal

Figure 2: Systematicity gap (difference between OOD and IID model accuracy),
averaged by held-out pair (HOP) diversity over 29 HOPs, each with 3 runs.

##  5 Related work

Systematicity has often been investigated through synthetic datasets. Lake and
Baroni (2018) introduced the SCAN benchmark to evaluate the compositionality
of sequence-to-sequence models, revealing a lack of systematicity. Followup
works Patel et al. (2022); Jiang et al. (2022) have shown that the conceptual
diversity of the training set has a significant effect on systematicity — our
work extends these findings to the multi-modal domain of VQA.

While compositionality in VQA has been studied, prior work has focused on
generalization to new question structures Bahdanau et al. (2019); Vani et al.
(2021); Bogin et al. (2021), or question-answer combinations Agrawal et al.
(2017), rather than new attribute combinations. One reason for this gap is
that, with natural data, it is hard to control for the model’s exposure to
particular attribute combinations. By using a controlled synthetic setting, we
can guarantee that generalization behavior is systematic based on the data
split.

The closest prior work is the CLEVR-CoGenT dataset: Johnson et al. (2017a)
created a train-test CLEVR split where at train time cubes and cylinders are
restricted to limited color palettes, that are reversed at test time. They
observed that model performance declined on held-out attribute combinations.
But, unlike CLEVR-HOPE, CLEVR-CoGenT does not change the question distribution
at train time — held-out combinations can leak by appearing in text at train
time. Furthermore, CLEVR-CoGenT has only a single train set with held-out
color-shape combinations — whereas CLEVR-HOPE expands the set of held-out
combinations to 29 train sets, covering all possible pairs of attribute types.
CLEVR-HOPE also independently assesses each HOP, including in a minimal
setting. In combination, these improvements allow us to study the impact of
train-time diversity.

Beyond CLEVR-CoGenT, our results align with concurrent work on the effects of
training diversity in VQA: Rahimi et al. (2023) modify CLEVR to study the
related question of productivity. Specifically, generalization to questions
with more reasoning steps, and generalization to new question combinations
(e.g., answering counting questions about shape, when all train-time counting
questions are about color or size). They conclude that increasing the
diversity of question combinations increases productivity. Unlike our work,
they do not use a transformer architecture, instead studying MAC Hudson and
Manning (2018), FiLM Perez et al. (2018), and Vector-NMN Bahdanau et al.
(2019). Additionally, as they study a fundamentally different question, their
dataset only alters the question distribution — their image distribution is
unchanged between train and test time.

Given that both systematicity and productivity fall under the larger umbrella
of compositional generalization Hupkes et al. (2020), and given the wide
variety of architectures collectively studied between both our and existing
work, the combined weight of the evidence suggests a close relationship
between train-time diversity and compositional generalization as a broad
phenomenon.

##  6 Conclusions

Using CLEVR-HOPE, we demonstrate that LXMERT and Tensor-NMN exhibit some
degree of systematic generalization to held-out object attribute pairs.
Furthermore, we illustrate that the systematicity gap (the difference between
in- and out-of-distribution performance) does not improve with more data, but
does with more attribute diverse data— i.e., the number of attribute pairs of
the same type seen at train time.

Figure 3: Systematicity gap on the complex splits (top corner) and minimal
splits (bottom corner) for all models trained on 560k training examples. The
systematicity gap is averaged according to the attribute types of the HOPs,
all 29 HOPs for LXMERT, HOPs 0-5 for Tensor-NMN — attributes are sorted by
increasing diversity on the axes (e.g., shape has 2 possible values, color has
8 possible values). As expected, we see a worse systematicity gap (i.e.
lighter colors) in the top left (low-diversity combinations), and better
systematicity gap (i.e., darker colors) in the bottom right (high-diversity
combinations).

## Limitations

First and foremost, while the synthetic nature of CLEVR-HOPE allows for a more
controlled study of models, it raises the question whether the observed
results will hold in more complex and diverse real-world settings.
Furthermore, as we did not modify the CLEVR attribute values, HOP diversity is
intrinsically tied to attribute type. e.g., the most diverse pairs are always
shape-color combinations, and the least diverse pairs are always material-size
combinations. Thus, it is possible that we are actually measuring the effects
of attribute type on generalization, rather than diversity.

However, in visualizing the systematicity gap by attribute-types in the pair
on both LXMERT and Tensor-NMN (see Fig. 3), we can see that while the
systematicity gap tends to be larger for material than size (with these
attribute types both having two possible values, i.e. the same diversity), it
still holds that the systematicity gaps are still sorted by the diversity of
the attribute pairs. In addition, with respect to raw accuracy, we find that
LXMERT tends to struggle when Shape-Material pairs (diversity 6) are held out
— more so than with the lower diversity Material-Size pairs (diversity 4) (see
Appx. Fig. 9). Yet despite this, the systematicity gaps remain sorted as
expected — i.e., worse on the lower diversity Material-Size pairs (see Fig.
3). Finally, our results are supported by prior work in other domains Patel et
al. (2022); despite the number of examples per combination decreasing with
increasing diversity, the systematicity gap still tends to improve.

The second major limitation arises from the choice of models. LXMERT uses a
pretrained F-RCNN Ren et al. (2015) for object detection, which we do not
alter. As the F-RCNN is pretrained, it may already possess implicit knowledge
of the attributes (e.g., shape), and may contribute systematic structure to
LXMERT. Any such visual knowledge or biases are therefore given to both
LXMERT-p and LXMERT-s. In contrast, note that the language component of
LXMERT-s is randomly initialized — whereas Tan and Bansal (2019) initialized
their language transformer with BERT Devlin et al. (2019) when pretraining
from scratch.

Finally, due to time and resource limitations we were unable to evaluate
Tensor-NMN on HOP-6 through 28.

## Acknowledgements

Resources used in preparing this research were provided, in part, by the
Department of Computer Science at the University of Toronto, the Province of
Ontario, the Government of Canada through CIFAR, companies sponsoring the
Vector Institute (www.vectorinstitute.ai/partnerships/current-partners/), the
Hyundai Motor Company (under the project Uncertainty in Neural Sequence
Modeling), the Samsung Advanced Institute of Technology (under the project
Next Generation Deep Learning: From Pattern Recognition to AI), and by a gift
from the Chan Zuckerberg Initiative Foundation to establish the Kempner
Institute for the Study of Natural and Artificial Intelligence.

Ian Berlot-Attwell is funded by a Natural Sciences and Engineering Research
Council of Canada Postgraduate Scholarship-Doctoral, and a Vector Institute
Research Grant. A. Michael Carrell is funded in part by a Microsoft Research
scholarship. The authors thank the International Max Planck Research School
for Intelligent Systems (IMPRS-IS) for supporting Yash Sharma.

We appreciate the invaluable ideas and discussion from Preetum Nakkiran,
Spencer Frei, Nicholas Schiefer, and Ferenc Huszár who helped shape the early
stages of this work, and Preetum in particular for enabling this
collaboration. We would also like to thank Jonathan Shi for his early
contributions to our code base, including help with, and modification to, the
CLEVR dataset generation code. We would also like to thank Frank Rudzicz for
being a wonderful PhD supervisor, and for his suggestions and advice
throughout the work.

## References

  * Agrawal et al. (2017) Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. 2017.  C-VQA: A compositional split of the visual question answering (VQA) v1.0 dataset.  _CoRR_ , abs/1704.08243.
  * Andreas et al. (2016) Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016.  Neural module networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 39–48.
  * Bahdanau et al. (2019) Dzmitry Bahdanau, Harm de Vries, Timothy J O’Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. 2019.  Closure: Assessing systematic generalization of clevr models.  _arXiv preprint arXiv:1912.05783_.
  * Bogin et al. (2021) Ben Bogin, Shivanshu Gupta, Matt Gardner, and Jonathan Berant. 2021.  COVR: A test-bed for visually grounded compositional generalization with real images.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9824–9846, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Csordás et al. (2021) Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. 2021.  Learning adaptive control flow in transformers for improved systematic generalization.  In _Advances in Programming Languages and Neurosymbolic Systems Workshop_.
  * Csordás et al. (2022a) Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. 2022a.  The devil is in the detail: Simple tricks improve systematic generalization of transformers.
  * Csordás et al. (2022b) Róbert Csordás, Kazuki Irie, and Jürgen Schmidhuber. 2022b.  The neural data router: Adaptive control flow in transformers improves systematic generalization.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fodor and Pylyshyn (1988) Jerry A Fodor and Zenon W Pylyshyn. 1988.  Connectionism and cognitive architecture: A critical analysis.  _Cognition_ , 28(1-2):3–71.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Hudson and Manning (2018) Drew A. Hudson and Christopher D. Manning. 2018.  Compositional attention networks for machine reasoning.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 6700–6709.
  * Hupkes et al. (2020) Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020.  Compositionality decomposed: How do neural networks generalise? _J. Artif. Intell. Res._ , 67:757–795.
  * Jiang et al. (2022) Yichen Jiang, Xiang Zhou, and Mohit Bansal. 2022.  Mutual exclusivity training and primitive augmentation to induce compositionality.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11778–11793, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Johnson et al. (2017a) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017a.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In _CVPR_.
  * Johnson et al. (2017b) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017b.  Inferring and executing programs for visual reasoning.  In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , pages 3008–3017. IEEE Computer Society.
  * Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _International conference on machine learning_ , pages 2873–2882. PMLR.
  * Patel et al. (2022) Arkil Patel, Satwik Bhattamishra, Phil Blunsom, and Navin Goyal. 2022.  Revisiting the compositional generalization abilities of neural sequence models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 424–434, Dublin, Ireland. Association for Computational Linguistics.
  * Perez et al. (2018) Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. 2018.  Film: Visual reasoning with a general conditioning layer.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 3942–3951. AAAI Press.
  * Press et al. (2022) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022.  Measuring and narrowing the compositionality gap in language models.  _arXiv preprint arXiv:2210.03350_.
  * Rahimi et al. (2023) Amir Rahimi, Vanessa D’Amario, Moyuru Yamada, Kentaro Takemoto, Tomotake Sasaki, and Xavier Boix. 2023.  D3: Data diversity design for systematic generalization in visual question answering.  _arXiv preprint arXiv:2309.08798_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In _Advances in Neural Information Processing Systems_ , volume 28. Curran Associates, Inc.
  * Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019.  A corpus for reasoning about natural language grounded in photographs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6418–6428, Florence, Italy. Association for Computational Linguistics.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_.
  * Teney et al. (2020) Damien Teney, Ehsan Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton Van Den Hengel. 2020.  On the value of out-of-distribution testing: An example of goodhart’s law.  _Advances in neural information processing systems_ , 33:407–417.
  * Vani et al. (2021) Ankit Vani, Max Schwarzer, Yuchen Lu, Eeshan Dhekane, and Aaron Courville. 2021.  Iterated learning for emergent systematicity in vqa.  In _International Conference on Learning Representations_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.

##  Appendix A CLEVR-HOPE: Additional details

The full list of held-out pairs (HOPs) can be found in Table 2. The HOPs were
selected by choosing two attribute values from each of large cyan rubber
cylinder, small brown rubber sphere, small red metal cylinder, large gray
metal cube, and small purple rubber sphere.

Note that there are only 4 possible material-size combinations, as there are
only 2 sizes and 2 materials. We include all 4 of these, as well as 5 HOPs for
every other pair of attribute types.

Before selecting the 5 4-tuples from which we created the HOPs in CLEVR-HOPE,
we first created a small set of minimal test questions for testing how well a
given model comprehends a given attribute in isolation — CLEVR-PRELIM. For
example, for the color cyan we had two types of tests. First, tests similar to
the minimal-OOD test tests (i.e., a single object and rephrasings of “Are any
cyan objects visible?”). Second, counting tests — all questions were rephrases
of “What number of cyan objects are there?”, and images had varying numbers of
cyan objects. Specifically, we fixed the position of 5 objects, and created 6
images, each with a different number of objects matching the attribute — i.e.,
0, 1, 2, 3, 4, or 5 cyan objects.

Note that, unlike CLEVR-HOPE which studies pairs of attributes values, CLEVR-
PRELIM evaluates only attribute values in isolation.

Using CLEVR-PRELIM, we performed a zero-shot evaluation of Tan and Bansal
(2019)’s VQA2.0 Goyal et al. (2017) fine-tuned LXMERT checkpoint. From this
preliminary study we found that zero-shot model performance was generally poor
(e.g., over all attribute values of all types, the highest count performance
was 49.1%). Given our interest in studying the impact of the amount of
training data, we created our first 4-tuple by individually selecting each
attribute value; specifically choosing the attribute value that zero-shot
LXMERT had the lowest performance on — this created the 4-tuple Large cyan
rubber cylinder. The remaining four tuples were selected uniformly at random.
Ultimately, as we did not see any significant difference between a small
sample of 6 HOPs (those created from attribute pairs in large cyan rubber
cylinder) and a larger sample of 23 HOPs (those created from random 4-tuples),
we present results aggregated over all 29 HOPs.

Note that as two 4-tuples were rubber spheres and small spheres, we added the
HOPs rubber cube and small cube so that we would maintain five material-shape
and five size-shape pairs.

HOP | Attribute Types | Diversity
---|---|---
Large rubber |  Size \+ Material | 4
Small rubber |  Size \+ Material | 4
Large metal |  Size \+ Material | 4
Small metal |  Size \+ Material | 4
Rubber cylinder |  Material \+ Shape | 6
Metal cylinder |  Material \+ Shape | 6
Rubber cube |  Material \+ Shape | 6
Metal cube |  Material \+ Shape | 6
Rubber sphere |  Material \+ Shape | 6
Large cylinder |  Size \+ Shape | 6
Small cylinder |  Size \+ Shape | 6
Small cube |  Size \+ Shape | 6
Large cube |  Size \+ Shape | 6
Small sphere |  Size \+ Shape | 6
Rubber cyan |  Material \+ Color | 16
Rubber brown |  Material \+ Color | 16
Rubber purple |  Material \+ Color | 16
Metal red |  Material \+ Color | 16
Metal gray |  Material \+ Color | 16
Large cyan |  Size \+ Color | 16
Small brown |  Size \+ Color | 16
Small purple |  Size \+ Color | 16
Small red |  Size \+ Color | 16
Large gray |  Size \+ Color | 16
Cyan cylinder |  Color \+ Shape | 24
Brown sphere |  Color \+ Shape | 24
Red cylinder |  Color \+ Shape | 24
Gray cube |  Color \+ Shape | 24
Purple sphere |  Color \+ Shape | 24
Table 2: Train set diversity of each held-out pair (i.e., HOP) of object
attribute values. Diversity is the number of possible pairs of attribute
values, corresponding to the HOPs attribute types.

For each HOP in CLEVR-HOPE, the approximate size of the corresponding splits
is outlined below:

  * •

train set: 62k images, and 560k image-question pairs

  * •

complex-IID test set: 13k images, 120k image-question pairs

  * •

complex-OOD test set: 15k images, 15k image-question pairs

  * •

minimal-IID test set: 2576-3200 images, 8640-11970 image-question pairs
(depending on HOP)

  * •

minimal-OOD test set: 448-3840 images, 448-3840 image-question pairs
(depending on HOP)

To reduce the resources required to generate the dataset, images are reused
throughout the dataset. Specifically, the images are reused across the train
sets for the HOPs, and reused from the original CLEVR training set.

Similarly, each of the test sets reuse images across HOPs. Note that while the
complex-IID test and complex-OOD test sets do not reuse eachother’s images,
the minimal-IID test and minimal-OOD test sets do for images that do not
involve the HOP under consideration.

To ensure that CLEVR can be fairly used for hyperparameter tuning, and to
prevent any data leakage, no CLEVR validation or test images are reused in
CLEVR-HOPE.

###  A.1 CLEVR-HOPE: minimal-OOD test set and minimal-IID test set

All images in the minimal-OOD test and minimal-IID test sets contain only a
single object. All questions ask whether there are any objects matching the
attribute value pair. E.g., for the HOP rubber cyan, some question variants
include “Are there any cyan matte things?” and “Are any cyan matte things
visible?”.

These splits are designed to test the model in a systematic manner: each image
matching the HOP has 3 corresponding images that do not match the HOP. These 4
images share identical question phrasing. The non-matching images maintain the
object position, lighting, and the attribute values that are irrelevant to the
HOP, but change the first attribute value in the HOP, the second attribute
value in the HOP, or both attribute values in the HOP, respectively. See Fig.
4 for an example.

Figure 4: Four example image-question pairs for the minimal-OOD test split of
the sub-dataset of CLEVR-HOPE corresponding to the first held-out attribute
pair — i.e., rubber cylinder. Note how the first image matches rubber cylinder
(material=rubber, and shape=cylinder), and the next three image have one
attribute value (material=metal), the other attribute value (shape=cube), or
both (material=metal, and shape=cube) attribute values not matching rubber
cylinder. This pattern repeats throughout the dataset, with the choice of
distractor values, object position, lightning, question-phrasing and the value
of the attribute-types not in HOP, all chosen randomly, but fixed within each
set of 4 images.

Note that the question template is taken directly from the original CLEVR
dataset generation code. The main change is the aforementioned systematic
design, and that the images used contain only a single object, whereas the
original CLEVR requires at least 3 objects in any scene.

The minimal-IID test split is created in the same way, but testing all other
attribute-value pairs of the same type as the HOP. Note that the distractor
attribute values in the negative examples were selected uniformly at random.
Since this may create the held-out pair (and indeed, must do so for one of the
four size-material images), after the initial creation of the minimal-IID test
split, we filter it to remove any image-question pairs where the object in the
image matches the HOP.

##  Appendix B Training details

All subsets of the train sets (i.e., of size 25k, 200k, and 560k) are created
by taking the first however many indices. This corresponds to a random subset
of images for 25k, which is consecutively randomly expanded. This is so
because the image-question pairs are unsorted, apart from all questions for
any given image having contiguous indices. Note that we fix the number of
gradient updates across subset sizes, i.e., smaller subsets are trained for
more epochs so that the total number of gradient updates is the same.

Hyperparameter | LXMERT-p | LXMERT-s
---|---|---
Learning Rate | 5e-5 | 1e-5
Gradient Updates | 218,750 | 481,000
Batch size | 32 | 32
Table 3: Key hyperparameter values used for LXMERT

For LXMERT, the maximum sequence length is increased to 49 so that CLEVR-HOPE
questions are not truncated.

For LXMERT-p, we follow Tan and Bansal (2019)’s procedure for finetuning their
pretrained LXMERT checkpoint on a VQA dataset. As part of their procedure, the
pretrained F-RCNN Ren et al. (2015) object detector is not altered in any way.

LXMERT-p hyperparameters were modified from the hyperparameters used by Tan
and Bansal (2019) for finetuning LXMERT for VQA. Specifically, Tan and Bansal
(2019) finetuned LXMERT for the VQA tasks of VQAv2 Goyal et al. (2017), NLVR2
Suhr et al. (2019), and GQA Hudson and Manning (2019) with a batch size of 32,
4 epochs, and a learning rate of either 1e-5 or 5e-5. We ultimately used a
learning rate of 5e-5, and increased the epochs to 10 as we found it yielded
better performance.

For LXMERT-s we randomly initialize all LXMERT weights (this excludes the
pretrained F-RCNN object detector), and apply the LXMERT finetuning procedure
(albeit with different hyperparamters) to train this randomly initialized
model.

LXMERT-s hyperparameter tuning was performed via grid search over learning
rate (1e-4, 5e-5, 1e-5) and training steps (218750, 481000, 700000). Note that
we ultimately used 481k gradient update steps, as its validation accuracy
(95.47%) was extremely close to 700k (96.99%), with nearly half the training
time.

The LXMERT hyperparameters used are summarized in Tab. 3.

Tensor-NMN is trained from scratch following the process used by Bahdanau et
al. (2019). Tensor-NMN is trained in a 3 stage process — initially the program
generator and execution engine are trained in a supervised manner, following
which they are trained together using REINFORCE. The default hyperparameters
for CLEVR from Bahdanau et al. (2019) are used.

##  Appendix C LXMERT Detailed Results

LXMERT performance on minimal-OOD test can be found in Fig. 5. Performance on
minimal-IID test can be found in Fig. 6. All plots mark 75% — this baseline
performance is achieved on the minimal-OOD test split by always predicting
false (i.e., the most common class). Always predicting false on minimal-IID
test yield a baseline performance between 66% and 75%, depending on the HOP.

Figure 5: Box plot of minimal-OOD test set performance on all 29 HOPs. The
average performance for each HOP is produced by averaging over 3 trials. The
variation captured by this boxplot is from the difference in average
performance between HOPs, rather than from the variation within the 3 trials.

Figure 6: Box plot of minimal-IID test set performance on all 29 HOPs. The
average performance for each HOP is produced by averaging over 3 trials. The
variation captured by this boxplot is from the difference in average
performance between HOPs, rather than from the variation within the 3 trials.

LXMERT performance on complex-OOD test can be found in Fig. 7. Performance on
complex-IID test can be found in Fig. 8.

Figure 7: Box plot of complex-OOD test set performance on all 29 HOPs. The
average performance for each HOP is produced by averaging over 3 trials. The
variation captured by this boxplot is from the difference in average
performance between HOPs, rather than from the variation within the 3 trials.

Figure 8: Box plot of complex-IID test set performance on all 29 HOPs. The
average performance for each HOP is produced by averaging over 3 trials. The
variation captured by this boxplot is from the difference in average
performance between HOPs, rather than from the variation within the 3 trials.

For LXMERT trained on the largest train sets (560k), we plot the complex and
minimal model accuracies, averaged by the attribute types of the HOPs, in Fig.
9.

Figure 9: Model accuracies for HOP-0 through 28. Note that the LXMERT models
often struggle on both IID and OOD questions when Material-Shape combinations
are held out at train time.

##  Appendix D Tensor-NMN Detailed Results

As Tensor-NMN was only evaluated on the first 6 HOPs, we include the subset of
LXMERT models trained on the same HOPs for comparison.

Model performance on minimal-OOD test can be found in Fig. 10. Performance on
minimal-IID test can be found in Fig. 11. All plots mark 75% — this baseline
performance is achieved on the minimal-OOD test split by always predicting
false (i.e., the most common class). Always predicting false on minimal-IID
test yield a baseline performance between 66% and 75%, depending on the HOP.

Figure 10: Average minimal-OOD test set Tensor-NMN performance for the first 6
HOPs over 3 trials. For comparison, we also plot the average LXMERT model
performances (i.e., Fig. 5), but restricted to only the first 6 HOPs. An area
corresponding to 1 standard deviation is shaded.

Figure 11: Average minimal-IID test set Tensor-NMN performance for the first 6
HOPs over 3 trials. For comparison, we also plot the average LXMERT model
performances (i.e., Fig. 6), but restricted to only the first 6 HOPs. An area
corresponding to 1 standard deviation is shaded.

Model performance on complex-OOD test can be found in Fig. 12. Performance on
complex-IID test can be found in Fig. 13.

Figure 12: Average complex-OOD test set Tensor-NMN performance for the first 6
HOPs over 3 trials. For comparison, we also plot the average LXMERT model
performances (i.e., Fig. 7), but restricted to only the first 6 HOPs. An area
corresponding to 1 standard deviation is shaded.

Figure 13: Average complex-IID test set Tensor-NMN performance for the first 6
HOPs over 3 trials. For comparison, we also plot the average LXMERT model
performances (i.e., Fig. 8), but restricted to only the first 6 HOPs. An area
corresponding to 1 standard deviation is shaded.

For Tensor-NMN trained on the largest train sets (560k), we plot the complex
and minimal model accuracies, averaged by the attribute types of the HOPs. The
results are visualized in Fig. 14. Again, we include the corresponding subset
of LXMERT models for comparison.

Figure 14: Model accuracies for only the first 6 HOPs. Note that while the
LXMERT models struggle with Material-Shape combinations on OOD questions,
Tensor-NMN does not.

##  Appendix E Systematicity Gap

As outlined in Section 4.2, we find that, on all models, averaged over HOPs,
the gap between performance on complex questions involving IID vs. OOD
attribute combinations does not trend to zero. Instead, it plateaus (see
Figures 15 and 17). In comparison, the performance gap on minimal questions
plateaus or decreases gently (see Figures 16 and 18).

Figure 15: Average systematicity gap on complex examples (i.e., complex-OOD
test accuracy minus complex-IID test accuracy) with 1 standard deviation;
averaged over 3 runs on each of the 29 HOPs. Note that the systematicity gap
plateaus, suggesting that the performance drop when generalizing to unseen
combinations does not improve with additional training data.  Figure 16:
Average systematicity gap on minimal examples (i.e., minimal-OOD test accuracy
minus minimal-IID test accuracy) with 1 standard deviation; averaged over 3
runs on each of the 29 HOPs.  Figure 17: Average systematicity gap on complex
examples (i.e., complex-OOD test accuracy minus complex-IID test accuracy)
with 1 standard deviation; averaged over 3 runs on only the first 6 HOPs. Note
that the systematicity gap plateaus, suggesting that the performance drop when
generalizing to unseen combinations does not improve with additional training
data. Figure 18: Average systematicity gap on minimal examples (i.e., minimal-
OOD test accuracy minus minimal-IID test accuracy) with 1 standard deviation;
averaged over 3 runs on only the first 6 HOPs.

###  E.1 Detailed Tensor-NMN Systematicity Gap

Averaging the systematicity gap in Tensor-NMN by diversity, we again find
stratification by diversity for complex examples (see Fig. 19). The trend on
minimal examples is noisier, but ultimately converges to the expected ordering
(see Fig. 20). Note that, as is to be expected, when limited to the first six
HOPs the LXMERT trend is also noisier. It is therefore reasonable to expect
the Tensor-NMN trend would be cleaner with additional HOPs.

Figure 19: Systematicity gap (i.e. difference between OOD and IID model
performance) for complex examples, averaged by HOP diversity over for the
first 6 held-out attribute pairs only, each with 3 runs.

Figure 20: Systematicity gap (i.e. difference between OOD and IID model
performance) for minimal examples, averaged by HOP diversity over for the
first 6 held-out attribute pairs only, each with 3 runs.