# Dispersion Characteristics of a Glide-Symmetric Square Patch Metamaterial
with Giant Anisotropy

Jim A. Enriquez<EMAIL_ADDRESS>School of Physics and
Engineering, ITMO University, 197101, Saint Petersburg, Russia  Eugene
Koreshin  School of Physics and Engineering, ITMO University, 197101, Saint
Petersburg, Russia  Juan P. Del Risco  National University of Colombia,
Department of Physics, 111321, Bogota, Colombia  Pavel A. Belov  School of
Physics and Engineering, ITMO University, 197101, Saint Petersburg, Russia
Juan D. Baena  National University of Colombia, Department of Physics, 111321,
Bogota, Colombia

###### Abstract

This paper investigates the dispersion characteristics of a highly anisotropic
metamaterial comprised of metal square patches arranged in a glide symmetry
pattern and submerged in vacuum. Theoretical formulas are proposed to describe
the electromagnetic tensors of a corresponding uniaxial effective medium with
dielectric and magnetic responses. In addition, this work employs theoretical
analysis and numerical simulations to examine the interaction between the
metamaterial and electromagnetic waves across a broad spectral range. Band
diagrams and isofrequency contours show good agreement between theoretical and
numerical results for low frequencies and certain directions of propagation at
higher frequencies. The ease of designing the metamaterial structure for
various applications is facilitated by the derived theoretical formulas, which
enable accurate prediction of the electromagnetic response across a wide range
of frequencies based on geometric parameters.

††preprint: APS/123-QED

##  I Introduction

Metamaterials are artificial materials with electromagnetic properties beyond
those found in natural materials. One particular class of metamaterial
exhibits anisotropy, meaning that material responses depend on the direction
of propagation of electromagnetic waves. Studies have revealed exotic
electromagnetic properties in anisotropic metamaterials such as negative-
refraction [1], near-zero parameters [2], and manipulation of polarization
states in reflection [3].

Symmetries, together with an adequate selection of parameters and constituents
of metamaterial structures, enable the design of materials with particular
electromagnetic wave propagation properties. A symmetry that has been
exploited to attain exceptional electromagnetic responses is a higher symmetry
known as glide symmetry. Structures with gliding symmetry are those built by
reflecting and translating building blocks. It has been shown that with an
adequate selection of parameters, metamaterials based on glide symmetry allow
to reduce the dispersion [4, 5] and increase the effective refractive index
[6, 7].

A unique anisotropic metamaterial based on glide symmetry consist of an array
of square metal patches that are repeated with a period $a$ along the
directions parallel to the patches (transverse directions), and a period $b$
along the direction perpendicular to the patches (axial direction), as shown
Fig. 1 for a supercell of the structure. With a much larger period along the
transversal directions compared to the axial direction ($a>>b$), a high
dielectric permittivity uniaxial material is obtained [8, 9]. Taking advantage
of the unique characteristics of the metamaterial, it has been proposed as a
component of practical applications related with microwave engineering [10,
11] and magnetic resonance imaging [12].

Effective medium models have been proposed to study the dispersion
characteristics of the square patch metamaterial along the axial direction,
considering a uniform electric charge across the metal patches [8].
Furthermore, analytical models have been put forth to realize the scattering
parameters of plane waves impinging structures based on layers with non
aligned metal patches [13]. However, the magnetic behavior of the structure
and the frequency range of validity for existing effective medium models are
not well established yet. In this article, we present theoretical formulas
that predict the dispersion properties of the square patch metamaterial
depending on the geometrical parameters of the structure. The theoretical
results are compared with numerical ones to check the scope of validity of the
proposed effective medium. Dispersion diagrams and isofrequency contours of
propagating modes supported by the structure are analyzed. In addition, the
electromagnetic parameters are theoretically and numerically retrieved for low
frequencies. Our results show that the effective medium approach is valid for
low frequencies and for high frequencies along certain directions of
propagation.

Figure 1: Supercell of the metallic square patch metamaterial, where the
spatial period along the $z$-axis is $b$, and the spatial periods along the
$x$\- and $y$-axes are both $a$.

##  II Theory

The theoretical model is based on a quasi-static approach including both
electrostatic and magnetostatic studies [14]. For simplicity, the host medium
is considered as vacuum, and the metallic patches are considered made of
Perfect Electric Conductor (PEC). Bianisotropic effects are discarded because
of the presence of a center of symmetry in the unit cell and non-linear
effects are not considered. Consequently, linear constitutive relations are
assumed. Moreover, due to mirror symmetries of the unit cell, both the
permittivity and permeability tensors are diagonal. Therefore, the
constitutive relations can be written as,

$\mathbf{D}=\epsilon_{0}\begin{pmatrix}\epsilon_{t}&0&0\\\ 0&\epsilon_{t}&0\\\
0&0&1\end{pmatrix}\mathbf{E},$ (1) $\mathbf{B}=\mu_{0}\begin{pmatrix}1&0&0\\\
0&1&0\\\ 0&0&\mu_{z}\end{pmatrix}\mathbf{H},$ (2)

in which the unknown parameters are the relative transverse permittivity
($\epsilon_{t}$) and the relative axial permeability ($\mu_{z}$). For
identifying the components of the electromagnetic tensors it is considered
that the field components $E_{z}$, $B_{x}$, and $B_{y}$ do not interact with
the conducting patches, then $\epsilon_{z}=\mu_{x}=\mu_{y}=1$. In addition,
the structure is symmetric under a $90^{\circ}$ rotation around the $z$-axis,
implying $\epsilon_{x}=\epsilon_{y}=\epsilon_{t}$.

###  II.1 Transverse relative permittivity

Figure 2: Electric response of the square patch metamaterial to a macroscopic
electric field oriented along the $x$-axis. (a) Induced charge and (b)
microscopic electric field ($\mathbf{e}$) distribution within the unit cell.

It is assumed that both the macroscopic electric field $\mathbf{E}$ and the
polarization $\mathbf{P}$ are oriented along the $x$-axis, considering that
the macroscopic electric field is understood as the average electric field
along the unit cell. The electric responses within the unit cell due to this
$x$-oriented $\mathbf{E}$ are illustrated in Fig. 2. As a consequence, from
the usual constitutive relation connecting the electric vectors,

$D_{x}=\epsilon_{0}\ E_{x}+P_{x}=\epsilon_{0}\ \epsilon_{t}E_{x},$ (3)

the transverse permittivity can be obtained,

$\epsilon_{t}=1+\frac{P_{x}}{\epsilon_{0}\ E_{x}\ }.$ (4)

Therefore, it is needed to analyze $P_{x}$ and $E_{x}$ to estimate the
transverse permittivity.

To analyze $E_{x}$, the microscopic electric field $\mathbf{e}$ is used, which
is the electric field in each point of the unit cell (see Fig. 2b). Following
a vertical line passing through the points F, A, B, and C marked in Fig. 2b,
it is noted that due to the boundary conditions in PEC, it is possible to
obtain a non-zero electric field only in the gap of vacuum, so that,

$E_{x}a=\int_{A}^{B}e_{x}\>dx.$ (5)

Conversely, applying Faraday’s Law in the quasi-static regime along the closed
integration path ABCDEF depicted in Fig. 2b,

$\oint\mathbf{e}\cdot\>d\mathbf{r}=\int_{A}^{B}e_{x}\>dx+\int_{C}^{D}e_{z}\>dz+\int_{E}^{F}e_{z}\>dz\approx
0.$ (6)

Utilizing Eqs. (5) and (6), and assuming a uniform and equivalent $|e_{z}|$
along paths CD and EF,

$E_{x}=\frac{2}{a}\int_{C}^{D}e_{z}\>dz\approx\frac{b}{2a}\frac{\sigma}{\epsilon_{0}},$
(7)

where $\sigma$ is the surface density in the half of a patch.

The polarization ($P_{x}$) is obtained by calculating the dipole moment due to
two quarters of patches divided by the quarter of volume of the supercell
since there are two patches in the cell,

$P_{x}=\frac{p_{x}}{ba^{2}/4}=\frac{qa/2}{ba^{2}/4}=\frac{2\sigma(\tfrac{a}{2}-g)^{2}}{ba},$
(8)

where $(\tfrac{a}{2}-g)^{2}$ represents the overlapping area between quarters
of patches in adjacent layers. Substituting Eqs. (7), and (8) into Eq. (4)
yields,

$\epsilon_{t}=1+\left(\frac{a-2g}{b}\right)^{2}.$ (9)

Including edge effects of the electric charge, the polarization is corrected
as,

$P_{x}^{\prime}=P_{x}+\Delta
P=\frac{2\sigma(\tfrac{a}{2}-g)^{2}}{ba}+\frac{16\lambda(\tfrac{a}{2}-g)}{ba}.$
(10)

where the last term comes from the charge density along the edges of quarter
of patches. $\lambda$ is a linear charge density, which is derived in
Supplemental Material, $\lambda=0.4423\sigma b/2$. Therefore, substituting
Eqs. (7) and (10) into Eq. (4),

$\epsilon_{t}=1+\left(\frac{a-2g}{b}\right)^{2}+1.7692\frac{a-2g}{b}.$ (11)

###  II.2 Axial relative permeability

Figure 3: Magnetic response of the square patch metamaterial to a macroscopic
magnetic flux density oriented along the $z$ axis. (a) Induced currents, (b)
microscopic magnetic field ($\mathbf{b}$) distribution within the unit cell.

It is assumed that both the macroscopic magnetic flux density $\mathbf{B}$ and
the magnetization $\mathbf{M}$ are oriented along the $z$-axis, considering
that the macroscopic magnetic flux density is understood as the average
magnetic flux density along the unit cell. The magnetic responses within the
unit cell due to this $z$-oriented $\mathbf{B}$ are illustrated in Fig. 3.
Consequently, from the usual constitutive relation connecting the magnetic
vectors,

$B_{z}=\mu_{0}(H_{z}+M_{z})=\mu_{0}\mu_{z}H_{z},$ (12)

the axial permeability can be obtained,

$\mu_{z}=\frac{1}{1-\frac{\mu_{0}M_{z}}{B_{z}}}.$ (13)

Once $M_{z}$ and $B_{z}$ are identified, the axial permeability can be
estimated. To analyze $B_{z}$, the microscopic magnetic flux density
$\mathbf{b}$ is used, which is the magnetic flux density in each point of the
unit cell. According to the boundary conditions for PEC, $b_{z}$ should be
negligible in the regions bounded by parallel plates since they are tightly
packed and $b_{z}=0$ over the PEC. On the other hand, $b_{z}\neq 0$ along the
canals formed by the crosses of orthogonal slots, see Fig. 3b. Therefore,
applying Ampere’s Law in the quasi-static regime along the closed integration
path ABCD depicted in Fig. 3b results in,

$b_{z}=-\frac{\mu_{0}I}{b}.$ (14)

Equating the magnetic flux in the unit cell across the $xy$-plane from the
macroscopic and microscopic points of view and using Eq. (14),

$B_{z}=\frac{b_{z}2g^{2}}{a^{2}}=-\frac{g^{2}\mu_{0}I}{a^{2}b/2},$ (15)

Regarding the magnetization, following the currents illustrated in Fig. 3a for
one patch that assure a $b_{z}\neq 0$ along the canals,

$M_{z}=\frac{m_{z}}{a^{2}b/2}\approx\frac{AI}{a^{2}b/2}=\frac{a^{2}/2-g^{2}}{a^{2}b/2}I.$
(16)

Substituting Eqs. (15) and (16) into Eq. (13),

$\mu_{z}=\frac{2g^{2}}{a^{2}}.$ (17)

Including edge effects due to the microscopic magnetic flux density, the
macroscopic magnetic field is corrected as,

$B_{z}^{\prime}=B_{z}+\Delta
B_{z}=\left(\frac{g^{2}\mu_{0}I}{a^{2}b/2}+8g\lambda_{m}\right),$ (18)

where the last term comes from the magnetic flux along the edges of the canals
within the unit cell. $\lambda_{m}$ is a linear density of magnetic flux,
which is derived in Supplemental Material, $\lambda_{m}=0.4423b_{z}b/2$.
Therefore, substituting Eqs. (16) and (18) into Eq. (13),

$\mu_{z}=\frac{2g^{2}+1.7692gb}{a^{2}+1.7692gb},$ (19)

###  II.3 Dispersion relations

Using Maxwell’s equations and considering plane wave propagation, the
dispersion equations for the modes supported by the effective uniaxial medium
are derived in accordance with [15],

$\frac{\omega a}{2\pi
c}=\frac{a}{2\pi}\left(\frac{k_{x}^{2}+k_{y}^{2}}{\epsilon_{t}\mu_{z}}+\frac{k_{z}^{2}}{\epsilon_{t}}\right)^{1/2},$
(20) $\frac{\omega a}{2\pi
c}=\frac{a}{2\pi}\left(k_{x}^{2}+k_{y}^{2}+\frac{k_{z}^{2}}{\epsilon_{t}}\right)^{1/2}.$
(21)

When waves propagate perpendicular to the metal patches, the material’s
behavior is primarily governed by the transverse permittivity
($\epsilon_{t}$). However, for grazing incidence, the transverse permeability
($\mu_{z}$) also becomes a significant factor. This type of metamaterial
belongs to the class of uniaxial dielectric-magnetic media, established in the
$1990_{s}$ [16, 17]. Recent studies have proposed the use of uniaxial
dielectric-magnetic materials as a foundation for realizing angle-selective
surfaces [18] and impedance matching layers [19].

##  III Numerical investigation

Figure 4: (a) Structure of the square patch metamaterial. (b) Primitive unit
cell with translation vectors: $\mathbf{a_{1}}$, $\mathbf{a_{2}}$, and
$\mathbf{a_{3}}$. (c) Brillouin zone and high-symmetry points with
coordinates: $\Gamma=\left(0,0,0\right)$,
$\mathrm{X}=\left(\tfrac{2\pi}{a},0,0\right)$,
$\mathrm{M}=\left(\tfrac{\pi}{a},\tfrac{\pi}{a},0\right)$,
$\mathrm{Z}=\left(0,0,\tfrac{\pi}{b}+\tfrac{b\pi}{a^{2}}\right)$,
$\mathrm{P_{1}}=\left(\tfrac{2\pi}{a},0,\tfrac{\pi}{b}-\frac{b\pi}{a^{2}}\right)$,
$\mathrm{P_{2}}=\left(\tfrac{\pi}{a},\tfrac{\pi}{a},\tfrac{\pi}{b}\right)$.

To illustrate the highly anisotropic response of the studied structure, we
consider the following geometrical parameters: $b/a=0.025$, and $g/a=0.075$.
Substituting these values into Eqs. (11) and (19) yields in a relative
transversal permittivity of $\epsilon_{t}=1217$ and a relative axial
permeability of $\mu_{z}=0.01451$.

The periodic nature of the square patch metamaterial reassembles a body-
centered tetragonal crystal system [20]. Therefore, a supercell of the
structure consists in translation vectors $a\hat{x}$, $a\hat{y}$, and
$b\hat{z}$ containing two patches, as illustrates Fig. 1. In this study, we
consider a primitive unit cell to numerically analyze the dispersion of the
square patch metamaterial, as shown Fig. 4b. The election of a primitive unit
cell to study a metamaterial is important to completely describe the
dispersion characteristics of the structure and avoid redundant information,
as was shown in [21], [22]. In addition, the selected unit cell fully exploits
the glide symmetry of the structure, considering that each node in the cell is
connected to adjacent points in different planes that are translated
$\tfrac{a}{2}$ along both $x$ and $y$ directions (see Fig. 4a). Consequently,
the components of the translation vectors of the primitive unit cell are given
by,

$\begin{split}\mathbf{a_{1}}=\left(\tfrac{a}{2},-\tfrac{a}{2},\tfrac{b}{2}\right),\\\
\mathbf{a_{2}}=\left(-\tfrac{a}{2},\tfrac{a}{2},\tfrac{b}{2}\right),\\\
\mathbf{a_{3}}=\left(-\tfrac{a}{2},-\tfrac{a}{2},\tfrac{b}{2}\right).\end{split}$
(22)

The Brillouin zone corresponding to the primitive unit cell, with the shape of
a dodecahedron, is illustrated in Fig. 4c. The dispersion characteristics of
the square patch metamaterial are numerically studied using the commercial
software COMSOL MULTIPHYSICS, implementing periodic boundary conditions to the
primitive unit cell, with $e^{i\mathbf{k}\cdot\mathbf{r}}$ spatial dependence.

###  III.1 Dispersion relations

Figure 5: Theoretical and numerical dispersion diagrams of the square patch
metamaterial along paths involving the high symmetric points marked in Fig.
4c, (a) path $\mathrm{\Gamma XM\Gamma}$ and (b) path $\mathrm{\Gamma
ZP_{1}XP_{2}\Gamma}$. TE mode and TM mode correspond to Eqs. (20) and (21),
respectively.

The dispersion relations are analyzed along paths involving the high symmetric
points illustrated in Fig 4c. As a result, Fig. 5 illustrates theoretical and
computational dispersion diagrams, in which TE mode and TM mode correspond to
Eqs. (20) and (21), respectively.

The dispersion relations along path $\Gamma\mathrm{XM}\Gamma$ are illustrated
in Fig. 5a. The TM mode is characterized by a lack of interaction with the
metal patches, resulting in a perfect degree of similarity with numerical
results and resembling the propagation of light in vacuum, as shown Eq. (21)
with $k_{z}=0$ . Conversely, the TE mode demonstrates good agreement for low
frequencies, with a relative error of $7.1\%$ observed at the $\mathrm{X}/3$
point. The agreement between theoretical and computational results starts to
degrade for regions close to the edges of the first Brillouin zone, which is a
typical limitation of effective medium models [23]. In this work, the relative
error is calculated by comparing simulation results relative to theoretical
predictions.

Including points with $k_{z}\neq 0$, Fig. 5b illustrates the dispersion
relations along path $\Gamma\mathrm{ZP_{1}XP_{2}}\Gamma$. There is a notable
concurrence between theoretical and numerical results for both TE and TM modes
below $\omega a/(2\pi c)=0.5$. In addition, along $k_{z}$ direction, which is
the direction exhibiting a giant relative permittivity, the higher the
frequency, the higher the relative error, with a maximum value of $8.3\%$
observed at $\mathrm{Z}$ point.

The agreement between theoretical and numerical results for the dispersion
relations is further supported by the computational validation of the assumed
electric and magnetic responses of the metamaterial. To this end, plane waves
with frequency $\omega a/(2\pi c)=0.1$ are impinged upon a supercell of the
structure, as shown in Fig. 6. The results corroborate the theoretical
electric responses in Fig. 2 and the theoretical magnetic responses in Fig. 3.

Figure 6: Computational results for the square patch metamaterial at frequency
$\omega a/(2\pi c)=0.1$: (a, b) Electric response: Induced charge and
microscopic electric field due to a plane wave with electric field along $x$.
(c, d) Magnetic response: Surface current density and microscopic magnetic
flux density due to a plane wave with magnetic flux density along $z$. Figure
7: Theoretical and computational isofrequency contours for the square patch
metamaterial in plane $k_{y}=0$ for normalized frequencies (a) $\omega a/(2\pi
c)=0.10$, (b) $\omega a/(2\pi c)=0.15$, (c) $\omega a/(2\pi c)=0.50$, and (d)
$\omega a/(2\pi c)=0.60$. The dotted black lines indicate the edges of the
first Brillouin zone.

Following the dispersion analysis of the structure, isofrequency contours in
plane $k_{y}=0$ are illustrated in Fig. 7, in which the black dotted line
indicates the edges of the first Brillouin zone. At low frequencies, i.e.
$\omega a/(2\pi c)=0.10$, Fig. 7a shows a perfect agreement between theory and
simulations for both TE and TM modes. However, this agreement weakens at
higher frequencies. At $\omega a/(2\pi c)=0.15$, illustrated in Fig. 7b, it is
noted that there is a perfect agreement for the TM mode, but the TE mode shows
a discrepancy between theoretical and numerical results along directions close
to $k_{x}$-direction. At $\omega a/(2\pi c)=0.50$, the agreement for the TE
mode is limited to $k_{z}$-direction, and the agreement for TM mode is limited
to $k_{z}$\- and $k_{x}$\- directions, see Fig. 7c. Finally, Fig. 7d shows a
significant discrepancy at $\omega a/(2\pi c)=0.60$, indicating a breakdown of
the model’s validity due to a topological transition in the isofrequency
contours.

###  III.2 Electromagnetic parameters dependence on geometry

In order to investigate the performance of the model under variations of the
gap ($g$) between metallic patches, Fig. 8 shows theoretical and simulations
results for the effective permittivity and permeability of the structure as a
function of $g/a$ when $b/a=0.025$. The theoretical results are obtained
directly from equations (9) and (17) for Model 1, and from equations (11) and
(19) for Model 2, which include edge effects. On the other hand, numerical
results are obtained from the dispersion along $k_{z}$-direction to retrieve
first the transversal permittivity, considering that a linear dispersion is
evidenced in Fig. 5b (path $\Gamma\mathrm{Z}$) at low frequencies,

$\epsilon_{t}=\left(k_{z}\frac{c}{\omega}\right)^{2}.$ (23)

Then, the axial permeability $(\mu_{z})$ is retrieved from the dispersion
along $k_{x}$-direction, considering a linear dispersion at low frequencies
(see Fig. 5a , path $\Gamma\mathrm{X}$),

$\mu_{z}=\frac{1}{\epsilon_{t}}\left(k_{x}\frac{c}{\omega}\right)^{2}.$ (24)

Figure 8: Effect of the gap size on (a) relative permittivity and (b) relative
permeability of the square patch metamaterial. Results are shown as a function
of the gap size per transversal period ($g/a$), according to Model 1 (without
edge effects), Model 2 (with edge effects), and numerical dispersion relations
at low frequencies.

Fig. 8a illustrates the effective relative permittivity ($\epsilon_{t}$),
showing an advantageous range of values from approximately $800$ to $1200$ for
the interval $0.07\leq g/a\geq 0.16$. Similarly, Fig. 8b demonstrates a range
of effective relative permeability values ($\mu_{z}$) from approximately
$0.010$ to $0.050$ within the same interval. It is worth noting that Model 2
exhibits improved agreement with the numerical results compared to Model 1. A
smaller $g/a$ ratio weakens the assumption of an exponential decay for the
edge effect (see Supplemental Material), leading to a larger relative error in
the calculated magnetic permeability.

Figure 9: Effect of axial period on (a) relative permittivity and (b) relative
permeability of the square patch metamaterial. Results are shown as a function
of the axial period per transversal period ($b/a$), according to Model 1
(without edge effects), Model 2 (with edge effects), and numerical dispersion
relations at low frequencies.

Figure 9 presents theoretical and simulation results, obtained using the
aforementioned methodology, for the effective relative permittivity and
effective relative permeability of the structure as a function of $b/a$ when
$g/a=0.075$. Within the domain $0.0125\leq b/a\geq 0.0228$ a wide range of
transversal permittivity values are achieved in Fig. 9a (from approximately
$1500$ to $4500$). In contrast, Fig. 9b shows minimal variation in
permeability within the same domain, with values ranging from approximately
$0.013$ to $0.017$. Notably, Model 2 exhibits significantly improved agreement
with the numerical results compared to Model 1. When the $b/a$ ratio
increases, the agreement of the theoretical models decreases with respect to
numerical results since the assumption of tightly packet patches weakens.

##  IV Conclusions

A theoretical and computational study was conducted to elucidate the
dispersion characteristics of the square patch metamaterial. Our findings
reveal that the structure emulates the behavior of a uniaxial dielectric
magnetic material at low frequencies. Furthermore, we have established
accurate and straightforward theoretical formulas for estimating the key
electromagnetic parameters of these metamaterials. These formulas unveil the
critical dependence of both the transversal permittivity and the axial
permeability on the spatial periods and the gaps between the constituent
patches. Moreover, the derived expressions predict that the square patch
metamaterial not only exhibits an exceptionally large transverse permittivity
but also possesses an extremely small axial permeability.

Comparing theoretical results with computational simulations based on a body-
centered tetragonal representation of the square patch metamaterial, the
theoretical model demonstrates its efficacy in predicting the dispersion
characteristics up to approximately $\omega a/(2\pi c)=0.5$ coinciding with
the onset of topological transitions. Notably, good agreement is observed
between theory and simulations, with a relative error of $7.1\%$ at
$\mathrm{X/3}$ point and $8.3\%$ at $\mathrm{Z}$ point. Furthermore, the
theoretical formulas allow for the prediction of the relative transverse
permittivity within a range of 800 to 4500, and the relative permeability
within a range of approximately 0.010 to 0.050. These findings highlight the
potential of the developed theoretical model for guiding the design of square
patch metamaterials with tailored electromagnetic properties. Further research
could explore the application of this model to similar structures with even
more degrees of freedom, such as glide-symmetric structures with metallic
objects with shapes different to the square shape, and investigate the
influence of fabrication tolerances on the predicted properties. This would
significantly advance the development of practical metamaterial devices,
enabling functionalities such as angle-selective surfaces with even finer
control over electromagnetic wave propagation or impedance matching layers
with broader bandwidths.

## References

  * Poddubny  _et al._ [2013] A. Poddubny, I. Iorsh, P. Belov, and Y. Kivshar, Nature Photonics 7, 948 (2013) .
  * Zhang and Wu [2015] X. Zhang and Y. Wu, Scientific Reports 5, 7892 (2015).
  * Hao  _et al._ [2007] J. Hao, Y. Yuan, L. Ran, T. Jiang, J. A. Kong, C. T. Chan, and L. Zhou, Physical Review Letters 99, 063908 (2007).
  * Quevedo-Teruel  _et al._ [2020] O. Quevedo-Teruel, G. Valerio, Z. Sipus, and E. Rajo-Iglesias, IEEE Microwave Magazine 21, 36 (2020).
  * Quevedo-Teruel  _et al._ [2021] O. Quevedo-Teruel, Q. Chen, F. Mesa, N. J. Fonseca, and G. Valerio, IEEE Journal of Microwaves 1, 457 (2021).
  * Ebrahimpouri and Quevedo-Teruel [2019] M. Ebrahimpouri and O. Quevedo-Teruel, IEEE Antennas and Wireless Propagation Letters 18, 1547 (2019).
  * Arnberg  _et al._ [2020] P. Arnberg, O. Barreira Petersson, O. Zetterstrom, F. Ghasemifard, and O. Quevedo-Teruel, Applied Sciences 10, 3216 (2020).
  * Chang  _et al._ [2016] T. Chang, J. U. Kim, S. K. Kang, H. Kim, D. K. Kim, Y.-H. Lee, and J. Shin, Nature Communications 7, 12661 (2016).
  * Barzegar-Parizi and Rejaei [2015] S. Barzegar-Parizi and B. Rejaei, IET Microwaves, Antennas & Propagation 9, 1287 (2015).
  * Ma  _et al._ [2008] Y. Ma, B. Rejaei, and Y. Zhuang, IEEE Microwave and Wireless Components Letters 18, 431 (2008).
  * Takahagi and Sano [2011] K. Takahagi and E. Sano, IEEE Transactions on Antennas and Propagation 59, 3624 (2011).
  * Vorobyev  _et al._ [2020] V. Vorobyev, A. Shchelokova, I. Zivkovic, A. Slobozhanyuk, J. D. Baena, J. P. del Risco, R. Abdeddaim, A. Webb, and S. Glybovski, Journal of Magnetic Resonance 320, 106835 (2020).
  * Cavallo and Felita [2017] D. Cavallo and C. Felita, IEEE Transactions on Antennas and Propagation 65, 5303 (2017).
  * Baena  _et al._ [2020] J. D. Baena, J. P. del Risco, and A. C. Escobar, in  _2020 Fourteenth International Congress on Artificial Materials for Novel Wave Phenomena (Metamaterials)_ (2020) pp. 367–369.
  * Depine  _et al._ [2006] R. A. Depine, M. E. Inchaussandague, and A. Lakhtakia, JOSA A 23, 949 (2006).
  * Lakhtakia and Weiglhofer [1997] A. Lakhtakia and W. S. Weiglhofer, International Journal of Applied Electromagnetics and Mechanics 8, 167 (1997).
  * Weiglhofer and Lakhtakia [1996] W. Weiglhofer and A. Lakhtakia, International journal of infrared and millimeter waves 17, 1365 (1996).
  * Huang and Shen [2020] H. Huang and Z. Shen, IEEE Antennas and Wireless Propagation Letters 19, 2457 (2020).
  * He and Eleftheriades [2018] Y. He and G. V. Eleftheriades, Phys. Rev. B 98, 205404 (2018).
  * Mehl  _et al._ [2017] M. J. Mehl, D. Hicks, C. Toher, O. Levy, R. M. Hanson, G. Hart, and S. Curtarolo, Computational Materials Science 136, S1 (2017).
  * Sakhno  _et al._ [2021] D. Sakhno, E. Koreshin, and P. A. Belov, Physical Review B 104, L100304 (2021).
  * Yang  _et al._ [2023] S. Yang, O. Zetterstrom, F. Mesa, and O. Quevedo-Teruel, IEEE Journal of Microwaves (2023).
  * Belov and Simovski [2005] P. A. Belov and C. R. Simovski, Phys. Rev. E 72, 026615 (2005).


# Same Task, More Tokens: the Impact of Input Length on
the Reasoning Performance of Large Language Models

Mosh Levy1 Alon Jacoby11footnotemark: 11 Yoav Goldberg1,2
1Bar-Ilan University 2Allen Institute for AI
{moshe0110<EMAIL_ADDRESS>These authors contributed equally to this work.

###### Abstract

This paper explores the impact of extending input lengths on the capabilities
of Large Language Models (LLMs). Despite LLMs advancements in recent times,
their performance consistency across different input lengths is not well
understood. We investigate this aspect by introducing a novel QA reasoning
framework, specifically designed to assess the impact of input length. We
isolate the effect of input length using multiple versions of the same sample,
each being extended with padding of different lengths, types and locations.
Our findings show a notable degradation in LLMs’ reasoning performance at much
shorter input lengths than their technical maximum. We show that the
degradation trend appears in every version of our dataset, although at
different intensities. Additionally, our study reveals that traditional
perplexity metrics do not correlate with performance of LLMs’ in long input
reasoning tasks. We analyse our results and identify failure modes that can
serve as useful guides for future research, potentially informing strategies
to address the limitations observed in LLMs.

Same Task, More Tokens: the Impact of Input Length on
the Reasoning Performance of Large Language Models

Mosh Levy††thanks: These authors contributed equally to this work.1 Alon
Jacoby11footnotemark: 11 Yoav Goldberg1,2 1Bar-Ilan University 2Allen
Institute for AI {moshe0110<EMAIL_ADDRESS>

##  1 Introduction

Recent advancements in Large Language Models (LLMs) show impressive
performance across a range of tasks OpenAI (2023); Anil et al. (2023); Jiang
et al. (2024), including answering correctly complex questions requiring
multiple reasoning steps Kojima et al. (2022); Wei et al. (2022). These models
also claim to support increasingly longer inputs. This development underscores
the need to examine their performance on the longer inputs they are now
technically supporting.

A reasonable assumption is that support for long inputs would transfer across
tasks and enable a model adept at solving a task when presented in a short
input prompt, to perform the same task when it is embedded within a longer
prompt. Does this assumption hold?

Figure 1: Reasoning performance drops as input grows, across a variety of
tasks. Inputs are composed of text containing information relevant to the task
(in red), and irrelevant text (grey) which is drawn from various sources and
extended incrementally. Two separate text spans are required to answer
correctly, and are located randomly in the input. Each point reflects the
performance across 600 samples.

Recent studies that benchmark models over tasks that involve longer inputs,
including reasoning tasks, indicate that indeed models often struggle with
reasoning over long inputs (Shaham et al., 2023; Li et al., 2023; Bai et al.,
2023). However, these studies do not properly control their variables, and
vary both the input length and the associated tasks to be performed. This
makes it it hard to say if the degraded performance is due to the requirement
to work with longer input, or due to the task being generally harder.

In this work, we study the effect of increasing the input length on model
performance, while keeping other factors as constant as possible.

We employ a methodology to measure model performance trends as a function of
input length, by isolating it as a variable, while keeping the underlying task
intact (§2).

To that end, we introduce Flexible LENgth Question Answering dataset (FLenQA)
111https://github.com/alonj/Same-Task-More-Tokens, a QA dataset for text-based
reasoning (§3). For each sample, composed of a True/False question over two
pieces of information required to answer it (the context), we create multiple
versions of different lengths by embedding the context parts within longer,
irrelevant texts. To ensure that models utilize their entire input, the
dataset is composed of tasks for which both pieces of information must
reasoned over together in order to correctly answer the question. At the same
time, we keep the tasks simple enough such that models answer most of them
correctly when the information pieces are presented on their own, with no
additional padding.

We show that LLMs quickly degrade in their reasoning capabilities, even on
input length of 3000 tokens, which is much shorter than their technical
maximum (on average over all tested models, a drop in accuracy from $0.92$ to
$0.68$).

Additionally, we explore the effect of embedding the information pieces in
various locations within the context, as well as with two kinds of contexts:
similar to the information pieces, or dissimilar to them (§4). We find that
regardless of the experimental setting, there are similar trends of
degradation.

We also show that next-word prediction performance of models on long inputs is
uncorrelated with their performance on downstream tasks of reasoning on long
inputs (§5).

Furthermore, we find that while Chain-of-Thought (CoT) prompting (Kojima et
al., 2022; Wei et al., 2022) increases performance in short inputs, in most
models it does not mitigate the degradation of performance when inputs are
longer: while CoT prompting increases the accuracy over non-CoT prompting, the
amount of increase is roughly consistent across context lengths, and is far
from closing the performance drop due to long context (§6). The only exception
to that is GPT4222we refer to the models gpt-4-1106-preview,
gpt-3.5-turbo-1106 as GPT4 and GPT3.5 accordingly., in which the gap between
CoT and normal prompting increases as the input is longer.

Finally, we analyse our results and identify several failure modes in model
responses (§7). We find that with longer inputs models tend not to follow
specific instructions in the input, either providing no answer, or - in the
case of CoT prompting - presenting the final answer before outlining the
reasoning steps. We also observe a bias towards answering "false", as well as
a decline in the models’ ability to incorporate relevant information in their
responses, as input length increases.

##  2 Desired Data Properties

Our goal is to understand how input length affects LLMs reasoning capabilities
over text, given that the relevant information remains the same. We thus use
question answering tasks that require models to reason over a given text. For
the investigation to be applicable to both open and closed models, we chose a
behavioral approach that relies on input intervention (Holtzman et al., 2023).

We aim for our data to satisfy the following requirements:

##### Ensuring models reason over the input.

To examine the performance of models on long inputs, we require that the task
can only be solved correctly by drawing conclusions from evidence in the text
(Huang and Chang, 2022).

  1. 1.

_Each data sample should contain several relevant text spans that are both
necessary and sufficient to correctly solve the task_.

  2. 2.

_All relevant spans must be consulted jointly to reach a successful solution_.
Some tasks, like text summarization, can be solved using a "divide-and-
conquer" approach (Gidiotis and Tsoumakas, 2020; Liu et al., 2022; Wolhandler
et al., 2022), where each relevant span is individually identified, and then
paraphrased and added to the output. We wish to avoid such decomposable tasks,
as they do not really require reasoning over long inputs.

  3. 3.

To avoid model reliance on parametric knowledge rather than on the text, and
to avoid data contamination Jacovi et al. (2023) _the question and supporting
relevant spans should consist of novel facts not seen in training._ 333Models
often answer the question correctly even if none, or only one of the required
supporting facts is present in its input. We discuss this further in Appendix
A.

##### Isolating the length factor.

To isolate the effect of length, we impose the following requirements:

  1. 1.

_The required reasoning should be independent of the length of the sample_ :
the relevant spans should remain the same in all length variations.

  2. 2.

The added material (a.k.a “padding”, text that is added to control the
samples’ length) _should not contradict or interfere with the reasoning over
the relevant text spans_.

  3. 3.

The location of each relevant span within the input should be controllable.

##### Maintaining natural-looking inputs.

The input should reflect something a user may naturally use in an LLM prompt.
For example, a sequence of unrelated sentences is not natural. In contrast, a
sequence of unrelated paragraphs but where each paragraph is cohesive is more
natural, as such an input may result from collecting relevant information from
multiple sources. To best maintain the naturality of the inputs while changing
an input’s length, we require that the input should be cohesive at least at
the level of paragraphs.

##  3 FLenQA

We introduce the Flexible LENgth Question Answering dataset (FLenQA), which
follows the requirements set in §2.

FlenQA is composed of three reasoning tasks: Monotone Relations (a new task),
People In Rooms (a new task) and a simplified version of Ruletaker (Clark et
al., 2021) (§3.2). Each task consists of 100 base instances, from which we
create variations of different lengths, different background texts, and
different dispersion of facts within the background texts (§3.3).

Each task is completely balanced in its label distribution (“True" and
“False"), and we ensure that most base-instances within it will be solved
correctly by the LLMs when presented in their un-expanded forms (§3.4).

We release the dataset to support future studies of reasoning and long input
performance. Details and statistics of the tasks appear in Appendix B.

###  3.1 Base instances.

Each base-instance consists of (1) an _optional prefix_ (for example
introducing the task or supporting facts); (2) _two key paragraphs_ , each of
which is thematically coherent and starts with a _key sentence_ needed for
solving the task; and (3) an _optional suffix_ (for example, asking a question
about the preceding context).444The optionality is at the task level, either
all instances in the task have a prefix/suffix, or they don’t. For each
instance, the different parts are joined by newlines and fed to the LLM.

Throughout the text, key paragraphs are typeset in red, the supporting
sentences within them in darker red, and the optional prefixes and suffixes in
black. The full prompts used for each dataset are in Appendix D.

##### Deriving the key paragraphs

Each task relies on two facts, expressed as simple sentences. Each of these
sentences is then expanded to a thematically-coherent paragraph, in order to
ensure the naturality requirement. This expansion is performed using GPT-4,
which we prompt to extend the sentences without adding new information,
followed by a manual verification of the results by the authors.

###  3.2 The tasks

##### Monotone relations (MonoRel)

Each key sentence is comparing two person names on monotone scale, e.g. “X is
larger than Y”, “Y is larger than Z”. The suffix is a True/False question that
asks about a relation between two entities that appear in different sentences
(they are not explicitly compared in the text). The relations are transitive
and monotone in nature.

MonoRel Example:
Julie Baker is younger than Julian Barton. This is a fact that remains
constant, unchanging like the northern star. It’s a truth that is as clear as
day that she ...
Samantha Arnold is younger than Julie Baker. It means that Samantha Arnold has
experienced fewer birthdays than Julie Baker. ...
Is Samantha Arnold younger than Julian Barton?

This data is inspired by different monotonic relations describing kinship,
introduced by Sinha et al. 2018. We define a new set of relation types in this
work. Following the requirements in §2, answering the question requires
reasoning over both key sentences. The data is created programmatically by
randomly drawing names from Faker python library (Faraglia and Contributors,
2012) and a relation from a list of hand-crafted relations.

##### People In Rooms (PIR)

In each sample in the task, in one key sentence person is said to be located
in a named room (“ _X is in the old library_ "), and the other key sentence
describes the room to have a certain property (“the old library has wooden
floors”). The task is then to infer whether the given person is located in a
room with the given property.

PIR Example:
John’s living room is marble-floored, a reality that is as intrinsic to the
building as its very foundations. The moment  ...
Ethan Washington is in John’s living room, a fact that has become as much a
part of the place as the walls and the ceiling. The truth that Ethan
Washington is in John’s living ...
Is Ethan Washington in a marble-floored room?

This dataset is inspired by the bAbI set of tasks (Weston et al., 2016), where
reasoning is conducted on paths taken by one or more agents. PIR is a
simplification of the task, involving just one agent. The names of people in
the task are drawn randomly (Faraglia and Contributors, 2012). Rooms and
properties were hand selected to be mutually exclusive (for example, a room is
either blue-walled or red-walled), so no ambiguous examples are created.

##### Simplified Ruletaker

We employ the task formulation from Ruletaker Clark et al. (2021), a benchmark
designed for theorem proving within texts that present explicit logical
theories in natural language. Each instance consists of a logical rule, two
sentences each introcuding a fact, and a question over the rule and
facts.555Initial experiments revealed that most LLMs still struggle with
instances involving multiple rules or more than two facts. Our Simplified
Ruletaker task consists of generated samples that fit these criteria.

Simplified Ruletaker Example:
Facts:
Erin is furry. Erin is known for his furriness. He has a lot of fur and ...
Erin is good. Erin was always known for how good he is. His goodness appears
on all matters of life ...
Rule:If X is big and X is good then X is tall.
Question: can the statement "Erin is tall" be derived from the rule and the
facts?

###  3.3 Length Variations

We expand each base instance to input lengths of roughly 250, 500, 1000, 2000,
and 3000 tokens.666We consider a sample to be of length N if its token count
as measured by the GPT4 tokenizer is in $(N-70,N+70)$. To extend the inputs to
those targets we add background text that is irrelevant to the question
(“padding”, §2). For each basic-instance and length pair we create different
versions that differ in their source of background text: either _duplicate_ ,
_similar_ or _different_ than the key paragraphs of the instance. For each of
these, we also vary the dispersion of the key-paragraph within the background
text.

####  3.3.1 Background Texts

##### Duplicate.

To evaluate the extreme case where the length changes but the information
remains the same, we perform an experiment where the each length text consists
of multiple copies of the key paragraph. We duplicate each key paragraphs
without any modification to achieve the target length of the input. The two
duplicated paragraphs appear in alternating order until the desired sample
length is achieved. In this case, of the two sub-tasks of QA reasoning -
identifying the key information and reasoning over it, the first sub-task is
trivial.

##### Similar: resampling from the same task.

To get background text that is similar to the key paragraphs, we pad using
paragraphs sampled from other base instances of the same task. To avoid
creating contradictions, we exclude paragraphs that contain entities appearing
in the key paragraphs. This padding therefore does not produce adversarial or
ambiguous versions of the samples.

##### Different: Book Corpus.

To get background text that differs from the key paragraphs, we use text from
the Books Corpus (Zhu et al., 2015). We sample a random (continuous) text from
the Book Corpus, and inject each of the key paragraphs within it, while
respecting sentence boundaries.

Figure 2: Inputs construction. Key sentences (dark red), are expanded to key
paragraphs (light red) which are dispersed in controlled locations among
padding text (grey) which is irrelevant to the task.

####  3.3.2 Location of key paragraphs in the text

We consider four distinct ways in which the key paragraphs are dispersed
within the background text: in the first three cases the key paragraphs appear
adjacent to each other, while in the fourth the key paragraphs are separated
by intervening text of various lengths.

(1) Key paragraphs first: The key paragraphs appear at the beginning of the
text followed by padding;

(2) Key paragraphs middle: Half of the padding is affixed before and half
after the key paragraphs, but not between them (the key paragraphs are exactly
in the middle);

(3) Key paragraphs last: The key paragraphs appear at the end of the text,
with padding prepended before them as a prefix;

(4) Random placement: padding is added before, between and after the
paragraphs, with random intervals.

A visual representation is provided in Figure 2.

###  3.4 Base instances are answerable

We estimate the baseline accuracy by evaluating the LLMs on the minimal text
of each sample in the dataset that includes only the question and the key
paragraphs relevant to it. We found that even when using non-CoT prompting,
four out of the five models achieve high accuracy (>0.89). The lowest
performing model (GPT3.5) achieve high enough accuracy for degradation to be
observable (0.77). Full results can be found in Appendix C.

##  4 Main Experiments

We report average accuracies over all three tasks, and maintain the same setup
(prompt, temperature, etc.) over all input lengths. We evaluate five recent
capable LLMs: GPT4, GPT3.5, Gemini-Pro, Mistral 70B and Mixtral 8x7B. See
Appendix E for a detailed breakdown of our setup parameters.

###  4.1 Impact of Length and Location

We start by validating the impact of input length on LLM reasoning performance
(Figure 1) in various experimental settings.

##### No irrelevant paragraphs

We first look into the extreme case where only relevant tokens are added
(“duplicate padding”). Shi et al. (2023) Demonstrate that appending irrelevant
texts to the input of a reasoning task (GSM-8K Cobbe et al. (2021)) reduces
model performance substantially. We isolate the effect of relevance by testing
a setting in which the padding is duplications of the exact text of the key
paragraphs. In this setup, the LLMs are not required to “search" the input to
find the key paragraphs, so any bias towards any position becomes irrelevant
Liu et al. (2023b). Also, any difficulty that might be imposed by the distance
between the key paragraphs also becomes irrelevant. Hence, we expect that
there will be no degradation in performance. Surprisingly, the _Results_ shown
in Figure 3, reveal that even in this setup length does play a factor, _and
accuracy decreases with length for all models_.

Figure 3:  The relevance of padding is a factor, but it is distinct from the
effect of length itself. Some models degrade in reasoning performance. Note,
both GPT3.5 and GPT4 are less affected by length when the added tokens are
relevant. Each point reflects 300 samples.

##### Adjacent paragraphs surrounded by irrelevant ones

We now move to the more realistic case where the prompt includes the key
paragraphs as well as additional irrelevant ones. In the first set of
experiments, we keep the key paragraphs adjacent to each other: the LLM just
needs to focus and operate on a single area of the input, ignoring the rest.
Liu et al. (2023b) Found that in the task of extractive QA, the position of
the answer in the text affects the ability of models to answer correctly. We
thus experiment with the three scenarios: positioning both key paragraphs at
the start, end or middle of the text. In all cases we average over both types
of irrelevant padding.

Figure 4: Effect of key paragraphs positions on accuracy. Accuracy decreases
as input length grows regardless of where the key paragraphs are placed within
the input. Each point reflects 300 samples. Results for all models appear in
Appendix E

_The results_ in Figure 4 show a significant drop in accuracy as length
increase beyond 500 tokens. For most models, adjacency of key paragraphs
produces higher accuracy, and when the key paragraphs appear last, accuracy is
often highest (suggesting recency bias).

##### Non-adjacent relevant paragraphs.

Finally, we test the scenario in which the relevant facts needs to be
collected from two non-adjacent locations within the text.

Here, _the results_ in Figure 1 show a very large drop in performance as
length increases, indicating that reasoning tasks becomes significantly harder
for LLMs when they need to collect evidence from two distinct locations in a
large-ish context length.

###  4.2 Kind of irrelevant material

We now focus only on the non-adjacent key-paragraphs case, and explore the
effect of the kind of irrelevant text. We consider two scenarios: when the
irrelevant paragraphs are _similar_ to the relevant ones (taken from the same
task), and when they are _different_ (taken from the books corpus).

Figure 5: Performance degrade in both types of padding. Books padding impact
is much greater in most models. Each point reflects the performance across 300
samples.

Our initial expectation was that the setup in which the irrelevant paragraphs
are _different_ from the relevant ones will be easier for the model, as the
irrelevant paragraphs will be easier to discard, aiding focusing on the
relevant ones. However, the results (Figure 5) show the that is not case: the
drop for the _different_ setup is mostly larger than for the _similar_ one.

##  5 Correlation with Next Word Prediction

Perplexity is used as the main benchmark to show that models utilize their
entire input Anil et al. (2023); Jiang et al. (2024). However, it was shown
that performance on downstream tasks does not necessarily correlate with model
perplexity (Liu et al., 2023a; Xia et al., 2022; Tay et al., 2022). Here, we
will use the flexibility of our dataset to understand the correlation between
perplexity and reasoning accuracy.

In closed models we lack access to full vocabulary token probabilities so
model perplexity cannot be measured, therefore we resort to measuring next
word accuracy on our data. We prompt models to complete the next word in a
given text, and the output is correct if it is an exact match to the true next
word. We use the samples in our dataset (without the questions) as the text
and compare the results to the reasoning performance on the same samples.

Our method finds similar trends on the next word prediction task to those
shown in other works Anil et al. (2023); Jiang et al. (2024), namely accuracy
increases as input is longer. However, as shown in Figure 1, next word
accuracy correlates negatively with reasoning on FlenQA
777$\rho_{Pearson}=-0.95$, $p=0.01$.

Figure 6: Next word accuracy correlates negatively with the reasoning accuracy
on FlenQA. Each point reflects the performance across 300 samples. Gemini-Pro
is not included as it answered empty replies to the next word prediction task
at any length.

This implies that measuring next word prediction and, similarly, perplexity,
cannot substitute downstream task evaluation on long inputs.

##  6 Does Chain of Thought Help?

Chain of Thought (CoT) prompting, introduced by Kojima et al. (2022); Wei et
al. (2022), is a technique by which the LLM is pushed to produce a text
comprising of reasoning steps before concluding the correct answer for a
question. Zhou et al. (2022) found that a more specific and optimised
instruction ("Let’s work this out in a step by step way to be sure we have the
right answer.").

The CoT technique was shown to significantly improve the accuracy on many
reasoning-based question-answering setups. Will using it change the trend and
allow the LLMs to perform effectively on longer inputs? We experiment with CoT
using the elicitation string of Zhou et al. (2022).

The results show (Figure 1) that CoT has different effects on different LLMs,
and overall does not mitigate the drop in performance due to length. In most
cases (GPT4, Mixtral 8x7B, Mistral 70B and GPT3.5) it improves performance,
but only in GPT4 it has an increased effect as length increases, making it a
limited mitigation technique. In the case of Gemini-Pro, we see that CoT
decrease performance as input length is increased, even though it increase
performance on short length.

The full results of the CoT prompting over all tasks and setups can be found
in Appendix E.

##  7 Length-induced Failure modes

We find in the results four _failure modes_ :888All failure modes can be
measured automatically using the code in our repository. consistent patterns
that correlate with incorrect responses.

##### Failure to answer

All of the samples in the dataset can be answered with either "True" or
"False", as instructed in our prompts (Appendix D). However, some of LLMs
responded with a refusal answer the question, often preceded by a sentence
such as "There is not enough information in the text". This tendency grows as
the input length increases, indicating a failure to comply to the instruction
that specified a clear choice between "True" and "False". The trend is
demonstrated in figure 7, and results over all models in Appendix 16.

Figure 7: The models exhibit two types of input-length dependent biases: (a)
They tend to generate "False" more often than "True", and (b) they ignore
instructions and generate answers which do not contain neither.

##### Label bias

As discussed in §3, our dataset is completely balanced in the label
distribution. We find that certain LLMs tend to favour one of the labels,
typically "false", as the input length grows. Results for all models are in
Appendix 16.

##### Answer first, reason later

When using Chain-of-Thought prompting, some LLMs were much more likely to
output the final true/false answer _before_ the expected reasoning steps, as
inputs grow longer. In recent work, Kojima et al. 2022 found that when models
are elicited to provide the reasoning steps after the answer their performance
does not increase (as expected when using auto-regressive models that only
attend to earlier tokens). This can be viewed as a case of failing to follow
prompt instructions (see prompt instructions in Appendix D) as the length
increases. In testing, we found that incorrect responses are statistically
dependent on the occurrence of answers before the reasoning
steps999Corresponding odds-ratio is 3.643 with $p<0.001$ obtained through
Fisher exact test..

Figure 8: Most of the models tend to generate an answer before the reasoning
steps, in a zero-shot CoT prompt setting, and do so more as input length
increases.

##### Chain-of-Thought lack of coverage

All the tasks in FlenQA require the LLM to: (1) locate the relevant texts
within the input; and (2) perform the relevant reasoning over them. Ideally,
the CoT prompt would elicit the LLM to first locate each of the relevant texts
and copy them to the “steps” part, hence avoiding the effect of long input on
reasoning. However, we find that as input length grows, LLMs ability to do
this degrades (Figure 9).

Figure 9: CoT coverage of relevant facts. As input grows, all models fail more
often in outputting the task-relevant information at the CoT reasoning steps
stage.

We measure this by computing the coverage of the relevant text (the key
sentences in each sample) in the models’ “steps" part of the outputs (details
in Appendix D.4). We find that in most models, the ability to locate the
relevant text within the input _decreases_ as the input length gets longer. We
found incorrect responses were statistically dependent on the incomplete
coverage of the facts101010Corresponding odds-ratio is 3.138 with $p<0.001$
obtained through Fisher exact test..

##  8 Related Work

The evaluation of LLMs on long inputs has followed two distinct pathways:
benchmarks of downstream tasks and next word prediction. In the realm of
benchmarks, studies proposed datasets of long input samples that can be used
to evaluate models Shaham et al. (2023, 2022); An et al. (2023b, a). Those
datasets are curated over inputs of different, but fixed, length. This
approach, while straightforward, limits our ability to inputs of varying
lengths, posing a challenge in understanding the true impact of input length
on model performance. On the other hand, next word prediction evaluations do
offer an insights into how models handle inputs of different lengths (like
done in Anil et al. 2023; Jiang et al. 2024). However, the correlation of this
task with downstream performance was found not consistent (Liu et al., 2023a;
Xia et al., 2022; Tay et al., 2022). In this paper we reproduce this finding
with respect to extended length.

This study builds upon prior research that examined different aspects through
input intervention, studying the semantic content (theme) of a task (Dasgupta
et al., 2022), prompting strategies (Kojima et al., 2022; Yao et al., 2023;
Jin et al., 2024) and various properties of the QA task (Levy et al., 2023).
Our investigation focuses on input length, isolating it, to reveal its impact
on performance.

##  9 Discussion

We study the effect of input length on reasoning performance of current Large
Language Models (LLMs). Our findings reveal a significant drop in performance
with longer inputs, occurring well before reaching the models’ maximum input-
length capacity. Our experiments relied on FLenQA, a dataset we constructed
that allows to isolate the length factor, by adjusting the parts in the input
that are irrelevant to the task. We show that regardless of how we adjust the
samples, there is still a strong effect of length on reasoning performance.

Finally, we identified specific failure modes, including difficulties in
following extended instructions and biases towards less relevant information.
Our analysis reveals specific failings, providing possible directions for
future studies to address and rectify the weaknesses found in LLMs.

In conclusion, our work indicates that evaluating a model’s performance based
on a single input length does not provide a full picture, and more nuanced
evaluation is required. We argue that for a model to be considered capable at
long range, it must maintain its performance at any length it technically
supports.

## Limitations

Because of the nature of behavioral testing, the observed drop in performance
with varying input lengths remains unexplained; because of lack of access to
many of the models, we suspect this direction will continue to be limited.
Secondly, our approach aimed to create a universally applicable test across
different LLMs, leading to the selection of tasks that cater to the lowest
common denominator. This approach potentially overlooks the nuanced
performance differences in more complex reasoning tasks (e.g 5 key
paragraphs), where, for instance, stronger models might exhibit performance
degradation at shorter input lengths compared to what our findings suggest.
Additionally, we focused on a subset of reasoning task types which may differ
behaviourally from other types. Finally, our study did not test the distance
between key paragraphs, leaving an aspect of LLM performance unexplored that
we leave for future research.

## Acknowledgements

## References

  * An et al. (2023a) Chen An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023a.  L-eval: Instituting standardized evaluation for long context language models.  _ArXiv_ , abs/2307.11088.
  * An et al. (2023b) Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023b.  L-eval: Instituting standardized evaluation for long context language models.
  * Anil et al. (2023) Rohan Anil, Sebastian Borgeaud, Yonghui Wu, and Gemini Team Google. 2023.  Gemini: A family of highly capable multimodal models.  _ArXiv_ , abs/2312.11805.
  * Bai et al. (2023) Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023.  Longbench: A bilingual, multitask benchmark for long context understanding.  _arXiv preprint arXiv:2308.14508_.
  * Chen and Durrett (2019) Jifan Chen and Greg Durrett. 2019.  Understanding dataset design choices for multi-hop reasoning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
  * Clark et al. (2021) Peter Clark, Oyvind Tafjord, and Kyle Richardson. 2021.  Transformers as soft reasoners over language.  In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , pages 3882–3890.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Dasgupta et al. (2022) Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2022.  Language models show human-like content effects on reasoning.  _arXiv preprint arXiv:2207.07051_.
  * Faraglia and Contributors (2012) Daniele Faraglia and Other Contributors. 2012.  Faker.
  * Gidiotis and Tsoumakas (2020) Alexios Gidiotis and Grigorios Tsoumakas. 2020.  A divide-and-conquer approach to the summarization of long documents.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:3029–3040.
  * Holtzman et al. (2023) Ari Holtzman, Peter West, and Luke Zettlemoyer. 2023.  Generative models as a complex systems science: How can we make sense of large language model behavior?  _arXiv preprint arXiv:2308.00189_.
  * Huang and Chang (2022) Jie Huang and Kevin Chen-Chuan Chang. 2022.  Towards reasoning in large language models: A survey.  _arXiv preprint arXiv:2212.10403_.
  * Jacovi et al. (2023) Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023.  Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5075–5084, Singapore. Association for Computational Linguistics.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of experts.
  * Jin et al. (2024) Mingyu Jin, Qinkai Yu, Haiyan Zhao, Wenyue Hua, Yanda Meng, Yongfeng Zhang, Mengnan Du, et al. 2024.  The impact of reasoning step length on large language models.  _arXiv preprint arXiv:2401.04925_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Levy et al. (2023) Mosh Levy, Shauli Ravfogel, and Yoav Goldberg. 2023.  Guiding llm to fool itself: Automatically manipulating machine reading comprehension shortcut triggers.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8495–8505.
  * Li et al. (2023) Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2023.  Loogle: Can long-context language models understand long contexts? _ArXiv_ , abs/2311.04939.
  * Liu et al. (2023a) Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. 2023a.  Same pre-training loss, better downstream: Implicit bias matters for language models.  In _International Conference on Machine Learning_ , pages 22188–22214. PMLR.
  * Liu et al. (2023b) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023b.  Lost in the middle: How language models use long contexts.  _arXiv preprint arXiv:2307.03172_.
  * Liu et al. (2022) Yang Liu, Chenguang Zhu, and Michael Zeng. 2022.  End-to-end segmentation-based news summarization.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 544–554.
  * Min et al. (2019) Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019.  Compositional questions do not necessitate multi-hop reasoning.  In _Annual Meeting of the Association for Computational Linguistics_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Sainz et al. (2023) Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023.  Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10776–10787.
  * Shaham et al. (2023) Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023.  Zeroscrolls: A zero-shot benchmark for long text understanding.  _arXiv preprint arXiv:2305.14196_.
  * Shaham et al. (2022) Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022.  Scrolls: Standardized comparison over long language sequences.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 12007–12021.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023.  Large language models can be easily distracted by irrelevant context.  In _International Conference on Machine Learning_ , pages 31210–31227. PMLR.
  * Sinha et al. (2018) Koustuv Sinha, Shagun Sodhani, William L. Hamilton, and Joelle Pineau. 2018.  Compositional language understanding with text-based relational reasoning.  _ArXiv_ , abs/1811.02959.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. 2022.  Scaling laws vs model architectures: How does inductive bias influence scaling?  _arXiv preprint arXiv:2207.10551_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Weston et al. (2016) Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016.  Towards ai-complete question answering: A set of prerequisite toy tasks.  In _4th International Conference on Learning Representations, ICLR 2016_.
  * Wolhandler et al. (2022) Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Dagan. 2022.  How “multi” is multi-document summarization?  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5761–5769.
  * Xia et al. (2022) Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2022.  Training trajectories of language models across scales.  _arXiv preprint arXiv:2212.09803_.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Zhou et al. (2022) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022.  Large language models are human-level prompt engineers.  _arXiv preprint arXiv:2211.01910_.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _The IEEE International Conference on Computer Vision (ICCV)_.

##  Appendix A Training Contamination in Reasoning Tasks

Data contamination is a major concern when evaluating models (Sainz et al.,
2023; Jacovi et al., 2023). Ensuring that a task requires reasoning across
multiple text spans is a stronger requirement then a task that requires multi
hop reasoning (§2). Evaluating models on questions they answer using
parametric knowledge prevents us from assessing their reasoning capabilities.
Datasets originating from internet sources are especially vulnerable to
contamination, thereby undermining the evaluation of a model’s capacity to
reason over novel facts.

Furthermore, it was shown that small models can answer existing reasoning
dataset when given one some of the parts that were claimed to be required for
the task (Chen and Durrett, 2019; Min et al., 2019). This raises question on
how well those dataset can be used to evaluate the effect of length, where we
require that the reasoning will be done on multiple parts of the text.

We conclude that parametric knowledge should be accounted for when evaluating
text-based reasoning capabilities. In this work we introduced FlenQA, which is
composed of novel generated data to make sure that reasoning over the input is
required.

##  Appendix B Datasets

Each task of the following contains 100 base instances. In each sample, there
are two paragraph-length texts (key paragraphs). To achieve paragraphs of
similar length, we edit them by truncating sentences beyond a specific length,
resulting in an average paragraph length of 125 tokens.

###  B.1 Ruletaker

The key paragraphs in the task are as evidence for the reasoning task, a rule
and a question. In the original data Clark et al. (2021), the samples contain
different number of reasoning steps. In this study, we generate new, simpler
samples of the task: each sample is composed of only two facts and one logical
rule. The samples we generate are of similar flavor to those that exist in the
original Ruletaker data, but are generated with new statements, rules and
facts. The key paragraphs and the padding apear as the facts of each sample.

Padding | Target Input | Mean Number
---|---|---
Type | Length | Tokens
Books | 250 | 249.8
500 | 508.78
1000 | 1009.56
2000 | 2009.64
3000 | 3008.38
Same | 250 | 249.8
500 | 503.535
1000 | 1004.41
2000 | 2005.51
3000 | 3005.125
Figure 10: Summary of statistics of the Ruletaker* task data.

###  B.2 MonoRel

The key paragraphs in the task act as evidence for the reasoning task, and a
question. Both key paragraphs describe a monotonic relation between two
people, where one person is shared between both. The key paragraphs are
embedded in padding text to create a text mixture.

Padding | Target Input | Mean Number
---|---|---
Type | Length | Tokens
Books | 250 | 238.06
500 | 490.84
1000 | 991.41
2000 | 1990.34
3000 | 2990.95
Same | 250 | 238.06
500 | 491.69
1000 | 991.43
2000 | 1991.31
3000 | 2991.44
Figure 11: Summary of statistics of the MonoRel task data.

###  B.3 People in Rooms (PIR)

One key paragraph describes the location of an individual, and the other
describes some attribute of that location. The key paragraphs are embedded in
padding text to create a text mixture.

Padding | Target Input | Mean Number
---|---|---
Type | Length | Tokens
Books | 250 | 305.36
500 | 491.85
1000 | 989.91
2000 | 1992.00
3000 | 2988.67
Same | 250 | 305.36
500 | 484.63
1000 | 985.82
2000 | 1985.04
3000 | 2984.80
Figure 12: Summary of statistics of the People In Rooms (PIR) task data.

##  Appendix C Base Instances Result

Model | Prompt | MonoRel | PIR | Ruletaker*
---|---|---|---|---
GPT3.5 Turbo | Direct | 0.77 | 0.81 | 0.74
CoT | 0.86 | 0.88 | 0.88
GPT4 Turbo | Direct | 1.00 | 1.00 | 0.98
CoT | 1.00 | 1.00 | 0.97
Gemini Pro | Direct | 0.84 | 1.00 | 0.92
CoT | 0.88 | 0.96 | 0.97
Mistral 70B | Direct | 0.99 | 1.00 | 0.73
CoT | 1.00 | 1.00 | 0.89
Mixtral 8x7B | Direct | 0.92 | 0.97 | 0.80
CoT | 0.86 | 0.97 | 0.93
Table 1: Minimal length accuracy. The evaluated models have high accuracy on
the tasks in our dataset when evaluated on the minimal text (250 tokens). CoT
improve performance across almost all tasks and models.

##  Appendix D Full Evaluation Setup

###  D.1 Prompts

Ruletaker prompt - Normal:
Answer whether the statement {statement} can be derived from the rule and the
facts. Answer with either "True" or "False".
Rule: {rule}
Facts: {facts + padding}
Answer with either "True or "False".

Ruletaker prompt - CoT:
Answer whether the statement {statement} can be derived from the rule and the
facts.
Show your steps then answer with either "True" or "False".
Rule: {rule}
Facts: {facts + padding}
Answer with either "True or "False". Let’s work this out in a step by step way
to be sure we have the right answer.

PIR prompt - Normal:
{facts + padding}
True/False Question: {question}
Answer only True or False.

PIR prompt - CoT:
Show your steps then answer with ’true’ or ’false’.
{facts + padding}
True/False Question: {question}
Let’s work this out in a step by step way to be sure we have the right answer.

MonoRel prompt - Normal:
Here are some facts. Answer the exact following question based on the text:
{question} Answer the question as it appears exactly. {facts + padding}
{question}
Answer only True or False.

MonoRel prompt - CoT:
Here are some facts. Answer the exact following question based on the text:
{question} Answer the question as it appears exactly.
Show your steps then answer with ’true’ or ’false’.
{facts + padding}
{question}
Let’s work this out in a step by step way to be sure we have the right answer.
Show your work and finally answer with ’true’ or ’false’. The final step
should include the exact text of the question and the answer.

###  D.2 Parameters

All models were evaluated with a temperature of 0 and "top p" of 0 where
available to make results as reproducible as possible. Additionally, We
configured Gemini Pro to ignore safety guardrails ("HARM_CATEGORY"
configurations) to overcome its blank answers in some samples.

###  D.3 Locating the answer in models’ replies

To identify the models’ answers in their responses, we searched for the
occurrences of "false" or "true," disregarding case sensitivity. In cases
where these words appeared multiple times, only the last instance was
considered relevant. We tested the reliability of this method by manually
examining a random sample of 100 responses and confirmed its accuracy in all
instances.

###  D.4 Evaluating the coverage of key facts in CoT

Coverage of the key facts that are relevant to the reasoning task in CoT
outputs, was done by searching for (case-insensitive) match of the key
sentences in the key paragraphs, within the output of each model. Full
coverage means that both key sentences from the input appear in the CoT
output. We verified the reliability of this method manually on a sample of 100
responses.

##  Appendix E Full results

Figure 13: Full results for the Ruletaker dataset. Figure 14: Full results for
the MonoRel dataset. Figure 15: Full results for the People In Rooms (PIR)
dataset. Figure 16: Differences in accuracy between different positions of key
paragraphs in input. Averaged over both types of irrelevant padding: similar
(resampling from the data) and dissimilar (Books corpus) padding. Figure 17:
Biases in answer generation and non-answers. Frequency of responses with True,
False, or neither, per model.