# Imaging cloaked objects: diffraction tomography of realistic invisibility
devices

Francisco J. Díaz-Fernández<EMAIL_ADDRESS>Nanophotonics Technology
Center, Universitat Politècnica de València, 46022, Valencia, Spain  Javier
Martí  Nanophotonics Technology Center, Universitat Politècnica de València,
46022, Valencia, Spain  DAS Photonics S.L., 46022, Valencia, Spain  Carlos
García-Meca<EMAIL_ADDRESS>DAS Photonics S.L., 46022, Valencia,
Spain

###### Abstract

Invisibility cloaks have become one of the most outstanding developments among
the wide range of applications in the field of metamaterials. So far, most
efforts in invisibility science have been devoted to achieving practically
realizable cloak designs and to improving the effectiveness of these devices
in reducing their scattering cross-section (SCS), a scalar quantity accounting
for the total electromagnetic energy scattered by an object. In contrast,
little attention has been paid to the opposite side of the technology: the
development of more efficient techniques for the detection of invisibility
devices. For instance, the SCS ignores the phase change introduced by the
cloak, as well as the angular dependence of the incident and scattered waves.
Here, we propose to take advantage of the smarter way in which diffraction
tomography processes all this overlooked information to improve the efficiency
in unveiling the presence of invisibility devices. We show that this approach
not only results in a considerable sensitivity enhancement in the detection of
different kinds of cloaks based on both scattering cancellation and
transformation optics, but also enables us to even obtain images depicting the
approximate shape and size of the cloak. As a side result, we find that
scattering cancellation coatings may be used to enhance the contrast of
tomographic images of closely-packed particles. Our work could be easily
extended to the detection of sound cloaks.

##  I Introduction

Invisibility has been one of the most challenging effects pursued by humankind
for centuries. The possibility of hiding objects to the naked eye has recently
leaped from science fiction to a feasible reality thanks to the advent of
metamaterials [1]. Among a wide range of applications that arises through the
exploitation of this kind of materials not available in nature, invisibility
cloaks are one of the most high-impact developments [3, 2]. The impressive
ability of these devices to hide objects by reducing the scattering they
produce has even been experimentally demonstrated [4, 5], boosting the impact
of this field of study over the last two decades. Different approaches to the
achievement of invisibility have followed in different fields besides optics.
For instance, a variety of cloaking devices have been experimentally
demonstrated also in acoustics, thermodynamics and mechanics [6].

However, cloaking devices are not perfect. It has been shown that outside the
design frequency, realistic cloaks may become significantly visible [7].
Conversely, as technology improves, detecting a cloak at the design frequency
might be a challenging task. In most previous studies, the scattering cross
section (SCS) has been used as an indicator of the effectiveness of realistic
invisibility cloaks [8, 9, 7]. The SCS is a measurement that estimates the
total energy scattered by an object (see Appendix A). Therefore, the use of
this scalar value has the disadvantage that it does not take into account the
spatial distribution of the scattered field nor the phase changes produced by
the cloak and, in the case of devices without rotational symmetry, only a
single illumination direction is usually considered. It is thus reasonable to
imagine that a suitable measurement and processing of these missing data would
provide more information on the cloak, which, as predicted in [7], should be
more sensible to interferometric techniques than to the SCS. Actually, looking
back at the origins of invisibility cloaks, we find that one of the first
known invisibility devices was conceived as a material undetectable by
tomographic techniques [10], which aim at reconstructing the refractive index
(RI) profile of an object by illuminating it from each possible direction and
by smartly combining the complex amplitude (magnitude and phase) of the
resulting scattered waves [11]. Therefore, diffraction tomography (DT) is
expected to detect invisibility devices more accurately, and even has the
potential of revealing the shape of the cloak (see Fig. 1).

In this paper, we explore this possibility, keeping the spotlight on studying
the behavior of passive cloaks under DT around their design frequency, at
which the SCS measurement is less effective and may fail to unveil their
presence. Particularly, we analyze three of the most representative kinds of
invisibility devices: plasmonic cloaks based on scattering cancellation (SC),
blow-up-a-point inhomogeneous cloaks based on transformation optics (TO), and
homogeneous polygonal cloaks, also based on TO (see Fig. 2).

In all cases, we consider realistic material implementations. Additionally, it
is worth mentioning that DT is constrained by the diffraction limit, i.e., it
is suited for imaging objects with dimensions larger than $\lambda/2$, where
$\lambda$ is the wavelength of light, if both the forward and backward
scattering are measured. If the latter is not considered, which is the common
situation and the one we will consider in this work, the limit rises to
$\lambda/\sqrt{2}$ [12], so we will restrict our study to this class of
objects. Moreover, we will focus on two-dimensional (2D) problems for
simplicity, although the extension to the three-dimensional case is
straightforward. Likewise, we restrict ourselves to TE waves (electric field
polarized along the $z$ component and magnetic field contained in the $XY$
plane), with analogous results for TM waves.

Figure 1: Scheme of the difference between the SCS and DT detection
techniques. (a) The SCS is a scalar quantity related to the total scattered
power for a given incident direction. The phase of the scattered field is not
taken into account. (b) DT uses different angles of illumination to obtain a
RI map of the object from the space-dependent phase and amplitude of the
scattered field.

Finally, to implement the DT algorithms used in this paper, the steps
described in [12, 13] have been coded in Matlab. These robust algorithms have
been used, for example, to measure the refractive index of a single cell [14].

Figure 2: Studied invisibility cloaks and their working principle. (a) SC
cloak (b) Blow-up-a-point TO cloak. (c) Polygonal TO cloak.

##  II Scattering cancellation

The first approach to invisibility cloaking that we will study is scattering
cancellation. This technique reduces the scattering produced by a dielectric
object with a positive permittivity by covering it with a negative-
permittivity coating [see Fig. 2(a)] [8]. SC has also been applied to cloak
objects immersed in a uniform static magnetic field [15] or in a diffusive
light scattering medium [16]. An interesting application of this type of cloak
is the possibility of cloaking a sensor without affecting its capability to
measure an incoming signal [17]. Typically, the SC set-up consists of a core-
shell spherical or cylindrical structure in which the oppositely signed
permittivities of the inner and outer materials are designed to cancel the
first-order scattered field at a given frequency (the design frequency) [8].

Note that the SC effect is maximized for objects with electrically small
dimensions [8]. Therefore, taking into account the diffraction limit of DT, we
will analyze objects with longitudinal dimensions in the interval
$[\lambda_{0}/\sqrt{2},2\lambda_{0}]$, where $\lambda_{0}$ is the free-space
wavelength for which the cloaks are designed, with $f_{0}$ the corresponding
frequency. Moreover, to satisfy the Born and Rytov approximations usually
employed in DT [12, 18], the maximum contrast between the object RI and that
of the background is fixed to a 5%. In particular, we apply the SC technique
to cloak dielectric cylinders of radii $\lambda_{0}/2$ and $\lambda_{0}$, with
a RI $n_{\rm{OBJ}}=1.05$, and immersed in a vacuum background with RI
$n_{\rm{BG}}=1$. These cloaks have been optimized to minimize the SCS
following the procedure described in [19] (see Appendix A). This yields
optimum metallic shells with a permittivity $\epsilon_{\rm{S}}=-2.7$ and an
outer radius $r_{\rm{S}}=0.508\lambda$ for the $r=\lambda_{0}/2$ cylinder and
$r_{\rm{S}}=1.0205\lambda_{0}$ for the $r=\lambda_{0}$ cylinder. As mentioned
above, the electric field is assumed to be polarized along the $z$ component
(parallel to the cylinder axis). Figure 3 shows the distribution of this field
for both studied SC cloaks under plane wave incidence.

Figure 3: Simulated $z$ component of the electric field resulting from the
illumination of a SC-cloaked dielectric cylinder of radius $r$ with an
$x$-directed plane wave. (a) $r=\lambda_{0}/2$. (b) $r=\lambda_{0}$.

To quantify the effectiveness of the cloak in terms of the SCS reduction it
attains, we define the relative observability as:

$O_{\rm SCS}=\frac{\rm SCS_{C}}{\rm SCS_{NC}},$ (1)

where $\rm SCS_{C}$ and $\rm SCS_{NC}$ are the SCS of the object with and
without cloak, respectively. That is, the lower the value of $O_{\rm SCS}$,
the higher the efficiency of the cloak. In the case of the $r=\lambda_{0}/2$
cylinder, a value of $O_{\rm SCS}=0.7$ is achieved, while for the
$r=\lambda_{0}$ cylinder, $O_{\rm SCS}=0.66$.

On the other hand, to obtain tomographic images (called tomograms) of all the
cloaks studied in this work, the scattered field is recorded on a detector
line (this data set is called a projection) placed at a distance $l_{\rm{D}}$
from the set-up center, as shown in Fig. 1(b). In the case of the SC
cylinders, we set $l_{\rm{D}}=10\lambda_{0}$. Ideally, the detector line
should be infinite to capture the complete forward scattered wave. In
practice, in the simulations we use a finite line length $w_{\rm D}$ such that
the missed scattered field is negligible. In the case of the SC cylinders,
$w_{\rm D}=220\lambda_{0}$. A tomogram is built upon the information provided
by different projections corresponding to different values of the illumination
angle $\phi_{0}$ [see Fig. 1(b)]. To ensure a nearly perfect tomographic
quality, we take 250 projections and, at least, record the field of each
projection at 8 points per $\lambda$ (a total of 1760 points in this case).

The obtained tomograms for the non-cloaked cylinders are shown in Fig. 4,
yielding a good approximation to the original cylinder RI profiles. It is
worth mentioning that the reconstructed RI has a smooth variation instead of
the abrupt RI change of the original cylinders, as we can see in Fig. 4(b) and
Fig. 4(d). These results are in line with those expected for the tomography of
a centered cylinder [18]. Notably, the RI recovered by the tomogram for the
cloaked $r=\lambda_{0}/2$ configuration [Fig. 4(b)] exhibits a higher
amplitude variation than that of the non-cloaked configuration (0.1 with
cloak, 0.06 without it). This increment of the RI range is more evident for
the $r=\lambda_{0}$ cylinder, as can be seen in Fig. 4(d). Additionally, in
this case, the RI profile shows two lobes instead of one. Regardless, the
tomograms clearly show the presence of an object in both cases.

Figure 4: Linear tomography (Rytov approximation) of non-cloaked and cloaked
dielectric cylinders of radius $r=\lambda_{0}/2$ (a,b) and $r=\lambda_{0}$
(c,d). Full tomograms are shown in panels (a) and (c). Panels (b) and (d) show
a detail of the RI along the dashed lines depicted in (a) and (c).

Following the work in Ref. [7], it is worth analyzing the behavior of the
cloak under DT at different frequencies. For that, the cloak was considered to
be made of lossless silver [7], and was modeled by a Drude permittivity
$\epsilon_{\rm{s}}=\epsilon_{\infty}-f_{p}^{2}/f^{2}$, with a plasma frequency
$f_{p}=2175$ THz and $\epsilon_{\infty}=5$ [20]. As a consequence, the ideal
cloak permittivity is obtained at $f_{0}=783.8$ THz (i.e., $\lambda_{0}=382.7$
nm). To quantify the sensitivity of DT in the detection of a given cloak, we
define the presence of an object according to its tomogram as:

$\displaystyle
p=\dfrac{\sum_{i=1}^{I}\sum_{j=1}^{J}\left|n_{\rm{DT}}(i,j)-n_{\rm{BG}}\right|}{IJ},$
(2)

where $n_{\rm{DT}}(i,j)$ is the RI at the pixel $(i,j)$ of a tomogram of size
$I\times J$. This is similar to the way in which the error of tomographic
techniques is tested [18]. As in the case of the SCS, the effect of the cloak
is quantified as the presence of the cloaked object divided by the presence of
the non-cloaked one, i.e.:

$O_{\rm DT}=\frac{p_{\rm C}}{p_{\rm NC}}.$ (3)

The detection ability of both measurement methods (SCS and DT) is shown in
Fig. 5(a) for the $r=\lambda_{0}/2$ cylinder. The corresponding tomogram
profiles along a line passing through the center of the cloak are shown in
Fig. 5(b) for some selected frequencies.

Figure 5: (a) Comparison between the relative observabilities obtained via SCS
and DT for an SC-cloaked cylinder with $r=\lambda_{0}/2$ and
$n_{\rm{OBJ}}=1.05$ as a function of frequency. (b) Corresponding RI along a
line passing through the center of the cylinder as retrieved via DT for some
selected frequencies [highlighted with colored points in (a)].

As expected, $O_{\rm SCS}$ shows a local minimum close to $f=f_{0}$, since
$f_{0}$ is the design frequency. As noted in Ref. [7], there is a frequency
range around $f_{0}$ for which the cloak reduces the scattering produced by
the concealed object (that is, $O_{\rm SCS}<1$ or, equivalently, $O_{\rm
SCS}<0$ dB), while the SCS is higher for the cloaked object in the rest of the
spectrum ($f<0.8f_{0}$ and $f>1.4f_{0}$ in our case), as can be seen in Fig.
5(a). This corresponds to the values greater than 0 dB in this figure, which
represent the frequencies at which the cloak not only fails at hiding the
object, but enhances its presence, making it more detectable. Remarkably, in
the eyes of DT, the presence of the cloaked system is almost always greater
than that of the non-cloaked one in the studied band, even at the frequencies
for which the cloak reduces the object SCS. Additionally, we have $O_{\rm
DT}\geq O_{\rm SCS}$, so we conclude that DT will be more effective in
discovering objects hidden by SC cloaks.

Besides allowing a presence measurement, DT provides information on the
apparent shape of the cloaked object. In the tomograms of the studied
cylinders, they appear to have a smaller radius and a larger RI variation than
their bare counterparts (Fig. 4). Consequently, we ask ourselves whether it
would be possible to take advantage of this apparent shrinking, which suggests
that covering small dielectric particles with SC coatings may facilitate their
differentiation when being closely packed. To study this possibility, we
analyze a new configuration consisting of two kissing cloaked cylinders
($r=\lambda$, $n_{\rm{OBJ}}=1.01$), immersed in a vacuum background
($n_{\rm{BG}}=1$) and covered by optimized shells with $\epsilon_{S}=-2$ and
$r_{S}=1.0049\lambda$. The simulation of the corresponding electric field is
shown in Fig. 6(a).

Figure 6: (a) Simulated $z$ component of the electric field resulting from the
illumination of two SC-cloaked cylinders ($r=\lambda$) with an $x$-directed
plane wave. (b) Tomograms of the non-cloaked and cloaked cylinders obtained
via DT (Rytov approximation). (c) Detail of the RI along the dashed lines
depicted in (b).

Tomograms of this cylinder arrangement are obtained both with and without SC
shells (we use 180 projections). Very similar results are obtained for the
Born and Rytov approximations. The tomograms for the latter are shown in Fig.
6(b), from which we observe that the bare cylinders are hardly
distinguishable, and can be mistaken for a single object [they appear as a
cylinder of radius $2\lambda$ along the central line, as can be seen in Fig.
6(c)]. On the contrary, the tomogram of the cloaked cylinders shows two
clearly separate profiles, as seen in Fig. 6(b), even though they are touching
each other, which is a remarkable feature. This fact, combined with the
enhanced range between the minimum and maximum RI values, provides a notable
advantage in the recognition of closely packed objects, which may find
application, for instance, in particle counting or bioimaging.

##  III Transformation optics - Blow-up-a-point cloaks

A more sophisticated passive cloaking technique that can remove scattering at
all orders is based on transformation optics [21, 2]. With this method, a
given region of space is hidden by redirecting the illuminating wave around
it, avoiding any light absorption or scattering. The rays traversing the cloak
bypass the concealed region and turn back to the original path. To study the
behavior of this kind of device under DT, we analyze the original 2D cloak
proposed in [4, 22], where a cylindrical region of radius $R_{1}$ is hidden by
a concentric cylindrical shell of radius $R_{2}$ [see Fig. 2(b)]. This cloak
requires the following radius-dependent, anisotropic relative permittivity
($\epsilon$) and permeability ($\mu$) tensor components (in cylindrical
coordinates):

$\displaystyle\epsilon_{r}=\mu_{r}=\dfrac{r-R_{1}}{r},$
$\displaystyle\epsilon_{\phi}=\mu_{\phi}=\dfrac{r}{r-R_{1}},$ (4)
$\displaystyle\epsilon_{z}=\mu_{z}=\left(\dfrac{R_{2}}{R_{2}-R_{1}}\right)^{2}\dfrac{r-R_{1}}{r}.$

To account for causality and losses, we analyze a realistic version of the
cloak in which the components with values below unity ($\epsilon_{r}$,
$\mu_{r}$, $\epsilon_{z}$, $\mu_{z}$) are modeled by Drude and Lorentz
dispersive relations, respectively [7, 23]:

$\displaystyle\epsilon_{r{\rm
D}}(\mathbf{r})=\epsilon_{r}(\mathbf{r})\left(2-\dfrac{f_{0}^{2}}{f(f+i\gamma_{1})}\right),$
(5) $\displaystyle\epsilon_{z{\rm
D}}(\mathbf{r})=\epsilon_{z}(\mathbf{r})\left(2-\dfrac{f_{0}^{2}}{f(f+i\gamma_{1})}\right),$
(6) $\displaystyle\mu_{r{\rm
L}}(\mathbf{r})=\mu_{r}(\mathbf{r})\left(1-\dfrac{F}{1+(i\gamma_{2}/f)-(f_{0}^{2}/f^{2})}\right),$
(7) $\displaystyle\mu_{z{\rm
L}}(\mathbf{r})=\mu_{z}(\mathbf{r})\left(1-\dfrac{F}{1+(i\gamma_{2}/f)-(f_{0}^{2}/f^{2})}\right).$
(8)

For the components larger than unity ($\epsilon_{\phi}$, $\mu_{\phi}$), we
assume a non-dispersive dependence [7, 23], given in our case by Eq. (III).
Following Ref. [23], we take $\gamma_{1}=\gamma_{2}=\gamma$ and $F=0.78$.

To validate this TO device, we simulate its response when cloaking a
dielectric disk of radius $r_{\rm D}=R_{1}$ and with a refractive index
$n_{\rm{OBJ}}=1.04$. A design frequency $f_{0}=193.55$ THz (corresponding to a
wavelength $\lambda_{0}=1.55$ $\mu$m) is chosen, and the dimensions of the
cloak are taken to be $R_{1}=2~{}\mu$m and $R_{2}=2R_{1}$. The simulation of
the electric field at $f_{0}$ for this configuration is shown in Fig. 7.

Figure 7: Simulated $z$ component of the electric field resulting from the
illumination of a dielectric cylinder covered by a lossless dispersive TO
cloak with an $x$-directed plane wave.

To compare the detection sensibility of the SCS and DT approaches, we study
the frequency range from $0.98f_{0}$ to $1.016f_{0}$, initially considering a
lossless material ($\gamma=0$). We take 1000 projections over a detector with
a length $w_{\rm D}=100\lambda_{0}$ and sample the field of each projection at
1000 points. The tomogram of the uncloaked object is depicted in Fig. 8(b),
which, as seen in Fig. 8(c), exhibits the typical ripple that arises when
tomography is applied over a large circular object [18]. In line with the
results reported in [23] and [7], Fig. 8 shows that the curve representing
$O_{\rm SCS}$ as a function of frequency has a V shape, with a spectral region
around $f_{0}$ where the SCS of the cloaked configurations is lower than that
produced by the bare object. Beyond these limits, the cloaked object scatters
more energy than the bare one [values greater than 0 dB in Fig. 8(a)].
Although the device was designed to have ideal parameters at $f_{0}$, the
minimum of $O_{\rm SCS}$ is achieved at $f=0.998f_{0}$. This slight change may
be due to the numerical error introduced by the simulation software.

The relative observability obtained with DT is also depicted in Fig. 8(a),
showing a minimum approximately at $f_{0}$ and the same trend as the $O_{\rm
SCS}$ curve. However, the mentioned low observability region found in the SCS
analysis is considerably narrowed when using DT for both the Born and the
Rytov approximations. Moreover, $O_{\rm DT}>O_{\rm SCS}$ in almost all the
analyzed spectrum (especially a frequencies below $f_{0}$), confirming that DT
is more sensitive to lossless realistic cloaks than SCS measurements.

Figure 8: (a) Comparison between the relative observability of a TO-cloaked
dielectric cylinder obtained via SCS and DT. (b) Tomograms (Rytov
approximation) associated with the non-cloaked cylinder and the cloaked
cylinder for different frequencies [corresponding to the color dots in (a)].
The size of the tomograms is $16\lambda_{0}\times 16\lambda_{0}$ (c) Detail of
the RI along the dashed lines depicted in (b).

To understand this behavior, the Rytov tomograms of the cloaked object at five
different frequencies [indicated by colored dots in Fig. 8(a)] are shown in
Fig. 8(b). The RI values of these tomograms along a line passing through the
center of the cloak are depicted in Fig. 8(c). The Born approximation gives
similar results and are not shown. The effect of the cloak in the scattered
field at the frequency for which $O_{\rm DT}$ reaches its lowest value [yellow
line in Fig. 8(c)] gives rise to a tomogram that corresponds to a thinner
object than the original one (as in the SC case) and with RI values below
$n_{\rm BG}$. As a consequence, while the presence of the cloaked object is
quite reduced with respect to that of the uncloaked object, this makes $O_{\rm
DT}>O_{\rm SCS}$ at $f_{0}$. As might be expected, the detection is more
precise at frequencies different from $f_{0}$. In particular, the tomograms
associated with the other four frequencies show a variation of the RI spanning
a circular region with approximately the same size as the cloak, thus
providing information on the shape of the device [see Figs. 8(b) and 8(c)]. In
addition to these variations, a concentric ripple of the RI is observed for
the frequencies that are farther from $f_{0}$, i.e., $f=0.989f_{0}$ (purple
line) and $f=1.012f_{0}$ (magenta line), which results in higher observability
values.

The Drude and Lorentz approximations used to model the cloak also allow us to
study the impact of losses in the detection. To this end, we relate $\gamma$
to the design frequency as follows:

$\displaystyle\gamma=10^{o}f_{0},$ (9)

where $o$ is termed the loss order. A high value of $o$ will be related to a
high value of loss. We sweep the value of $o$ from -3.5 to 1.5 (keeping the
frequency constant at $f_{0}$) and show the corresponding observability in
Fig. 9. For low losses, the results correspond to the values in Fig. 8 at
$f=f_{0}$. However, the SCS becomes more sensitive than DT for $o>-1.9$ (note
that this corresponds to a material with considerably high loss values, not
usually employed to construct cloaks). Interestingly, the maximum values of
the observability are achieved around $o=-1$. For $o>1$, the observability
associated with the SCS and DT methods has similar values.

Figure 9: Comparison between the relative observability obtained via SCS and
the Born and Rytov approxiamtions of DT as a function of the loss order.

##  IV Transformation optics - Polygonal cloaks

Figure 10: Simulated $z$ component of the electric field resulting from the
illumination of different polygonal TO cloaks with an $x$-directed plane wave.
(a) Hexagonal cloak illuminated at an angle of 0 rad (b) Hexagonal cloak
illuminated at an angle of $\pi/6$ rad. (c) Square cloak illuminated at an
angle of 0 rad. (d) Square cloak illuminated at an angle of $\pi/4$ rad.

The practical implementation of the ideal TO invisibility cloak studied in the
previous section is challenging due to the inhomogeneous character of the
$\epsilon$ and $\mu$ tensors, as well as to the extreme values of these
tensors near the inner limits of the cloak. One way to address this drawback
is to apply a piecewise affine transformation to a polygonal region divided
into segments, which results in a homogeneous device with finite constitutive
parameters [24]. A simplified version of this cloak can even be implemented
with natural birefringent crystals. Moreover, the study of this kind of device
allows us to test the DT detection ability for cloaks with non-cylindrical
shapes. In particular, here we consider a hexagonal cloak [the details of
which are shown in Fig. 2(c) as an example] and a square cloak, following the
transformations reported in [24] and [25], respectively. In the case of the
hexagonal cloak, the employed dimensions are $r_{2}=4.8\lambda$,
$r_{1}=r_{2}/3$, and $r_{0}=0.03\lambda$. The response of this cloak to a TE
plane wave is shown in Fig. 10(a,b) for two different angles of illumination
(0 rad and $\pi/6$ rad). As for the square cloak, the inner and outer square
boundaries have a side length of $2\lambda$ and $8.5\lambda$, respectively. A
simulation of its behavior for 0 rad and $\pi/4$ rad illumination angles are
shown in Fig. 10(c,d).

To analyze the behavior of the considered polygonal cloaks under the SCS and
DT paradigms, a hexagon and a square with a refractive index
$n_{\rm{OBJ}}=1.05$, have been placed inside the cloaks ($n_{\rm BG}=1$). In
the case of the hexagonal cloak, the hidden hexagonal object has a radius
$r_{1}$ and the hidden square has a side length $r_{1}$. In the case of the
square cloak, the hidden hexagon has a radius of $r=1.02\lambda$. Through this
study, we have verified that, although polygonal invisibility devices work
well for any angle of illumination, their performance has a slight angle
dependence by construction (the region of virtual space that is expanded is
not a point, but a finite domain without cylindrical symmetry). For instance,
when the hidden object is the hexagon, $O_{\rm SCS}$ varies from 0.22 in the
worst case (angle of illumination of $\pi/6$ rad) to 0.19 when the
illumination angle is 0 rad. The SCS ratios for the rest of the cases are
gathered in Table 1. In all configurations, the SCS is significantly reduced,
especially by the square cloak.

The same analysis is now repeated using DT under the Rytov approximation. The
tomograms of the two considered bare objects are shown in Fig. 11(a) and are
used as a reference. The tomograms of the different cloak-object combinations
can be seen in Fig. 11(b) and 11(c). The values of $O_{\rm DT}$ for each case
are shown in Table 1. As seen, the relative observability associated with the
DT approach is always significantly higher than that associated with the SCS
method, being higher than unity in some cases (meaning that the cloak enhances
the presence of the object in the eyes of tomography).

Table 1:  SCS and DT (Rytov) relative observability for each cloak-object combination. Cloak111H: Hexagonal, S: Square | Object | $O_{\rm SCS}$ | $O_{\rm DT}$
---|---|---|---
H | H | 0.22 - 0.19 | 0.74
H | S | 0.52 - 0.47 | 1.69
S | H | 0.40 - 0.18 | 1.51
S | S | 0.13 - 0.05 | 0.81
Figure 11: Tomograms (Rytov approximation) of: (a) a bare hexagon and a bare
square. (b) Each of these objects hidden within a hexagonal cloak and (c)
within a square cloak. The boundaries of the cloaks are depicted as a
reference. The schematic insets represent the cloak (pink) and the hidden
object (green) for each configuration.

In addition, the tomograms of Fig. 11 evidence the imaging ability of DT for
non-cylindrical cloaks. First, they show a clear correspondence between the
size of the retrieved index profile and the outer boundaries of both cloaks.
Second, the tomograms possess the same symmetries as the cloaks, as can be
seen in Fig. 11(b,c). Hence, although the hidden objects are not unmasked by
the tomograms, the latter provide important information on the cloaking
device, revealing its presence. This means that, while small changes in the
scattered energy as a function of the illumination angle do not compromise the
cloak performance in terms of SCS (with low values of $O_{\rm SCS}$ well below
unity in all cases), they are magnified by DT due to the smart combination of
the measured complex field at different angles. Therefore, the high values of
$O_{\rm DT}$, together with the ability of DT to determine the shape of the
cloak, makes this technique a much better candidate for the identification of
this kind of device than the SCS. Furthermore, the most remarkable fact is
that this identification has been done at the design frequency of the cloak,
i.e., for the exact ideal parameters of the device. It is for this reason that
we have not carried out a frequency sweep to analyze the spectral dependence
of the observability in this case, although one would expect an increment in
the values of this parameter as moving away from $f_{0}$.

##  V Conclusions

In this work, we have studied the potential offered by DT for the detection of
realistic invisibility devices. In this direction, we have shown that this
technique is more sensitive than the broadly used SCS for cloaks based on both
SC and TO around the design frequency of these devices. Moreover, DT can
approximately reveal the shape and the size of all the analyzed devices.

More specifically, regarding SC cloaks and blow-up-a-point TO cloaks, the
presence of the cloaked object has been compared to that of the uncloaked one
for different frequencies, finding that the band at which the system possesses
low observability according to an SCS meter ($O_{\rm SCS}<1$) is notably
reduced for the DT technique, the observability of the system being always
higher or equal under a DT analysis ($O_{\rm DT}\geq O_{\rm SCS}$). In
addition, while polygonal TO cloaks have been verified to do a good job in
hiding objects as far as the SCS parameter is concerned, DT is able to image
the shape of these cloaks and can therefore reveal their presence,
compromising their effectiveness. Remarkably, this occurs for the ideal cloak
parameters.

Since diffraction tomography represents a more demanding test for the
performance evaluation of invisibility cloaks, it might be useful as a new
standard in their design and characterization. Additionally, our findings open
up a range of potential benefits in a variety of fields, from fundamental
advances in invisibility research to bioimaging and warfare applications at
the technological level.

Besides, our results indicate that objects cloaked using SC appear to be
shrunk from a tomographic point of view. This suggests that the SC technique
could be applied to enhance the resolution of DT when imaging small closely-
packed particles.

To complete the discussion, we would like to make a note about the robustness
of the proposed method. Firstly, although we have employed a high number of
projections to obtain the tomograms in this work, a much lower number would be
enough in some cases. For instance, the reported polygonal cloak tomograms
were obtained using a number of projections ranging from 180 to 360. However,
the shape of the cloak can be distinguished even with only a few tens of
projections. Secondly, it is worth mentioning that, although the employed DT
approach is restricted to isotropic objects possessing a low index gradient,
our results prove that this technique also shows a high efficiency when it
comes to detecting cloaks with extreme parameters (such as a negative
permittivity in the case of SC cloaks, or as anisotropic rapidly-varying
materials in the case of TO cloaks), as well as a high index contrast with
either the background or the cloaked object (which is the situation in all the
analyzed cases).

Finally, given the analogy between optics and acoustics, as well as the
advanced state of sonic diffraction tomography [28, 29], the proposed
technique could be easily extended to the detection of sound cloaks [30, 31].

## Acknowledgements

F.J. D.-F. acknowledges funding from project TEC2015-63838-C3-1-R OPTONANOSENS
(MINECO/FEDER, UE) under grant FPI BES-2016-076818.

##  Appendix A Scattering Cross Section (SCS)

To evaluate the rate of the electromagnetic energy that is scattered by a
particle, the scattering cross-section [26] is defined as:

$\displaystyle{\rm SCS}=\frac{W_{\rm{SCA}}}{P_{\rm{INC}}}~{},$ (10)

where $P_{\rm{INC}}$ is the incident irradiance, defined as energy flux of the
incident wave [W/m]. $W_{\rm{SCA}}$ is the scattered energy rate [W], which,
in 2D, is derived by contour integration of the Poynting vector associated
with the scattered field ($\mathcal{P}_{\rm SCA}$) over a circumference
enclosing the particle ($L$):

$W_{\rm{SCA}}=\oint\limits_{L}\mathcal{P}_{\rm
SCA}\cdot\hat{\textbf{n}}~{}dl,$ (11)

where $\hat{\mathbf{n}}$ is the unit vector normal to $L$. In the present
work, all electromgnetic field and SCS calculations were performed in Comsol
Multiphysics.

## References

  * [1] N. Engheta and R. W. Ziolkowski, Metamaterials: Physics and Engineering Explorations, (Wiley, 2006).
  * [2] J. B. Pendry, D. Schurig, and D. R. Smith, Controlling electromagnetic fields, Science 312, 1780 (2006).
  * [3] A. Alù and N. Engheta Achieving transparency with plasmonic and metamaterial coatings, Phy. Rev. E 72, (2005).
  * [4] D. Schurig, J. J. Mock, B. J. Justice, S. A. Cummer, J. B. Pendry, A. F. Starr, and D. R. Smith, Metamaterial Electromagnetic Cloak at Microwave Frequencies, Science 314, 977 (2006).
  * [5] D. Rainwater, A. Kerkhoff, K. Melin, J. C. Soric, G. Moreno, and A. Alù, Experimental verification of three-dimensional plasmonic cloaking in free-space, New J. Phys. 14, 013054 (2012).
  * [6] M. Kadic, T. Bückmann, R. Schittny, and M. Wegener Experiments on cloaking in optics, thermodynamics and mechanics, Philos. Trans. A Math. Phys. Eng. Sci. 373, 20140357 (2015).
  * [7] F. Monticone and A. Alù, Do Cloaked Objects Really Scatter Less?, Phys. Rev. X 3, 041005 (2013).
  * [8] P.-Y. Chen, J. Soric, and A. Alù, Invisibility and Cloaking Based on Scattering Cancellation, Adv. Mater. 24, OP281 (2012).
  * [9] F. Monticone and A. Alù, Invisibility exposed: physical bounds on passive cloaking, Optica 3, 718 (2016).
  * [10] A. Greenleaf, M. Lassas, and G. Uhlmann, Anisotropic conductivities that cannot be detected by EIT, Physiol. Meas. 24, 413 (2003).
  * [11] A. J. Devaney, A filtered backpropagation algorithm for diffraction tomography, Ultrason. Imaging 4, 33 (1982).
  * [12] P. Müller, M. Schürmann, and J. Guck, The Theory of Diffraction Tomography, arXiv:1507.00466 [q-bio.QM] (2015).
  * [13] P. Müller, M. Schürmann, and J. Guck, ODTbrain: a python library for full-view, dense diffraction tomography, BMC Bioinformatics 16, 367 (2015).
  * [14] M. Schürmann, G. Cojoc, S. Girardo, E Ulbricht, J. Guck, and P. Müller, Three-dimensional correlative single-cell imaging utilizing fluorescence and refractive index tomography J. Biophotonics 11, e201700145 (2017).
  * [15] F. Gömöry, M. Solovyov, J. Souc, C. Navau, J. Prat-Camps, and A. Sanchez, Experimental Realization of a Magnetic Cloak, Science 335, 1466 (2012).
  * [16] R. Schittny, M. Kadic, T. Bückmann, and M. Wegener, Invisibility cloaking in a diffusive light scattering medium, Science 345, 427 (2014).
  * [17] A. Alù and N. Engheta, Cloaking a Sensor, Phys. Rev. Lett. 102, 233901 (2009).
  * [18] B. Chen and J. J. Stamnes, Validity of diffraction tomography based on the first born and the first rytov approximations, Appl. Opt. 37, 2996, (1998).
  * [19] M. G. Silveirinha, A. Alù, and N. Engheta, Parallel-plate metamaterials for cloaking structures, Phys. Rev. E 75, 036603 (2007).
  * [20] P. B. Johnson and R. W. Christy, Optical constants of the noble metals, Phy. Rev. B 6, 4370 (1972).
  * [21] U. Leonhardt, Optical Conformal Mapping, Science 312, 1777 (2006).
  * [22] S. A. Cummer, B.-I. Popa, D. Schurig, D. R. Smith, and J. Pendry, Full-wave simulations of electromagnetic cloaking structures, Phys. Rev. E 74, 036621 (2006).
  * [23] B. Zhang, B.-I. Wu, H. Chen, and J. A. Kong, Rainbow and Blueshift Effect of a Dispersive Spherical Invisibility Cloak Impinged On by a Nonmonochromatic Plane Wave, Phys. Rev. Lett. 101, 063902 (2008).
  * [24] H. Chen and B. Zheng, Broadband polygonal invisibility cloak for visible light, Sci. Rep. 2, 255 (2012).
  * [25] B. Orazbayev, M. Beruete, A. Martínez, and C. García-Meca, Diffusive-light invisibility cloak for transient illumination, Phys. Rev. A 94, 063850 (2016).
  * [26] S. Yushanov, J. S. Crompton, and K. C. Koppenhoefer, in Mie Scattering of Electromagnetic Waves, proceedings of the COMSOL Conference in Boston (2013).
  * [27] H. Chen, U. Leonhardt, and T. Tyc, Conformal cloak for waves, Phys. Rev. A 83, 055801 (2011).
  * [28] O. M. Gaddoura, and M. Ding, Diffraction Tomography: It’s Application in Ultrasound, Int. J. Manuf. Eng. 1, 10 (2011).
  * [29] N. Ozmen, R. Dapp, M. Zapf, H. Gemmeke, N. V. Ruiter, and K. W. van Dongen, Comparing different ultrasound imaging methods for breast cancer detection, IEEE Trans. Ultrason. Ferroelect. Freq. Control 62, 637 (2015).
  * [30] A. N. Norris, Acoustic cloaking, Acoust. Today 11, 38 (2015).
  * [31] S. A. Cummer, J. Christensen, and A. Alù, Controlling sound with acoustic metamaterials, Nat. Rev. Mater. 1, 16001 (2016).


# Cross-Axis Transformer with 2d Rotary Embeddings ††thanks: Citation:
Authors. Title. Pages…. DOI:000000/11111.

Lily Erickson
Independent Researcher
Minneapolis
<EMAIL_ADDRESS>

###### Abstract

Despite lagging behind their modal cousins in many respects, Vision
Transformers have provided an interesting opportunity to bridge the gap
between sequence modeling, and image modeling. Up until now, however, vision
transformers have largely been held back, due to both computational
inefficiency, and lack of proper handling of spatial dimensions. In this
paper, we introduce the Cross-Axis Transformer. CAT is a model inspired both
by Axial Transformers, and Microsoft’s recent Retentive Network, that
drastically reduces the required number of floating point operations required
to process an image, while simultaneously converging faster and more
accurately than the Vision Transformers it replaces.

_K_ eywords Computer Vision $\cdot$ Transformer $\cdot$ Retentive Network
$\cdot$ AI

##  1 Introduction

There have been many attempts at processing visual data using neural networks.
Convolution Neural Networks use learned kernels to allow local patches of data
to share information with each other. ResNets perform a similar role, but
along a different axis, attempting to carry a residual version of their state
forward.

Recently, transformers have broken onto the scene, which instead of caring
strictly about their neighboring features, instead attempt to create a
probability distribution over the entire feature space, using a process known
as attention.

$Attention(Q,K,V)=softmax(QK^{T}\sqrt{dk})V$ (1)

There are a few problems with attention, notably that the softmax operation is
computationally expensive, and that expanding the sequence length scales
quadratically with its input size. Despite these flaws, however, transformers
have shown excellent performance.

The Vision Transformer (ViT) is a model that takes an image, breaks it into
patches, and applies the standard transformer attention over the patches to
transform that image into an objective function. Classic implementations have
typically used the standard sinusoidal positional embeddings, and the typical
quadratic attention.

Earlier this year, researchers at Microsoft unveiled their Retentive Network
[1], which showcased a version of a language model that ignores Softmax
completely, and manages to reduce the computational complexity in their
recurrent version by splitting the matrix multiplication steps into separate
channels.

We combine these ideas, along with concepts proposed in Axial Transformers
[2], to reduce the computational complexity of the large input size of images,
while simultaneously showcasing the lack of dependence on Softmax as a whole.

The most similar in execution to our architecture appears to be MaxViT’s[3] in
concept, which uses a strided variant of axial attention to ensure total
coverage of the feature dimensions, however where we differ is in the fact
that in our model, we manage to both enable every position within the image to
attend to every other position in the image in a single Q * K * V pass with
O(n) linear complexity, as well as in our formulation for a 2d positional
embedding.

##  2 Architecture: Cross-Axis Transformer

Table 1: ImageNet 1k accuracy after 10 training epochs Method | Train Acc | Val Acc | FloPs | FPP
---|---|---|---|---
Classic ViT | 0.184 | 0.137 | 37.23bn | 784.94
DINOv2 | 0.257 | 0.215 | 37.23bn | 752.50
BeiT | 0.415 | 0.352 | 24.83bn | 784.58
CAT w/ Imprint | 0.641 | 0.389 | 24.85bn | 760.24

We make several changes to the contemporary Vision Transformer. We note that
for the purposes of our testing, our model achieved almost 2.8 times the
validation accuracy as the baseline vision transformer, while taking half the
training time, and requiring only 2/3rds the number of floating point
operations. We observe that Floating-Point-Operations Per Parameter (FPP)
remain roughly consistent between models, however CAT and BeiT manage to
achieve impressive results with much fewer weights. CAT itself manages to
deliver significantly smaller gradients thanks to its linear derivation,
enabling much larger batch sizes than even BeiT.

We’d also like to note that our research is being conducted on a single
tower/laptop combo, and so although our results are extremely promising, there
is a clear need for much wider scale testing, using compute platforms that are
far more powerful than our own, hyper-parameters that are optimized for longer
training sessions, and larger training datasets. We publish these findings in
the interest of broadcasting the need for such larger scale testing, and in
doing so, to hopefully define a new paradigm for vision transformers.

As a quick aside, it’s become clear to me that Dino v2 does a fantastic job at
minimizing overfitting.

Algorithm 1 Pseudo Code

1:class CrossAxisAttention(nn.Module):

2: def __init__(config):

3: super().__init__()

4: self.qkv = nn.Linear(hidden_size, hidden_size * 3)

5: self.gr_norm = nn.GroupNorm(num_heads, num_heads)

6: self.o_prog = nn.Linear(hidden_size, hidden_size)

7: def forward(inputs):

8: …see details 2

9:

10:class CatBlock(nn.Module):

11: def __init__(config):

12: super().__init__()

13: self.in_norm = nn.LayerNorm(hidden_size)

14: self.attn = CrossAxisAttention(config)

15: self.out_norm = nn.LayerNorm(hidden_size)

16: self.ffn = FFN(hidden_size)

17: def forward(inputs):

18: inputs = self.attn(self.in_norm(inputs)) + inputs

19: inputs = self.ffn(self.out_norm(inputs)) + inputs

###  2.1 Cross-Axis Attention

As a review, standard attention[4] takes as input a length $i$ sequence of $d$
dimensional embeddings, and produces three learned linear output embeddings, Q
K and V. Let X be the hidden State, and let W be the DxD learned parameter
matrices:

$Q=XW_{Q},K=XW_{k},V=XW_{V}$

We calculate attention by performing matrix-multiplication between Q and the
transpose of K, then by dividing that by the square root of the embedding
depth d (commonly referred to as the hidden dimensionality) for numerical
stability and normalization. We then perform softmax over this result to
reduce the predictions to a probability distribution over the input sequence
(which allows the model to "attend to" or focus on which parts of the sequence
are most important".

$A=\text{softmax}(QK^{T}/\sqrt{D},\quad Y=AV$

We would like to first point towards Jonathan Ho et al[2]’s work on Axial
Attention. They note that the quadratic complexity makes attention ill-suited
for long sequence lengths like vision transformers, and instead opt to perform
attention over each axis separately. This breaks the attention’s compute
complexity down (assuming a square image of size $N=SxS$ or rather
$S=\sqrt{N}$) from $O(N^{2})=O(S^{4})$ to $O(S\cdot S^{2})=O(N\sqrt{N})$.

However, quite recently, work by Jonathan Ho et al[2]’s team on their
Retentive Network has laid bare two very important things. First, softmax is
not required to get good results from attention, and second, you can still
achieve great results even if you split the input sequence into chunks and
perform cross-retention between chunks.

This lays bare a very simple logical leap: If we can reduce computational
complexity by performing attention across axes, then we can combine the
benefits of chunk-wise retention and axial attention into a single model.
Doing this produces a linear attention in terms of computational complexity,
however we’re still attending to every patch during our attention operation,
all without the need to sacrifice the quality we’ve grown to expect from
quadratic attention or needing to employ lossy kernels to approximate the
process.

Consider the following tensor operations for a square image after patching:

Algorithm 2 Cross-Axis Forward Pass

1:def forward(inputs, image_embed): # (batch, height, width, dim) = (b h w d)

2: inputs = inputs + image_embed

3: q, k, v = self.qkv(inputs)

4: k = k * self.gamma

5: q, k = pos(q, k)

6: QK = Q @ K.T # (b h w d) @ (b h d w) -> (b h w w)

7: input = QK.transpose(1, 2) @ V # (b w h w) @ (b h w d) -> (b w h d)

8: input = self.gr_norm(inputs.transpose(1, 2))

9: return self.o_proj(inputs)

Of note, we’ve left out the multi-head attention operations for simplicity.
You’ll notice that we have effectively performed a linear variant of axial
attention, in a manner that reduces the computational complexity to
$O(S^{2})=O(N)$. Not only this, but any attempt on our end to apply a
normalization term in between these calculations degraded performance, so I
think the results speak for themselves.

Additionally, we have borrowed the attention head operations from RetNet’s[1]
GroupNorm and gamma scaling implementations, to keep up with modern
architectures of whose inspiration we are building upon.

###  2.2 Architecture: Multi-Scale Rotary Axial Embeddings

It has been observed in previous works such as Pyramid Transformers[5], that
heirarchical representations of features lead to an improved ability to learn
and adapt and following the work of Jianlin Su et al[6] with Rotary Positional
Embeddings, it has become apparent that traditional learned positional
embeddings fall short in many areas.

We notice that not only are most ViT architectures still using traditional
transformer positional embeddings, but that they also fail to properly account
for both the height and width dimensions simultaneously. We’d like to point
out recent progress in this department from the likes of Ze Liu et al’s[7]
implementation with Swin2 and similar publishings, as they attempt to expand
positional embeddings in a scale-invariant, 2-dimensional fashion.

We base our positional embeddings on a rotation matrix, in the same vein as
Roformer. In fact, Roformer begins its derivation of their 1d rotational
embeddings with the 2d variant. Let’s look at the equation of a standard
rotation matrix.

$R=\begin{bmatrix}\cos(\theta)&-\sin(\theta)\\\
\sin(\theta)&\cos(\theta)\end{bmatrix}\begin{bmatrix}x\\\
y\end{bmatrix}=\begin{bmatrix}x^{\prime}\\\ y^{\prime}\end{bmatrix}$

Whereas the original Roformer opts to use the following decay function in
order to solve for their sum of the rotation matrix like so:

$\displaystyle
Q=Q_{\text{pos},2i}*\cos\left(\frac{\text{pos}}{10000^{2i/d}}\right)+\left[\begin{array}[]{c}Q_{\text{pos},2i+d/2}\\\
-Q_{\text{pos},2i}\end{array}\right]*\sin\left(\frac{\text{pos}}{10000^{2i/d}}\right)$
(4)

We observe that not only would such a decay function be actively detrimental
within the image domain due to the increasing focus towards the end of the
sequence, but by reformulating the sequence information to be a fixed distance
between the range of $-/pi$ and $+pi$, we can capture the full range of the
rotation matrix’s expressive power in a scale-invariant manner, creating an
effective relative positional embedding for 2d images of any size or scale. We
do this for both the height and width dimensions separately, and then
interleave the two to intertwine both a height and width embedding together.

$0$$0.2$$0.4$$0.6$$0.8$$1$$0$$0.2$$0.4$$0.6$$0.8$$1$Feature Dimension Depth
%Embedding IntensityDecay Curve

For simplicity, let $p=d_{i},x_{i},y_{i}$ where i is the $i$th position in the
sequence and $x$, $y$, and $d$ are the axes of the matrix.

$\text{Decay Curve }d_{i}=1/10000^{2i/d}$

For frequency modulation along the hidden dimension axis, we’ll borrow the
original sinusoidal embedding’s [4] decay function, which we’ll project along
the hidden dimension instead of along the sequence dimension to create a
multi-scale representation of the positional embeddings. We concatenate two of
these half-depth dimensions together to form our frequency basis.

We note that this still allows for a relative 2d embedding to exist by
multiplying the smaller axis by an aspect ratio (not shown for simplicity,
though a version exists in the public implementation’s code base).

$pos\begin{cases}x_{i},d_{2i}=(x_{i}*2-x)/x*pi\\\
y_{i},d_{2i+1}=(y_{i}*2-y)/y*pi\end{cases}$

Finally, we’ll perform the matrix rotation operations on these embeddings, to
encode the data per Roformer’s[6] additive method. Of note, we can’t
interleave these if we’re interleaving our height and width coordinates. We
are, in essence, forced to interleave one and only one of these operations,
lest we perform the rotation on only a single axis.

$p_{di}=\cos(d_{i})+\sin\begin{cases}d_{i}&\text{if }i<=d/2\\\ -d_{i}&\text{if
}i>d/2\end{cases}$

###  2.3 Architecture: Residual Imprint

We found that we could increase validation accuracy significantly (by about
0.6%), by adding a copy of the image embedding to the model immediately prior
to the attention operation. This gain only works when using a Conv2d layer to
do the embedding. Although convolution layers are historically heavier on
compute than linear projections, we only need to carry the initial embedding
forward rather than recompute it every time. Curiously, training accuracy
increased at only a fraction of the pace, which seems to imply that our
imprint method actually reduces overfitting, and there may need to be more
tests to determine the optimal application of this technique. We have some
interesting findings surrounding the exact details leading up to the
formulation of this iteration in the Ablation 4 Studies notes.

Currently, my leading theory as to why we achieve these results, is due to the
much more expressive nature of convolutions over traditional linear layer
patching. The BatchNorm layers get a much larger range of data in the later
layers than they would in standard models, and therefore need to work harder,
and learn more in less time, leading to faster training and less dependence on
other learned features.

The benefit disappears, and in fact inverts, if the imprint is applied only to
the early layers of the model. Improvement is dependent upon augmentation of
the late stage layers of the model.

##  3 Training Parameters

Table 2: Training Settings Trainer Parameters |
---|---
Epochs | 10
Optimizer | AdamW
LR Schedule | Cosine Annealing
Learning Rate | 3e-4
Dataset | ImageNet1k (HuggingFace)

Our model was primarily trained on a pair of RTX 3090’s. The intent of this
paper is, in part, to show that even consumers and independent researchers are
able to contribute to the community, and that their contributions can in fact
still be valuable.

We’d like to mention however, that this has imposed some restrictions.

  * •

We only train our models for 10 epochs. For CAT, this is about 9 hours when
not compiled. For ViT, this is about 20 hours. For DINOv2, this is about 13
hours of training time. This is all performed on a single device, without the
ability to run tests in parallel.

  * •

We don’t have access to the software stack that big tech companies do. This
means failed runs, restarts, datasets, and tooling, all require custom
implementations.

  * •

We do not have large teams or standardized resources, knowledge bases, or
other benefits that large tech companies do.

  * •

Our hyperparameters were largely chosen by small initial tests based on our
small epoch window. We note that scaling them up requires revisiting these
settings.

Table 3: Training Settings CAT, ViT, BeiT, Dino v2 Hyperparameters |
---|---
Patch Size | 8
Attention Heads | 8
Batch Size | 128
Hidden Layers | 5
Hidden Size | 1024

We’ve open-sourced the tools that we used, and plan on continuing to do so as
we build more on top of them.

As a final note, we had to reduce the batch size for Dinov2 down to 64, and
for BeiT and ViT down to 32, implying that our model’s gradients are
significantly smaller. We’d advise checking out the Floating Point Operation
counts in Table 12

##  4 Ablation Tests

We attempted a few novel ideas to see if we could achieve superior
performance. Of note, all models used our 2d Rotary Embeddings, as performance
was not even comparable without them.

Table 4: Imagenet 1k ablation results, 10 epochs Model | Train Acc | Val Acc
---|---|---
Baseline Cross-Axis | 0.638 | 0.383
Cross-Axis w/ Embedding Imprint | 0.641 | 0.389
Cross-Axis w/ Learned X/Y Mask | 0.637 | 0.372
Cross-Axis w/ Backwards Tanh Residual | 0.612 | 0.390
Cross-Axis w/ Tanh Residual | 0.579 | 0.362

  * •

We started with some base intuition that the purpose of a neural network was
to transform an input image in the earlier layers, into an objective function
in the later layers. Following this, our initial hypothesis was that adding a
decaying image imprint $1-(\text{layer}_{i}/\text{num layers})$ to the state
at the start of the attention layer would make this process smoother (Tanh
residual in table 4). This produced worse results. However, in an interesting
turn of events, our initial implementation forgot to subtract 1 by the result,
which led to an improvement in performance and validation accuracy, in
addition to a decrease in training accuracy.

  * •

We conclude that carrying an imprint of the image into the later layers of the
model actually decreased overfitting. We made sure to add a class token to the
imprint to ensure we weren’t sampling from image data, and removed the decay
factor entirely for the current Embedding Imprint model, to remove as many
outside variables as possible for the experiment.

  * •

Learned X/Y Mask: Similar to how humans observe only a central spatial field,
the idea was to create a focal mask that emphasized only the learned region of
an image in the same vein as SWIN[7], though in a lighter weight and more
modular method of allowing the model to focus its own attention. This resulted
in worse performance by about 1%. We effectively defined a kernel that
produced a single [x, y] coordinate pair per image, and then used that to
define an exponential decay mask. Given the poor results, we won’t derive the
full method, though the implementation will be available in the code for those
curious.

Of additional interest, we note that Microsoft applies a normalization
function between the two attention operations as part of its RetNet
implementation, however implementing it here resulted in severe accuracy
degeneration of the model, so we were forced to remove it. My current theory
is that having a 2d Convolution layer handle the patching for the imprint,
results in much more expressive LayerNorm layers, that are otherwise lost with
the added smoothing.

To ensure that I’ve made note of this, there are many papers making use of
methods like cross-covariance, or axial-attention variants, that use similar
naming schemes or verbiage to our model, we were unable to find a perfectly
lossless linear attention permutation in any of them. Regardless, I will make
mention of it here to ensure that it’s on people’s radar.

##  5 Conclusion

Our research shows that cross-axis attention is an extremely computationally
efficient and empirically strong framework for image classification, and
likely other tasks that have positional information along multiple axes.

Our work is simultaneously able to solve two issues:

a) We provide a solution to the quadratic scaling problem with regard to data
types with large input features (images), allowing us to perform attention on
the full range of input features, without the need to resort to convolving
strictly neighboring segments, or by performing attention across individual
axes separately.

b) We enable efficient encoding of spatial information in 2d space using a
rotation matrix, a method that has overlap with existing language models like
Llama[8].

Despite being only a single individual doing research on consumer grade
hardware, I believe these contributions are still extremely useful and
practical, and will pave the way for future research and additional
efficiencies.

## Acknowledgments

We would like to mention that we were really looking forward to comparing our
results with Swin2[7], however we were unable to get a model of similar size
to converge in time. Their results seem excellent, and I would love to make an
amendment at a later date comparing one of our models to a more complex
architecture such as theirs at a future date.

We will make all of our testing code open source on GitHub immediately
following publishing.

Special thanks to Omid Manzari for both the endorsement, and for being lucky
enough to coincidentally be my source of heirarchical inspiration nearly a
year ago.

I wished to include Emme as an author here, because this paper would
absolutely not have been made without her, however Arxiv currently has a rule
against including AI language tools as authors, and I am unclear where that
line exactly falls. As such, we will include a special shout out here until
such a time that the rule is loosened.

Also of note, ChatGPT was critical in helping to learn a new field over the
past year and a half, in helping to produce this document at the speed
required of modern research at the current pace, as well as for providing some
of the theoretical understanding behind the math, which has had approximately
10 years to atrophy since I last attended a physics lecture in person. I am
ecstatic to be able to contribute, and look forward to refining my tools,
methods, and contributions. Special mention to Bard near the end too, in
helping bring this home during the final cram session.

And finally, although our formulation of 2d Rotary Embeddings was done
independently (sans AI aid, of course), while researching prior art I did
uncover a similar implementation by Phil Wang (Lucidrains on Github) in his
Rotary Embedding library, which we opted not to use due to an unfortunat
perfect modulation along the peaks and troughs of the cosine frequency during
unit-testing. Regardless, I believe it is still worth mentioning that the idea
in question has been visited and before.

If anyone would like to donate compute resources in order to properly test our
models at scale, or if anyone would like to collaborate on further unrelated
research, I am interested in discussing further research opportunities.

## References

  * [1] Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei.  Retentive network: A successor to transformer for large language models, 2023.
  * [2] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans.  Axial attention in multidimensional transformers, 2019.
  * [3] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Title:maxvit: Multi-axis vision transformer, 2022.
  * [4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2023.
  * [5] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021.
  * [6] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding, 2022.
  * [7] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo.  Swin transformer v2: Scaling up capacity and resolution, 2022.
  * [8] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.