# Tuneable homogeneous kG magnetic field production using permanent magnets

Danielle Pizzey<EMAIL_ADDRESS>Joint Quantum Centre (JQC)
Durham-Newcastle, Department of Physics, Durham University, South Road,
Durham, DH1 3LE, United Kingdom

###### Abstract

We present a permanent magnet arrangement that can achieve a tuneable axial
magnetic field from 1.80(8) – 2.67(2) kG along the $z$-axis, where ($x$,$y$) =
(0,0), with a rms field variation of less than 1% over a length of 25 mm. The
instrument consists of an arrangement of off-the-shelf N42 neodymium-iron-
boron (NdFeB) axially magnetized ring magnets, of varying outer and inner
diameters. The magnets are organized into four cylindrical brass holders,
whose relative separation can be manipulated to achieve the desired magnetic
field strength and homogeneity over the region of interest. We present
magnetic field computations and Marquardt-Levenberg fits to experimental data
and demonstrate excellent agreement between theory and experiment. The
apparatus has been designed to accommodate a cylindrical atomic vapor cell of
length 25 mm and diameter 25 mm to lie within the bore of the ring magnets.
The design has a clear aperture of 20 mm in the $x$-$y$ plane along the
$z$-axis, which opens up new avenues for imaging exploration in atomic
spectroscopy.

††preprint: AIP/123-QED

##  I Introduction

There are a vast number of situations in experimental atomic and optical
physics where magnetic fields are utilized, from a Zeeman slower necessary to
slow and cool an atomic beam Hopkins  _et al._ (2016); Cheiney  _et al._
(2011), to magneto-optical filters (MOFs) that use thermal atomic vapors in
magnetic fields Zentile  _et al._ (2015a). MOFs find a wide range of
applications in other disciplines including e.g. quantum key distribution Shan
_et al._ (2006); optical isolators Weller  _et al._ (2012); atmospheric LIDAR
Yang  _et al._ (2011); Xia  _et al._ (2020); and laser frequency stabilization
Chang  _et al._ (2019); Miao  _et al._ (2011). In certain cases there is the
need to produce large, uniform magnetic fields (i.e. > 1 kG) in order to study
a specific branch of physics, in particular thermal vapor experiments working
in the hyperfine Paschen-Back regime Ponciano-Ojeda  _et al._ (2021); Whiting
_et al._ (2018). It is not an easy feat to produce large uniform magnetic
fields: a work-around is typically to reduce the physical dimensions of the
interrogation region over which the experiments are performed Olsen  _et al._
(2011); Sargsyan  _et al._ (2014). However, this is not applicable to all
situations and the need to produce large magnetic fields over increasingly
large length-scales is ever pressing. A particular example is in solar flare
forecasting Cacciani  _et al._ (1990): the Sun is imaged through an optical
system referred to as a solar filter Cimino  _et al._ (1968); Agnelli  _et
al._ (1975). The solar images taken are used to extract the line-of-sight
solar magnetic field that is necessary for the forecasting model Korsós  _et
al._ (2018, 2020); Erdelyi  _et al._ (2021). The optimum parameters for a
solar filter of a particular atomic species depends on numerous factors,
including the axial magnetic field, vapor cell length and vapor temperature.
The optimum parameters can be determined theoretically using a modified
version of ElecSus, which is a software package that calculates the electric
susceptibility of an atomic medium Zentile  _et al._ (2015b), and it is these
parameters that set the physical constraints on the magnetic field strength,
geometry and dimensions of the solar filter. Often, solar filters require
axial magnetic fields of several kilogauss that are uniform over length-scales
of tens of millimeters along the $z$-axis Stangalini  _et al._ (2018).
Additionally, since these filters are used in imaging, large clear apertures
are required in the $x$-$y$ plane, which adds to the complexity of designing a
system that can produce large magnetic fields. It is these constraints that we
have complied with in this work.

Magnetic fields can be produced via an electromagnet, which uses a coil of
wire and a current supply, or by a permanent magnet. There are many pros and
cons to choosing either method. One particular benefit to using the
electromagnet is the ability to tune the magnetic field by simply changing the
current supplied to the coil; this is advantageous when testing a new idea or
characterizing an experiment. However, to produce magnetic fields of several
kG using electromagnets, cooling is typically required to prevent over-heating
of the coils if used continuously Kiefer  _et al._ (2014); McCarron  _et al._
(2007); Gerhardt (2018). On the flip side, permanent magnet arrangements do
not have any power requirements, hence no cooling, nor do they suffer from
magnetic field noise due to current fluctuations originating from the power
supplies. A “set and forget” approach is ideal when making a system
transportable and controlled remotely, as this reduces the number of potential
technical failures during field-work. It is for this reason that we have opted
to pursue the permanent magnet route. With appropriately designed hardware, we
address the tuneability problem and demonstrate field homogeneity over the
required length-scale.

The remainder of the paper is organized as follows: in Section II.1 we present
the magnetic field computation; Section II.2 discusses the functionality of
the hardware and limitations to the maximum achievable magnetic field; we
illustrate the performance of the assembly in Section III; and finally
conclusions are drawn and an outlook provided in Section IV.

##  II Design Overview

The magnet geometries described in this paper were designed specifically for a
potassium solar filter. The requirements are: a maximum axial magnetic field
of $\approx$ 2.7 kG; a field homogeneity over 25 mm along the $z$-axis; a
clear imaging window of a minimum of 20 mm diameter in the $x$-$y$ plane;
minimal contribution from the $x$\- and $y$-components of the magnetic field
across the imaging window; and $\approx$ 1 kG tuning range of the magnetic
field strength along the $z$-axis.

###  II.1 Magnetic field computation

Finding a permanent magnet arrangement that abides by the constraints stated
in section II is a complex task. There are many software packages available
that enable the magnetic field to be determined, however we chose to use
Magpylib Ortner and Bandeira (2020), which is a python package for magnetic
field computation. The Magpylib package is open source and provides visual
aids of the magnet geometry, which assisted the design of hardware mount.

The criteria of the instrument, outlined in section II, state there must be a
clear aperture of 20 mm in diameter in the $x$-$y$ plane along the magnetic
field axis (i.e. $z$-axis) to permit solar imaging through the atomic vapor
cell. Since the magnetic field is a vector quantity,
$\bf{B}\,\equiv\,(B_{x},B_{y},B_{z})$, using magnets with cylindrical symmetry
will make the design simpler to facilitate. We focus our studies on ring
magnets with opposite polarity on opposite faces as this produces an axial
magnetic field.

Figure 1: Simple picture of a) a ring magnet with dimensions 2$R_{\mathrm{o}}$
= 10 mm, 2$R_{\mathrm{i}}$ = 5 mm and 2$d$ = 5 mm and remanence field $B_{0}$
= 12.8 kG, typical of N42 grade NdFeB magnets and the associated b) magnetic
field profile calculated using Magpylib Ortner and Bandeira (2020). Slices of
the magnetic field components i) $B_{x}$ and ii) $B_{z}$ as a function of $x$
taken at $z$ = 0 mm (green), $z$ = 1 mm (blue), and $z$ = 2 mm (red). Vertical
black dotted lines represent the physical profile of the magnet, i.e. $\pm$ 5
mm illustrates the outside diameter of the ring, whereas $\pm$ 2.5 mm
illustrates the inside diameter of the ring. Also shown is the total magnitude
of the magnetic field given by Eq. 2 as a function of $x$ taken at $z$ = 0 mm
(green), $z$ = 1 mm (blue), and $z$ = 2 mm (red). iv) Magnetic field profile
displayed in the $x$-$z$ plane illustrated in magnitude by color and direction
by arrows.

In order to design an appropriate magnet assembly we must first understand the
magnetic field profile of a single ring magnet. Figure 1 illustrates the
magnetic field profile of a ring magnet with outer diameter 2$R_{\mathrm{o}}$
= 5 mm, inner diameter 2$R_{\mathrm{i}}$ = 2.5 mm, length 2$d$ = 5 mm, as
shown in Fig. 1 a), and remanence field $B_{0}$ = 12.8 kG, which is typical of
N42 grade NdFeB magnets. Although this ring magnet does not meet our
constraints, we can learn a lot from the figure. In Fig. 1 b) iv), the
magnetic field profile is shown in the $x$-$z$ plane and, due to cylindrical
symmetry, we can assume the profile is identical in the $y$-$z$ plane also.
The arrows demonstrate the field direction, while the color of the arrow
represents the magnitude of the field. Inside the physical profile of the
magnet, the field is strongest. Within the bore of the ring magnet we see that
the field is the opposite to that inside the magnet and a lot weaker, as
evidenced by the change in color of the arrow. Additionally, the field lines
within the bore of the ring magnet are not evenly spaced along the $x$-axis
and display curvature, indicating that the field along $x$ is not homogeneous.
This is evident in Fig. 1 b) i) and ii), which show the $B_{x}$ and $B_{z}$
component of the magnetic field as a function of $x$ for three $z$ positions.
At $x$ = 0 mm, we see that $B_{x}$ = 0 for all $z$ positions. However, as we
translate along $x$, the argument that $B_{x}$ = 0 is no longer valid except
for the case when $z$ = 0 mm. The further we move from ($x$,$z$) = (0,0), the
larger the $x$ component of the field becomes. Additionally, we see that
$B_{z}$ begins to reduce as we move further from $z$ = 0 mm. In practice it is
difficult to measure an individual component of the magnetic field and it is
typically the magnitude of the total field that is measured at a particular
point in space. Figure 1 b) iii) illustrates the magnitude of the total field
as a function of $x$ for three $z$ positions discussed in Fig. 1 b) i) and
ii). The magnitude of the total field along $x$ is not homogeneous and
displays curvature similar to that of the $B_{z}$ component of the magnetic
field, since this component is dominant over $B_{x}$ and $B_{y}$ in this
design. The details of why this occurs will not be discussed in detail here,
as it is beyond the scope of this paper, however a thorough description and
investigation can be found in ref Trénec  _et al._ (2011).

This simple picture aids our understanding of the magnetic field profile of
ring magnets. There are three take home messages that we learn from Fig. 1.
First, there is always a radial component of the magnetic field when ($x$,$y$)
$\neq$ (0,0). Nevertheless, with an appropriately designed magnet assembly,
the radial component can be made to be significantly smaller than the axial
field component, so much so that we can neglect it in our computations. It is
for this reason that we can assume that a cylindrical magnet with uniform
magnetization, whose axis is aligned with the $z$ = 0 axis and $B_{x}$ =
$B_{y}$ = 0, has the following $z$-component of the magnetic field Weller
(2013),

$B_{z}(z)=\frac{B_{0}}{2}\left(\frac{z-z_{0}+d}{\sqrt{(z-z_{0}+d)^{2}+R^{2}}}-\frac{z-z_{0}-d}{\sqrt{(z-z_{0}-d)^{2}+R^{2}}}\right).$
(1)

Here, 2$d$ is the length of the cylinder, $R$ is its radius, $B_{0}$ is the
remanence field, and $z_{0}$ is the position of the cylinder’s centre along
the $z$-axis. The magnetic field for a ring is calculated using the principle
of superposition. For a ring magnet with outer radius $R_{\mathrm{o}}$ and
inner radius $R_{\mathrm{i}}$ the field is that from a cylinder of radius
$R_{\mathrm{o}}$ minus the field from a cylinder with $R_{\mathrm{i}}$.

Second, to produce field homogeneity along $x$, and similarly $y$, the
physical dimensions of the magnets used should significantly exceed the area
of required field homogeneity. For example, if we require a clear aperture of
diameter 4 mm in the $x$-$y$ plane with field homogeneity along $z$ and
minimal radial components, the magnet ring specifications illustrated in Fig.
1 a), particularly the dimension 2$R_{\mathrm{i}}$ = 5 mm, would be inadequate
as there are huge field variations when $x$ > +1 mm and $x$ < –1 mm. This ring
magnet is more likely suited for a region of interest of diameter 2 mm in the
$x$-$y$ plane.

Third, there will always be a reduction of the field strength and a flip of
the field direction on the axis of the ring magnet upon exit of the bore of
the magnet. This occurs at approximately $\pm$ 3 mm in Fig. 1 b) iv). Thus it
is pertinent to have the length of the ring magnet exceeding the desired
homogeneity length along the $z$-axis.

Using Magpylib, we were able to address the three take home messages specific
to our case. We prioritized our search on ring magnets with inner diameter
greater than 25 mm for two reasons: we needed to fit a 25 mm diameter vapor
cell inside the central bore of the magnets and; for our imaging purposes, we
required a clear aperture of diameter 20 mm with minimal magnetic field
variations across $x$-$y$ and along $z$. Additionally, since we require field
homogeneity along the $z$-axis of 25 mm (the length of our vapor cell), the
length (or thickness) of the ring magnet must exceed 25 mm. In an ideal
scenario, the length of the ring magnet would be infinite, but this is not
feasible when making an experimental instrument. An alternative to using
excess magnet length is to increase the magnetic field at the boundaries of
the region of interest, in our case at the edges of the vapor cell (i.e. at
$z$ = $\pm$ 12.5 mm). The field can be increased by either using an additional
ring magnet on the boundaries that has a higher remanence field (i.e. higher
grade/strength) or by using a greater quantity of magnetic material. We chose
the latter route as working with higher strength magnets are more difficult to
manoeuvre and more dangerous to handle.

Figure 2: a) Illustration of the concentric magnet arrangement in the $x$-$y$
plane used in the computation model. "OD" and "ID" correspond to the outer and
inner diameters of the magnets, respectively, with the number succeeding
either OD or ID indicating the dimension in millimeters. The design can be
thought of as four magnet groups, illustrated by use of four colors: purple;
red; green; and blue. The red and green magnet groups consist of three magnet
layers, with each layer containing two concentric magnets. The purple and blue
magnet groups consist of one magnet layer with three concentric magnets. b)
Illustration of the magnet arrange in the $x$-$z$ plane. The magnet groups are
positioned accordingly: purple positioned at $z_{1}$; red at $z_{2}$; green at
$z_{3}$; and blue at $z_{4}$. c) Theoretical calculation of the axial magnetic
field, $B_{z}$, using Eq. 1. The field from each individual magnet group is
shown and illustrated via their corresponding color, with the magnet group
position shown using dotted lines and $z_{i}$ label, where $i$ = 1, 2, 3, or
4. The total magnetic field is a superposition of the fields from the four
magnet groups and is shown via a solid black line. Also shown are dashed black
vertical lines that define a boundary between which is the region of interest
for the field homogeneity, where for this case $B_{\mathrm{mean}}$ = 2.8 kG
with a root-mean-square field variation of less than 1%.

Figure 2 illustrates a ring magnet geometry that meets the criteria outlined
in section II. The magnets used in this computation are commercial off-the-
shelf neodymium-iron-boron (NdFeB) (N42) ring magnets of thickness 5 mm. We
use three sizes: outer diameter (OD) 90 mm, inner diameter (ID) 64 mm; OD60,
ID40; OD40, ID25. The arrangement and positioning of the magnets is shown in
Fig. 2 a) and b). The design is composed of two halves that have exact-
reflective symmetry about the $z$ = 0 mm mirror line: one half contains the
magnets positioned at $z_{1}$ and $z_{2}$ (colored purple and red,
respectively); while the other half contains the magnets positioned at $z_{3}$
and $z_{4}$ (colored green and blue, respectively). The red and green magnet
groups comprise of three magnet layers, with each layer consisting of two
concentric ring magnets (OD90, ID64 and OD60, ID40) with the same polarity.
The polarity between each adjacent face of the layers is opposite, hence the
layers attract, resulting in zero separation between the layers. The length,
or thickness, of both the red and green group is 15 mm; this exceeds the edges
of the vapor cell that are positioned at $z$ $\pm$ 12.5 mm. Furthermore, the
smallest inner diameter magnet used in these groups is 40 mm, which can also
accommodate a vapor cell of diameter 25 mm.

The purple and blue magnet groups comprise of one magnet layer composed of
three concentric ring magnets (OD90, ID64; OD60, ID40; and OD40, ID25) with
the same polarity: these magnet groups are used to boost the field at the
vapor cell boundaries, as discussed in the third take home message. In total
there are 18 magnets in the system 111Note: Should the reader wish to
reproduce this design, custom made magnets can be used instead to minimize
magnet layering.. The axial magnetic field for this arrangement is calculated
using Eq. 1 and the total field is shown in Fig. 2 c), as well as the
individual contributions from the four magnet groups. The black dashed
vertical lines define the positions of cylindrical vapor cell faces. The mean
axial magnetic field between the two cell faces is $B_{\mathrm{mean}}$ = 2.8
kG with a root-mean-square field variation of less than 1%. By changing the
positions ($z_{1}$, $z_{2}$, $z_{3}$ and $z_{4}$) with respect to $z$ = 0, we
can change the magnetic field strength while still maintaining field
homogeneity over the region of interest. As a rule of thumb, the red and green
magnet groups control the strength of the field, while the purple and blue
magnet groups tend to control the extent of the homogeneity.

Figure 3: Magpylib magnetic field computation for the magnet assembly shown in
Fig. 1. a) Slices of the magnetic field components i) $B_{x}$ and ii) $B_{z}$
as a function of $x$ taken at $z$ = 0.0 mm (green), $z$ = 6.0 mm (blue) and
$z$ = 12.5 mm (red). The vertical, black dashed lines illustrate the clear
imaging window we require. The $B_{z}$ component greatly exceeds that of the
$B_{x}$, and correspondingly $B_{y}$, components of the magnetic field. From
Eq. 2, the contribution is less than 1%. Bottom: Magnetic field displayed in
the $x$-$z$ plane illustrated in magnitude by color and direction by arrows.

Figure 3 illustrates the magnetic field profile in the $x$-$z$ plane
determined using Magpylib for the magnet arrangement shown in Fig. 2. As shown
in Fig 3 b), the magnetic field strength is at its strongest inside the
magnets, as expected. Within the bore of the magnets, where the vapor cell
resides, the field lines are uniform across $x$, which is evident in Fig. 3 a)
i) and ii). Shown are the $B_{x}$ and $B_{z}$ components as a function of $x$
for three $z$ positions, which encompass the region of interest. Since the
magnitude of the total magnetic field, $B_{\mathrm{{total}}}$, is given by

$|B_{\rm{total}}|=\sqrt{(B_{x})^{2}+(B_{y})^{2}+(B_{z})^{2}},$ (2)

the contribution of $B_{x}$ and $B_{y}$ is less than 1%. This is evident in
Fig 3 a) iii) where the magnitude of the field along $x$ echoes the $B_{z}$
profile shown in Fig 3 a) ii).

###  II.2 Hardware criteria

The magnet holder consists of two halves, that are mirror images of one
another, and can be separated to permit insertion of a vapor cell to lie
within the bore of the magnets. Each half consists of a square block and two
cylinders, as shown in Fig. 4 a): one we define as the “large" cylinder; the
other we define as the “small" cylinder. The large cylinder contains either
the green or red magnet group, as shown in Fig. 2. The small cylinder contains
either the purple or blue magnet group.

Figure 4: a) An exploded view schematic illustration of one half of the magnet
holder, which is composed of a square block, a “large” cylinder and a “small”
cylinder. Also shown are the magnets and black plastic spacers. b) Photographs
of the complete assembly. c) Measured magnetic field profile $B_{z}$ (black
dots) when the assembly is set to give maximum field. The average magnetic
field over the region of interest, illustrated via the black dashed lines, is
$B_{\mathrm{m}ean}$ = 2.67 kG with a root-mean-square deviation of 20 G over
this region, i.e. 1% variation. Also shown are the individual contributions
from the four magnet groups that are colored accordingly, as outlined in
figure 2, with the positions and corresponding errors extracted from the
Marquardt-Levenberg fit (grey solid line) Hughes and Hase (2010) using Eq. 1
and the experimental data. The normalized residuals, $R$, are also shown.

The large cylinder back plate is threaded on the outside with a pitch of 2 mm
and screws into the square block, permitting axial translation of 2 mm per
rotation. The thickness of the square block sets how much the large cylinder
can translate along the $z$-axis – a square block thickness of 20 mm is
sufficient to produce the fields investigated in this paper. The front plate
of the large cylinder is threaded on the inside of its bore with a pitch of 2
mm. The back plate of the small cylinder is threaded on the outside which
enables the small cylinder to be screwed into the internal bore of the large
cylinder front plate. This permits an axial translation of the small cylinder
of 2 mm per one full 2$\pi$ rotation, with respect to the large cylinder. This
pitch is sufficient to provide the tuning control we require. Photographs of
the complete assembly are shown in Fig. 4 b). In the bottom photograph the
large imaging window is clearly visible.

Figure 4 c) illustrates the measured maximum field strength and the field
profile along the $z$-axis, for the magnets and hardware used in this design
222Note: The thickness of the front and back plates should be taken into
account when simulating the magnetic field in Magpy (see Supplementary
Materials for a further discussion).. The mean magnetic field along the
$z$-axis over the region of interest of $z~{}=~{}\pm$12.5 mm is
$B_{\mathrm{{mean}}}$ = 2.67 kG with a root-mean-square field variation of 20
G, which is on the order of $\approx$ 1% variation. Also stated are the
positions of the magnet groups in their respective colors. Below, normalized
residuals are also displayed.

The material chosen for the magnet holder is brass. Brass is non-magnetic and
is durable enough to withstand cross-threading while screwing the cylinders
closer together or further apart when tuning the magnetic field strength. To
vary the cylinders positions, additional holes were included on each cylinder
front plates such that custom tools could be inserted and used to rotate the
cylinders and, hence, translate them. CAD drawings for these tools are
included in the Supplementary Materials.

The assembly has been designed such that N42 grade NdFeB magnets can be used
and achieve the fields necessary for our investigation. Magnets of this
strength are easier to control and position, compared to N52 grade magnets,
however, assembling the magnets into the cylinders must be approached with
caution. An assembly guide has been provided in the Supplementary Materials,
including CAD drawings for assembly jigs and photographs for visual aids.

##  III Results

In the remainder of the paper, we demonstrate how we can manipulate the
magnetic field profile by varying the positions of the magnet groups using the
hardware discussed in section II.2. We compare our measurements to theoretical
calculations using Eq. 1.

An axial Hall probe was used to measure $B_{z}$ as a function of $z$. Figure 5
shows three examples of the measured field profile. In each example, the
maximum measured magnetic field is distinct due to different positioning of
the magnet groups, as discussed in section II.1. The experimental data is
fitted using Eq. 1, where we have assumed the magnet dimensions are fixed. We
have extracted the positions of the four magnet groups and the remanence field
using a Marquardt-Levenberg fit: the errors in the fit parameters are
estimated using the square-root of the diagonal elements of the covariance
matrix Hughes and Hase (2010). The positions of the magnet groups, the mean
magnetic field and root-mean-square deviation within the region of interest,
shown via a vertical dashed line, are stated in each subplot. Also shown are
the associated normalized residuals for each subplot, which demonstrate
excellent agreement between theory and experiment.

In Fig. 5 a) we produce an axial magnetic field of 1.80(8) kG and we achieve
homogeneity over a length of 25 mm as required. However, we see that we have
two local field maxima at approximately $\pm$ 40 mm. We attribute these to
inappropriate small cylinder positions, and bringing the positions of the
small cylinders closer to $z$ = 0 will eradicate these. In spite of this, we
have achieved the field required over the region of interest. As shown in Fig.
4 c), the maximum field we can achieve is $B_{\mathrm{{mean}}}$ = 2.67(2) kG
due to the hardware constraints, as illustrated in Fig. 4 a). We can achieve
any value of field between the maximum and 1.80(8) kG; examples are shown in
Fig. 5 b) and c).

Figure 5: Measured axial magnetic field (black dots) with an optimized fit
(solid grey line) is shown for three arrangements of the magnet groups. Also
shown are the normalized residuals for each subplot. Vertical black dashed
lines indicate the vapor cell face positions. The mean magnetic field along
the length of the vapor cell, i.e. between the two black dashed vertical
lines, and root-mean-square deviation is stated in each subplot, as well as
the magnet group positions.

##  IV Conclusion

In conclusion, we have presented a permanent magnetic field design that can
produce an axial magnetic field of several kilogauss with field homogeneity
over a length-scale of tens of millimeters, with negligible contribution from
the radial field components. The hardware permits field tuneability ranging
from 1.80(8) kG to 2.67(2) kG with rms field variation of less than 1%, and we
demonstrate that Marquardt-Levenberg fits to experimental data demonstrates
excellent agreement between theory. The design is such that a cylindrical
vapor cell of length 25 mm and diameter 25 mm can be placed within the bore of
the ring magnets, with a clear aperture of 20 mm for imaging through the vapor
cell. The design will be of significance, and benefit, to those interested in
atomic spectroscopy in large axial magnetic fields.

##  V Supplementary Material

The supplementary material contains hardware CAD drawings (including drawings
for jigs required for assembly and tuning tools), parts list (including a
bill-of-materials) for the design discussed here and assembly advice.

##  VI Acknowledgments

We gratefully acknowledge Ifan Hughes and Steven Wrathmall helpful
discussions, Clare Higgins for editorial discussions, Stephen Lishman and
Malcolm Robertshaw for mechanical design assistance and fabrication, and EPSRC
(Grant No. EP/R002061/1) and STFC (Grant No. ST/V005979/1) for funding.

##  VII Data Availability

The data that support the findings of this study are openly available in DRO
at https://doi.org/10.15128, reference number r2xs55mc08w 333The data
presented in this paper are available from DRO
https://doi.org/10.15128/r2xs55mc08w.

## References

  * Hopkins  _et al._ (2016) S. A. Hopkins, K. Butler, A. Guttridge, S. Kemp, R. Freytag, E. A. Hinds, M. R. Tarbutt, and S. L. Cornish, Review of Scientific Instruments 87, 043109 (2016), https://doi.org/10.1063/1.4945795 .
  * Cheiney  _et al._ (2011) P. Cheiney, O. Carraz, D. Bartoszek-Bober, S. Faure, F. Vermersch, C. M. Fabre, G. L. Gattobigio, T. Lahaye, D. Guéry-Odelin, and R. Mathevet, Review of Scientific Instruments 82, 063115 (2011), https://doi.org/10.1063/1.3600897 .
  * Zentile  _et al._ (2015a) M. A. Zentile, D. J. Whiting, J. Keaveney, C. S. Adams, and I. G. Hughes, Optics Letters 40, 2000 (2015a), https://doi.org/10.1364/OL.40.002000 .
  * Shan  _et al._ (2006) X. Shan, X. Sun, J. Luo, Z. Tan, and M. Zhan, Applied Physics Letters 89, 191121 (2006), https://doi.org/10.1063/1.2387867 .
  * Weller  _et al._ (2012) L. Weller, K. S. Kleinbach, M. A. Zentile, S. Knappe, I. G. Hughes, and C. S. Adams, Optics Letters 37, 3405 (2012), https://doi.org/10.1364/OL.37.003405 .
  * Yang  _et al._ (2011) Y. Yang, X. Cheng, F. Li, X. Hu, X. Lin, and S. Gong, Optics Letters 36, 1302 (2011), https://doi.org/10.1364/OL.36.001302 .
  * Xia  _et al._ (2020) Y. Xia, X. Cheng, F. Li, Y. Yang, X. Lin, J. Jiao, L. Du, J. Wang, and G. Yang, Applied Optics 59, 1529 (2020), https://doi.org/10.1364/AO.382077 .
  * Chang  _et al._ (2019) P. Chang, Y. Chen, H. Shang, X. Guan, H. Guo, J. Chen, and B. Luo, Applied Physics B – Lasers and Optics 125 (2019), https://doi.org/10.1007/s00340-019-7342-5 .
  * Miao  _et al._ (2011) X. Miao, L. Yin, W. Zhuang, B. Luo, A. Dang, J. Chen, and H. Guo, Review of Scientific Instruments 82, 086106 (2011), https://doi.org/10.1063/1.3624696 .
  * Ponciano-Ojeda  _et al._ (2021) F. S. Ponciano-Ojeda, F. D. Logue, and I. G. Hughes, Journal of Physics B: Atomic, Molecular and Optical Physics 54, 015401 (2021), https://doi.org/10.1088/1361-6455 .
  * Whiting  _et al._ (2018) D. J. Whiting, R. S. Mathew, J. Keaveney, C. S. Adams, and I. G. Hughes, Journal of Modern Optics 65, 713 (2018), https://doi.org/10.1080/09500340.2017.1377308 .
  * Olsen  _et al._ (2011) B. A. Olsen, B. Patton, Y.-Y. Jau, and W. Happer, Physical Review A 84, 063410 (2011), https://doi.org/10.1103/PhysRevA.84.063410 .
  * Sargsyan  _et al._ (2014) A. Sargsyan, G. Hakhumyan, C. Leroy, Y. Pashayan-Leroy, A. Papoyan, D. Sarkisyan, and M. Auzinsh, Journal of the Optical Society of America A 31, 1046 (2014), https://doi.org/10.1364/JOSAB.31.001046 .
  * Cacciani  _et al._ (1990) A. Cacciani, D. Ricci, P. Rosati, E. J. Rhodes, E. Smith, S. Tomczyk, and R. K. Ulrich, Il Nuovo Cimento C 13, 125 (1990), https://doi.org/10.1007/BF02515781 .
  * Cimino  _et al._ (1968) M. Cimino, A. Cacciani, and N. Sopranzi, Solar Physics 3, 618 (1968), https://doi.org/10.1007/BF00151943 .
  * Agnelli  _et al._ (1975) G. Agnelli, A. Cacciani, and M. Fofi, Solar Physics 44 (1975).
  * Korsós  _et al._ (2018) M. B. Korsós, P. Chatterjee, and R. Erdélyi, Astrophys. J. 857, 103 (2018), https://doi.org/10.3847/1538-4357/aab891 .
  * Korsós  _et al._ (2020) M. B. Korsós, M. K. Georgoulis, N. Gyenge, S. K. Bisoi, S. Yu, S. Poedts, C. J. Nelson, J. Liu, Y. Yan, and R. Erdélyi, Astrophys. J. 896, 119 (2020), arXiv:2005.12180 [astro-ph.SR] .
  * Erdelyi  _et al._ (2021) R. Erdelyi, M. B. Korsos, X. Huang, Y. Yang, D. Pizzey, S. A. Wrathmall, I. G. Hughes, M. Dyer, V. S. Dhillon, B. Belucz, R. Brajsa, P. Chatterjee, X. Cheng, Y. Deng, S. V. Dominguez, R. Joya, P. Gomory, N. G. Gyenge, A. Hanslmeier, A. Kucera, D. Kuridze, F. Li, Z. Liu, X. Long, M. Mathioudakis, S. Matthews, J. R. T. McAteer, A. A. Pevtsov, W. Potzi, P. Romano, J. Shen, J. Temesvary, A. G. Tlatov, C. Triana, D. Utz, A. M. Veronig, Y. Wang, Y. Yan, T. Zaqarashvili, and F. Zuccarello, J. Space Weather Space Clim., Forthcoming article (2021), https://doi.org/10.1051/swsc/2021025 .
  * Zentile  _et al._ (2015b) M. A. Zentile, J. Keaveney, L. Weller, D. J. Whiting, C. S. Adams, and I. G. Hughes, Computer Physics Communications 189, 162 (2015b), https://doi.org/10.1016/j.cpc.2014.11.023 .
  * Stangalini  _et al._ (2018) M. Stangalini, R. Piazzesi, R. Speziali, and L. Dal Sasso, in  _Proceedings of SPIE_, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 10700 (2018) p. 107001K, https://doi.org/10.1117/12.2313373 .
  * Kiefer  _et al._ (2014) W. Kiefer, R. Löw, J. Wrachtrup, and I. Gerhardt, Scientific Reports 4, 6552 (2014), https://doi.org/10.1038/srep06552 .
  * McCarron  _et al._ (2007) D. J. McCarron, I. G. Hughes, P. Tierney, and S. L. Cornish, Review of Scientific Instruments 78, 093106 (2007), https://doi.org/10.1063/1.2785157 .
  * Gerhardt (2018) I. Gerhardt, Optics Letters 43, 5295 (2018), https://doi.org/10.1364/OL.43.005295 .
  * Ortner and Bandeira (2020) M. Ortner and L. G. C. Bandeira, SoftwareX 11, 100466 (2020), https://doi.org/10.1016/j.softx.2020.100466 .
  * Trénec  _et al._ (2011) G. Trénec, W. Volondat, O. Cugat, and J. Vigué, Applied Optics 50, 4788 (2011), https://doi.org/10.1364/AO.50.004788 .
  * Weller (2013) L. Weller,  _Absolute Absorption and Dispersion in a Thermal Rb Vapour at High Densities and High Magnetic Fields._ (Durham University, 2013) http://etheses.dur.ac.uk/7747/ .
  * Note (1) Note: Should the reader wish to reproduce this design, custom made magnets can be used instead to minimize magnet layering.
  * Hughes and Hase (2010) I. G. Hughes and T. P. A. Hase,  _Measurements and their Uncertainties: A Practical Guide to Modern Error Analysis_ (OUP, Oxford, 2010).
  * Note (2) Note: The thickness of the front and back plates should be taken into account when simulating the magnetic field in Magpy (see Supplementary Materials for a further discussion).
  * Note (3) The data presented in this paper are available from DRO https://doi.org/10.15128/r2xs55mc08w.


# Looking 3D: Anomaly Detection with 2D-3D Alignment

Ankan Bhunia Changjian Li Hakan Bilen
University of Edinburgh
https://groups.inf.ed.ac.uk/vico/research/Looking3D

###### Abstract

Automatic anomaly detection based on visual cues holds practical significance
in various domains, such as manufacturing and product quality assessment. This
paper introduces a new conditional anomaly detection problem, which involves
identifying anomalies in a query image by comparing it to a reference shape.
To address this challenge, we have created a large dataset,
_BrokenChairs-180K_ , consisting of around 180$K$ images, with diverse
anomalies, geometries, and textures paired with 8,143 reference 3D shapes. To
tackle this task, we have proposed a novel transformer-based approach that
explicitly learns the correspondence between the query image and reference 3D
shape via feature alignment and leverages a customized attention mechanism for
anomaly detection. Our approach has been rigorously evaluated through
comprehensive experiments, serving as a benchmark for future research in this
domain.

##  1 Introduction

Anomaly detection (AD) [9, 26], identifying instances that are irregular or
significantly deviate from the normality, is an actively studied problem in
several fields. In standard vision AD benchmarks, ‘irregularities’ are
typically caused by either high-level (or semantic) variations such as
presence of objects from unseen categories [1, 5, 8], defects such as
scratches, dents on objects [4], low-level variations in color, shape, size
[10], or pixel-level noise [16]. The standard approach has been to learn
representations along with classifiers that are robust to the variations
within the regular set of instances, and, at the same time, sensitive to the
ones causing irregularities. However, this paradigm performs poorly when the
irregularities are arbitrary and conditional to the context and/or individual
characteristics of the instance which may not be known in prior or observed.
For instance, in an object category such as ‘chair’ that contains visually
very diverse instances with huge intra-class variation, having three legs may
imply a missing leg and hence an anomaly for a chair instance, while
regularity for another instance. The AD here depends on whether the chair
instance was originally designed to have three legs.

Figure 1: We propose a new conditional AD task that aims to identify and
localize anomalies in a query image by comparing it to a reference shape. The
anomalous region is shown in a yellow bounding box. For instance, the right
leg of the blue sofa is rectangular unlike the cylindrical one in its
reference shape.

Motivated by the intuition above, this paper introduces a novel _conditional_
AD task, along with a new benchmark and an effective solution, that aims to
identify and localize anomalies from a photo of an object instance (i.e., the
_query image_), in relation to a reference 3D model (see Fig. 1). The 3D model
provides the reference shape for the regular object instance, and hence a
clear definition of regularity for the query image. This setting is motivated
by real-world applications in inspection and quality control, where an object
instance is manufactured based on a reference 3D model, which can then be used
to identify anomalies (_e.g_., production faults, damages) from a photo of the
instance.

The proposed task goes beyond the single image analysis in standard AD
benchmarks and requires the detection of subtle anomalies in shape by
comparing two modalities, an image with its reference 3D model, which is
challenging for three reasons. First, we would like our model to detect
anomalies in previously unseen object instances from image-shape pairs at test
time. Generalizing to unseen instances demands learning rich representations
encoding a diverse set of 3D shapes and appearances while enabling accurate
localization of anomalies. Second, the reference 3D model contains only shape
but not texture information to simulate a realistic scenario where the 3D
model can be used to produce instances with different materials, colors, and
textures. The resulting domain gap between two modalities requires learning
representations that are invariant to such appearance changes and sensitive to
variations in geometry. Finally, in our benchmark, the viewpoint of the object
instances in query images is not available in training. This requires the
model to establish the local correspondences between the modalities, i.e.,
corresponding 3D location for each image patch in an unsupervised manner.

To tackle the first challenge, we propose a new large-scale dataset,
_BrokenChairs-180K_ , consisting of around 180$K$ query images with diverse
anomalies, geometries, and textures paired with 8,143 reference 3D shapes.
Training on such a diverse dataset enables learning rich multi-modal
representations to generalize to unseen objects. To address the domain gap
between the query images and reference shapes, we follow two strategies.
First, we render each reference shape from multiple viewpoints to generate a
set of multi-view images to represent the 3D shape and use them as input along
with a query image to our model. The multi-view representation facilitates
learning domain-invariant representations through sharing the same encoder
across query and multi-view images. Second, our model, _Correspondence
Matching Transformer_ (_CMT_) learns to capture cross-modality relationships
by applying a novel cross-attention mechanism through a sparse set of local
correspondences. Finally, to address the third challenge, we use an auxiliary
task that forces the model to learn viewpoint invariant representations for
each local patch in query and multi-view images enabling our method to align
local features corresponding to the same 3D location regardless of its
viewpoint without ground truth correspondences.

In summary, our main contributions are threefold, introducing a novel AD task,
a large-scale benchmark to provide a testbed for future research, and a
customized solution. Our model includes multiple technical innovations
including a hybrid 2D-3D representation for 3D shapes, a transformer-based
architecture that jointly learns to densely align query and multi-view images
from image-level supervision and detect anomalies. Our results in extensive
ablation studies clearly demonstrate that 3D information along with
correspondence matching yields significant improvements. We also perform an
additional perceptual study that evaluates the human performance on the task,
showing that the proposed task is challenging. Finally, we evaluate our
technique on real images showing promising results.

##  2 Related Work

AD methods. We refer to [9, 26] for detailed literature reviews. Unlike the
standard AD techniques, we focus on a conditional and multi-modal AD problem
which requires a joint analysis of a query image with a reference 3D shape to
detect local irregularities in the image.

Figure 2: Example anomaly instances from our _BrokenChairs-180K_ dataset. Our
dataset consists of around 100$K$ anomaly images. In the top row, some example
anomaly instances are shown, along with the ground truth bounding boxes and
segmentation masks in the bottom row. The red mask is used to indicate parts
with anomalies, and a green contour line highlights their respective regions
prior to applying any anomaly, and the bounding box is shown as blue
rectangular boxes. (figure best viewed in zoom)

Conditional/referential AD. In many AD applications, the anomaly of an
instance depends on its specific context [32]. For instance, anomalous
temperature changes can be more accurately detected in a particular spatial
and temporal context. We also study a specific application of the conditional
AD problem where the context information is instance-specific and comes from a
reference 3D shape.

AD image benchmarks. A major problem in the development of AD is the lack of
large datasets with realistic anomalies. For semantic anomalies, a common
practice (_e.g_., [7, 30]) is to select an arbitrary subset of classes from an
existing classification dataset (_e.g_., MNIST [20], CIFAR10 [18]), treat them
as an anomalous class, and train a model only on the remaining classes. There
also exist multiple datasets that contain real-world anomalous instances
including irregularly shaped objects [31], objects with various defects such
as scratches, dents, contaminations [3], various defects in nanofibrous
material [6] which focus on one sample at a time. A concurrent work, PAD [38]
targets a similar objective with ours, while our task has fewer assumptions
and is designed to detect fine-grained geometrical anomalies. Moreover,
compared to the PAD dataset consisting of only 20 LEGO bricks of animal toys,
ours comprises a large-scale collection of realistic chairs with diverse
geometries, textures, and a wider range of fine-grained anomalies.

2D-3D cross-modal correlation. Image-based 3D shape retrieval [13, 14, 23] is
a related problem that aims to retrieve the most similar shape for a given 2D
image. Most existing works learn to embed 2D images and 3D shapes into a
common feature space and perform metric learning using a triplet loss.
Different to the retrieval task that primarily involves global-level matching,
our focus is comprehending the correlation of fine-grained local details
between the shape and the image to detect anomalies within the image. Another
related area focuses on learning of 2D-3D correspondences [12, 28, 36, 21] by
matching 2D and 3D locally with a triplet loss [12, 28], matching images and
point clouds with a coarse-to-fine approach [21], improving matching
robustness using a global-to-local Graph Neural Network [36]. 2D-3D
correlation is also studied for specific applications such as object pose
estimation [22, 37], 3D shape estimation [15] and object detection in images
by using a set of 3D models [2]. Unlike the methods discussed here, our
objective is to identify and localize anomalies in a given 2D query image in
relation to a reference 3D model.

##  3 Building _BrokenChairs-180K_ Dataset

To the best of our knowledge, there is no prior large public dataset with
paired 3D shapes and images. Hence we introduce _BrokenChairs-180K_ , a new
benchmark for the proposed conditional AD task. Our dataset focuses on
generating samples from one category, namely ‘chair’, which includes various
subcategories like sofas, office chairs, and stools, while our generation
pipeline is general and applicable to other categories. We picked this
category as chairs contain a very wide range of shapes, appearances, and
material combinations making them appealing for our experiments. In the
following, we describe the generation procedure, including anomaly creation
and realistic image rendering. More details can be found in the supplementary.

###  3.1 Creating Anomaly from 3D Objects

3D shape collection. To cover a wide variety of fine-grained anomalies across
various parts of chairs (_e.g_., leg, arm, and headrest), we strive to collect
3D shapes that come with part annotations and thus opt to utilize the PartNet
[25] as our starting point. PartNet is a large-scale dataset of 3D objects
annotated with fine-grained part labels. Its chair category is among the most
populous, providing a rich source of 3D shapes for our task. In particular, we
use 8,143 3D chair shapes from PartNet. Given a 3D model of a chair and its
part annotation, we automatically create anomalies by applying geometric
deformations as described below.

Generation of anomaly shapes. Our dataset covers five anomaly scenarios (see
Fig. 2) relevant to real-world applications. (1) Positional anomalies pertain
to deviations from the designated position of chair parts. To create a
positional anomaly, we randomly select a part from a normal 3D model and apply
random translation. (2) Rotational anomalies are created by applying a 3D
rotational transformation to a randomly selected 3D part. (3) Broken or
damaged parts consist cases that structural components are broken or damaged.
We synthetically generate breaks using Boolean subtraction following [19]
where we fracture a chair part by subtracting a random spherical or cubical
geometric primitive from the part mesh. (4) Component swapping involves
swapping common parts across different chair instances (_e.g_., ‘back-
connector’ of one chair is exchanged with a ‘back-connector’ from another
chair), simulating an incorrect assembly during manufacturing. (5) Missing
components involve randomly choosing one part and removing it from the 3D
shape. Next, we discuss the generation of query images with photo-realistic
texture.

Table 1: Dataset statistics. The first row shows the number of distinct 3D
chair instances utilized in our dataset, the following row indicates the total
number of images rendered from these shapes, and the third row denotes [min.,
max., median] values corresponding to the number of views rendered for each
shape object.

| Different types of Anomalies |  |
---|---|---|---
| Position | Rotation | Broken | Swapped | Missing | #Anomaly | #Normal
#Shapes | 5,646 | 5,551 | 6,113 | 6,427 | 5,182 | 28,919 | 8,143
#Images | 20,023 | 20,017 | 20,008 | 20,010 | 20,023 | 100,076 | 77,994
#Views | [1,7,4] | [1,7,4] | [1,5,4] | [1,7,4] | [1,8,4] | [1,8,4] | [1,12,8]

###  3.2 Photo-realistic Rendering of 3D objects

Assigning materials to 3D shapes. The shapes in PartNet only contain basic
textures but no realistic materials. To enable realistic rendering, we use
photo-realistic relightable materials from [27] represented as SVBRDF. In
total, we utilize 400 publicly available SVBRDF materials, encompassing
various types such as wood, plastic, leather, fabric, and metal. Following
PhotoShape [27], we automatically assign a material to each semantic part of a
3D shape, and use Blender’s “Smart UV projection” algorithm to estimate the UV
maps needed for texturing.

Rendering and view selection. We render each shape from various viewpoints
sampled from a hemisphere around the object. The viewpoint is parameterized in
spherical coordinates where azimuth values are sampled uniformly over
$\left[0,2\pi\right)$ with an interval of $\pi/10$ and elevation values are
uniformly sampled in $[\pi/9,2\pi/9]$. The radius is fixed at $2.5$ for all
views. For anomaly shapes, we only keep the rendering if the anomalous part is
visible from the camera view. We employ a quality control and verification
step (see supplementary) to discard bad-quality samples.

Figure 3: Overall architecture of our proposed CMT framework for conditional
AD task. Our CMT takes the following inputs: the query image $\bm{q}$ and the
rendered multi-view images $\\{\bm{v}_{n}\\}_{n=1}^{N}$. We extract query
features $\bm{f}^{q}$ and multi-view features $\bm{F}^{v}$ using the encoder
$\varphi$. Additionally, we use 3D positional encoding (3DPE) to obtain 3D
positional features $\bm{P}^{v}$ for the multi-view images. Next, $\bm{F}^{v}$
and $\bm{P}^{v}$ are concatenated and fed to the correspondence-guided
attention (CGA) network, denoted as $\phi$, along with the query features
$\bm{f}^{q}$. The CGA network selectively conditions the final prediction on a
small subset of the most related patches from multi-view images through a
top-$k$ sparse cross-attention (TKCA) mechanism. The view-agnostic local
feature alignment (VLFA) serves to align the encoder output features to
achieve view-agnostic representation through semi-supervised learning.

Dataset Statistics. Our dataset comprises a total of 8,143 reference 3D shapes
(normal), along with around 180$K$ images rendered at a resolution of
$256\times 256$ pixels. Among these images, 100$K$ contains anomalies, while
the remaining are categorized as normal. Since in our solution, we use
textureless multi-view images to represent the reference 3D shape, we further
provide grayscale multi-view images111In practice, we clone grayscale values
and convert each view to a three-channel image before feeding them as input to
our model. rendered from 20 regularly sampled viewpoints for each reference
shape. However, the 3D representation is not necessary to be multi-view
images, alternative representations like mesh, point cloud, or voxel can be
obtained from the reference shape and adopted by future algorithms when
solving the conditional AD problem.

A detailed breakdown of these statistics is provided in Tab. 1. We divided the
dataset into three distinct sets: 138$K$ images for training, 13$K$ for
validation, and 26$K$ for testing. Each set contains rendered images from a
set of mutually exclusive 3D shapes. Hence, the evaluation is performed on
previously unseen 3D shapes. Our dataset also contains bounding box and
segmentation mask, localizing any anomalous region.

##  4 Proposed Method

###  4.1 Overview

Let $\bm{q}\in\mathbb{R}^{3\times H\times W}$ be an $H\times W$ dimensional
RGB image of an object captured from an unknown viewpoint and
$\mathcal{V}=\\{\bm{v}_{n}\\}_{n=1}^{N}$ be a set of $H\times W$ dimensional
images that are rendered from the reference shape at $N$ regularly sampled
viewpoints on a hemisphere. We assume the model has access to the camera pose
and depth map of each multi-view image. We wish to learn a classifier
$\psi:\mathbb{R}^{3\times H\times W}\times\mathbb{R}^{N\times 3\times H\times
W}\rightarrow[0,1]$ that takes in $\bm{q}$ and $\mathcal{V}$ and predicts the
ground-truth binary anomaly label $y\in\\{0,1\\}$. Given a labeled training
set $\mathcal{D}$ including $|\mathcal{D}|$ query, multi-view, and label
triplets $(\bm{q},\mathcal{V},y)$, the classifier can be optimized by
minimizing the loss term:

$\mathcal{L}_{\text{bce}}(\mathcal{D})=\sum_{(\bm{q},\mathcal{V},y)\in\mathcal{D}}\ell_{\text{bce}}(\psi(\bm{q},\mathcal{V}),y)$
(1)

where $\ell_{\text{bce}}$ is the binary cross-entropy loss function.

An ideal classifier $\psi$ must identify subtle shape irregularities in
$\bm{q}$ by finding the relevant patches in $\mathcal{V}$ for each patch in
$\bm{q}$ and comparing them. One straightforward design to relate patches
across query and multi-view images is to use the cross-attention module [34].
In particular, one can use local features extracted from $\bm{q}$ as query and
ones from $\mathcal{V}$ as key and value matrices as input to the scaled dot-
product attention in [34] to cross-correlate them while predicting the anomaly
label. While this design can implicitly capture such cross-correlations
between patches from only image-level supervision when trained with the loss
in Eq. 1, it fails to perform better than a similar model that is trained
_only_ on the query images in practice (see Sec. 5). We posit that the failure
to utilize $\mathcal{V}$ is due to the difficulty in establishing the correct
correspondences from noisy correlations between all patches pairs across query
and multi-view images only from image-level supervision.

To address this challenge, we propose a new model, correspondence matching
transformer (CMT) that consists of a CNN encoder, a 3D positional encoding
(3DPE) module, a correspondence-guided attention (CGA) network, and lastly a
view-agnostic local feature alignment (VLFA) mechanism (see Fig. 3). While the
3DPE module encodes the 3D location of the patches in multi-view images and
facilitates finding local correspondences across views, the CGA network
selectively conditions the final prediction on a small subset of the most
related patches from multi-view images through a top-k sparse cross-attention
(TKCA) mechanism. Finally, VLFA provides a richer supervision signal to
establish correspondences between similar regions in the query and the multi-
view images by using semi-supervised learning. Next, we describe them in
detail.

Figure 4:  Our proposed correspondence-guided attention (CGA). The CGA
comprises $B$ transformer-based blocks, each consisting of a standard self-
attention module followed by a top-$k$ sparse cross-attention (TKCA) module.

###  4.2 Correspondence Matching Transformer

CMT uses ResNet18 feature pyramid network [24] as the feature encoder, which
is denoted as $\varphi:\mathbb{R}^{3\times H\times
W}\rightarrow\mathbb{R}^{d\times h\times w}$ where the input is down-scaled 8
times through the network ($h=H/8$ and $w=W/8$). Once we extract the features
of $\bm{q}$ and for each $\bm{v}$, $\varphi(\bm{q})$ and $\varphi(\bm{v})$
respectively, we reshape each of them to be $d\times n^{q}$ dimensional
matrices, denote them as $\bm{f}^{q}$ and $\bm{f}^{v}$ respectively, where
$n^{q}=h\times w$. Each column in $\bm{f}^{q}$ and $\bm{f}^{v}$ corresponds to
a $d$ dimensional local feature. We use $\bm{f}[.j]$ notation to indicate
$j$-th local feature or patch encoding, as each encoding approximately
corresponds to a local patch in the input image due to locality in the
convolutional encoder. Next, we describe key components of the CMT including
the 3DPE and CGA modules.

#### 3D Positional Encoding (3DPE).

While the multi-view representation allows for a simple and efficient model
design through a shared feature encoder for our task, it also makes 3D
information less accessible and hence hampers relating local features across
different views accurately. To mitigate this problem, we propose complementing
the multi-view images with 3D information. For each patch encoding
$\bm{f}^{v}[.j]$, we first locate the corresponding image patch in $\bm{v}$
and then compute the 3D position of the corresponding patch
$\bm{x}_{j}\in\mathbb{R}^{3}$ in the world coordinates 3D using the known
camera parameters and depth maps. Then we use Fourier encoding to obtain a
higher-dimensional vector for each $\bm{x}_{j}$ and further process it through
an MLP block to obtain a $d$ dimensional 3DPE. Formally, we denote the joint
mapping by $\gamma:\mathbb{R}^{3}\rightarrow\mathbb{R}^{d}$.

Compared to the 2D standard positional encoding used in transformer models
[11], 3DPE encodes 3D object geometry in the world space. For each
$\bm{f}^{v}$ including $n^{q}$ patch encodings, we compute a corresponding
$d\times n^{q}$ dimensional matrix $\bm{p}^{v}$. For the next steps, we gather
$\bm{f}^{v}$ and $\bm{p}^{v}$ over $N$ views, and concatenate each set along
their second dimensions, resulting in $\bm{F}^{v}\in\mathbb{R}^{d\times
n^{v}}$ and $\bm{P}^{v}\in\mathbb{R}^{d\times n^{v}}$ respectively where
$n^{v}=N\times n^{q}$. Augmenting $\bm{F}^{v}$ with $\bm{P}^{v}$ results in a
novel hybrid 2D-3D representation by incorporating explicit 3D information
into the 2D multi-view images.

#### Correspondence-Guided Attention (CGA).

The CGA network $\phi$, as illustrated in Fig. 4, takes in $\bm{f}^{q}$,
$\bm{F}^{v}$, $\bm{P}^{v}$ and predicts the anomaly label while efficiently
computing the correlations across two modalities. CGA comprises $B$
consecutive transformer blocks where each block contains multiple operations
and is indexed by subscript $b$. In particular, the block $b$ starts with
concatenating $\bm{F}^{v}$ and $\bm{P}^{v}$ along their first dimension, then
the resulting $2d\times n^{v}$ dimensional matrix is reduced to $d\times
n^{v}$ dimensional $\bar{\bm{F}}^{v}$ matrix through a linear projection layer
$\alpha^{(b)}:\mathbb{R}^{2d}\rightarrow\mathbb{R}^{d}$ (Eq. 2). After self-
attention operation (SA) is applied to the query features $\bm{f}^{q}_{(b)}$
where $\bm{f}^{q}_{(1)}=\bm{f}^{q}$ (Eq. 3), it computes the query
$\bm{Q}_{(b)}\in\mathbb{R}^{d\times n^{q}}$ (Eq. 4) and key-value matrices
$\bm{K}_{(b)}\in\mathbb{R}^{d\times n^{v}}$,
$\bm{V}_{(b)}\in\mathbb{R}^{d\times n^{v}}$ (Eq. 5) by applying the linear
projections $\bm{W}^{Q,K,V}\in\mathbb{R}^{d\times d}$ respectively.

$\displaystyle\bar{\bm{F}}^{v}_{(b)}\leftarrow\alpha_{(b)}(\begin{bmatrix}\bm{F}^{v}\\\
\bm{P}^{v}\end{bmatrix})$ (2)
$\displaystyle\bar{\bm{f}}^{q}_{(b)}\leftarrow\texttt{SA}(\bm{f}^{q}_{(b)})$
(3) $\displaystyle\bm{Q}_{(b)}\leftarrow\bm{W}^{Q}\bar{\bm{f}}^{q}_{(b)}$ (4)
$\displaystyle\bm{K}_{(b)}\leftarrow\bm{W}^{K}\bar{\bm{F}}^{v}_{(b)},\quad\bm{V}_{(b)}\leftarrow\bm{W}^{V}\bar{\bm{F}}^{v}_{(b)}$
(5)
$\displaystyle\bm{O}_{(b)}\leftarrow\texttt{TKCA}{(\bm{Q}_{(b)},\bm{K}_{(b)},\bm{V}_{(b)},\bm{M})}$
(6)
$\displaystyle\bm{O}_{(b)}\leftarrow\texttt{Norm}{(\bm{O}_{(b)}+\bm{Q}_{(b)})}$
(7)
$\displaystyle\bm{O}_{(b)}\leftarrow\texttt{Norm}{(\texttt{FFN}(\bm{O}_{(b)})+\bm{O}_{(b)})}$
(8) $\displaystyle\bm{f}^{q}_{(b+1)}\leftarrow\bm{O}_{(b)}$ (9)

Next, we pass $\bm{Q}_{(b)},\bm{K}_{(b)},\bm{V}_{(b)}$ to our top-k sparse
cross-attention (TKCA) module (see Eq. 6). Unlike the vanilla cross-attention
module in standard transformers [34, 11] ingesting all tokens for the
attention computation, which is inefficient for our task and may introduce
noisy interactions with irrelevant features, potentially degrading
performance, TKCA calculates the attention between query and only a small
subset of relevant multi-view features using a similarity matrix $\bm{M}$:

$\texttt{TKCA}{(\bm{Q},\bm{K},\bm{V},\bm{M})}=\textup{softmax}\left(\mathcal{T}^{\bm{M}}_{k}(\frac{\bm{Q}\bm{K}}{\sqrt{d}})\right)\bm{V}$
(10)

where $\mathcal{T}^{\bm{M}}_{k}$ is given by:

$\mathcal{T}^{\bm{M}}_{k}(\mathbf{A})[ij]=\begin{cases}A[ij],&\text{if
}\bm{M}[ij]\in\texttt{top}_{\textit{k}}(\bm{M}[i.])\\\
-\infty,&\text{otherwise}\end{cases}$ (11)

where $\texttt{top}_{\textit{k}}(\bm{M}[i.])$ operation selects the $k$ most
similar features from multi-view representation (see Fig. 5) for $i$-th query
feature. To compute $\bm{M}$, we use an auxiliary function
$\beta:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$, instantiated as a four
layered MLP followed by a final channel-wise normalization, that projects
$\bm{f}^{q}$ and each view in $\bm{F}^{v}$ to a view-agnostic feature space
where images corresponding to same object part in 3D has similar
representations regardless of their viewpoint. To obtain the similarity
between the query and multi-view patches, we compute the dot product between
their projected features:

$\bm{M}=\beta(\bm{f}^{q})^{\mathrm{T}}\beta(\bm{F}^{v})\in\mathbb{R}^{n^{q}\times
n^{v}}.$ (12)

In contrast to other transformer architectures using sparse attention [35],
TCKA chooses the top-$k$ elements based on a different source of information,
geometric correspondences computed across two modalities, and enables an
efficient computation of the cross-attention, as the same $\bm{M}$ is used
throughout the transformer blocks.

Figure 5: Top-$k$ sparse attention-span visualization. For the query point
(yellow), similarity heatmaps (first row) and top-$k$ attention-span (second
row) across multiple views are shown.

After the cross-correlation, the standard residual addition, normalization and
feedforward (FFN) layers are applied to obtain $\bm{f}^{q}$ as input to the
next block $b+1$ (Eqs. 7, 8 and 9). Note that we use multiple heads,
concatenate the outputs from multi-head attention and then derive the final
attention results through linear projection. We append a learnable token
denoted as [tok] to construct the query inputs of the CGA network. Through the
transformer blocks, the output state of the [tok] token develops a
consolidated representation enriched by learned shape-image correlation, which
is used as input to the classification head.

###  4.3 View-Agnostic Local Feature Alignment

As discussed above, image-level supervision alone is too weak to capture fine
localized correlations between $\bm{q}$ and $\mathcal{V}$. Thus, we introduce
an auxiliary task, VLFA that aims to densely align corresponding parts between
query images and related views. Through $\beta$, we learn to map $\bm{f}^{q}$
and $\bm{f}^{v}$ to a view-agnostic space such that their local features
corresponding to the same object part are mapped to a similar point regardless
of the viewpoint from which the image is captured. As the viewpoint of
$\bm{q}$ is unknown, the ground-truth correspondences between query and
reference views cannot be obtained through inverse rendering.

To this end, we use a self-labeling strategy to generate pseudo-
correspondences by finding the most similar local feature in the reference
view to each local feature in the query at each training step, after mapping
their features to the view-invariant space and normalizing them:

$\hat{c}_{i}=\underset{j}{\textup{argmax}}\;{\bm{z}^{q}_{i}}^{\mathrm{T}}\bm{z}^{v}_{j},$
(13)

where $\bm{z}^{q}_{i}=\frac{\beta(\bm{f}^{q}[.i])}{\|\beta(\bm{f}^{q}[.i])\|}$
and $\bm{z}^{v}_{j}=\frac{\beta(\bm{f}^{v}[.j])}{\|\beta(\bm{f}^{v}[.j])\|}$.
We compute the pseudo-label for each $\bm{z}^{q}_{i}$ and store them in a
look-up table $\hat{\mathcal{P}}(\bm{q},\bm{v},i)=\hat{c}_{i}$. In another one
$\hat{\mathcal{N}}(\bm{q},\bm{v},i)$, we store the remaining set of reference
view and index values that are not the corresponding location. Then, using
$\hat{\mathcal{P}},\hat{\mathcal{N}}$ as positive and negative correspondences
respectively, we minimize a contrastive loss
$\ell_{\textup{va}}(\bm{q},\bm{v})$ over each $\bm{q}$-$\bm{v}$ pair:

$\sum_{i=1}^{n^{q}}-\textup{log}\frac{\textup{exp}({\bm{z}^{q}_{i}}^{\mathrm{T}}\bm{z}^{v}_{+}/\tau)}{\textup{exp}({\bm{z}^{q}_{i}}^{\mathrm{T}}\bm{z}^{v}_{+}/\tau)+\underset{{j\in\hat{\mathcal{N}}(\bm{q},\bm{v},i)}}{\sum}\textup{exp}({\bm{z}^{q}_{i}}^{\mathrm{T}}\bm{z}^{v}_{j}/\tau)},$
(14)

where $\tau$ is a temperature parameter and
$\bm{z}^{v}_{+}=\bm{z}^{v}_{\hat{\mathcal{P}}(\bm{q},\bm{v},i)}$. Due to the
cost of computing the pseudo-correspondences for all query features across all
views, we compute them only for a random subset of query features across
randomly sampled views at each training iteration.

Learning correspondences through self-learning alone in the presence of the
domain gap between query and reference views is a noisy process. Hence, we
also exploit the known viewpoints of the multi-view images by densely aligning
their local features in each view pair $\bm{v},\bm{v}^{\prime}$ after
computing the ground truth dense correspondences between them and discarding
the ones that are occluded in one of the views. The key assumption here is
that aligning different views by using their ground truth labels enables a
more accurate correspondence learning between query images and views, as the
parameters of the projection $\beta$ are shared across two domains. As before,
we form two look-up tables $\mathcal{P}(\bm{v},\bm{v}^{\prime})$ and
$\mathcal{N}(\bm{v},\bm{v}^{\prime})$ to store the positive and negative
correspondences between two views, and randomly subsample them. After mapping
them to the view-invariant space and normalizing them, we compute and minimize
Eq. 14 for the pairs in the look-up tables.

Figure 6: Anomaly detection results on the test set using our proposed CMT
framework. Incorrect predictions are marked in red.

The objective in Eq. 1 can be rewritten as:

$\mathcal{L}_{\text{bce}}(\mathcal{D})+a\mathcal{L}_{\textup{va}}(\hat{\mathcal{P}},\hat{\mathcal{N}})+(1-a)\mathcal{L}_{\textup{va}}(\mathcal{P},\mathcal{N})$
(15)

where $\mathcal{L}_{\textup{va}}(\hat{\mathcal{P}},\hat{\mathcal{N}})$ and
$\mathcal{L}_{\textup{va}}(\mathcal{P},\mathcal{N})$ are the alignment loss
functions over query-view pairs and view-view pairs respectively, $a$ is a
loss balancing weight set to $0.5$.

##  5 Experiments

Implementation Details: The encoder $\varphi$ takes a $3\times 256\times 256$
image as input and returns a $128\times 32\times 32$ feature block. Within the
CGA network, we employ three transformer blocks ($B=3$), and each applies
8-headed attention. The value of $k$ in TCKA is set to $100$. During training,
we randomly select a subset of $N=10$ views, and during testing, we utilize
all 20 views. We apply basic data augmentation to the query images, which
includes random horizontal flips and random cropping of $224\times 224$
regions, followed by resizing the cropped regions back to the original size of
$256\times 256$. We train our model for 20 epochs using 4 Titan RTX GPUs,
maintaining a batch size of 8 in each GPU, and use the Adam optimizer with a
learning rate of $2\times 10^{-5}$. We refer to the supplementary for more
details.

###  5.1 Results

Since there is no related public benchmark for our task, we define several
challenging baselines to evaluate our CMT. We report the quantitative results
using two evaluation metrics – the area under the ROC curve (AUC) and accuracy
in Tab. 2, and provide qualitative results in Fig. 6.

Importance of 3D reference shape. To assess the significance of using the
reference shape, we establish baselines that solely rely on the query image
for detecting anomalies. As our first baseline, we use a ResNet18-FPN model
that takes in only query images as input. For the next two baselines, we add
three self-attention blocks to ResNet18-FPN and use a ViT [11] respectively.
Tab. 2 shows that the reference 3D shape is crucial to good performance while
CMT outperforms the baselines by more than 10% in accuracy.

Table 2: Quantitative Comparison of the proposed CMT framework with several
baselines in terms of area under the ROC curve (AUC) and accuracy score. For
both metrics, higher is better.

3D Ref. | Methods | AUC ($\uparrow$) | Accuracy ($\uparrow$)
---|---|---|---
✗ | ResNet18-FPN [24] | 74.6 | 64.7
ResNet18-FPN w/ SA blocks | 75.2 | 65.1
Vision Transformer (ViT) [11] | 75.4 | 65.2
✓ | LFD [CVPR’19] [14] | - | 64.9
Lin  _et al_. [ICCV’21] [23] | - | 67.8
A: CMT (w/o CGA, VLFA, 3DPE) | 76.1 | 66.8
B: CMT (w/o VLFA, 3DPE) | 76.3 | 67.1
C: CMT (w/o CGA, 3DPE) | 80.8 | 72.3
D: CMT (w/o 3DPE) | 82.6 | 73.7
Ours: CMT | 84.7 | 75.4

Table 3:  Ablation of view-agnostic alignment loss. The loss function
comprises two components: $\mathcal{L}_{\textup{va}}(\mathcal{P},\mathcal{N})$
and $\mathcal{L}_{\textup{va}}(\hat{\mathcal{P}},\hat{\mathcal{N}})$. Optimal
performance is achieved when both losses are combined.

$\mathcal{L}_{\textup{va}}(\mathcal{P},\mathcal{N})$ | $\mathcal{L}_{\textup{va}}(\hat{\mathcal{P}},\hat{\mathcal{N}})$ | AUC ($\uparrow$) | Accuracy ($\uparrow$)
---|---|---|---
✗ | ✗ | 78.5 | 68.3
✗ | ✓ | 78.6 | 68.1
✓ | ✗ | 81.6 | 73.7
✓ | ✓ | 84.7 | 75.4

Comparison with related work. As there is no prior work designed for our
problem, we take two recent image-based 3D shape retrieval techniques [14, 23]
that learn to embed 2D images and 3D shapes into a common feature space and
perform metric learning using a triplet loss. Once we train them in our
dataset, we evaluate them by using the distance between the query and
reference shape embeddings to obtain the classification score after a
thresholding step. Based on results in Tab. 2, we argue that these methods
fail to locate subtle variations in geometry, as the cross-modal correlations
are only learned at the image level missing fine-grained local correspondence
learning.

Ablation of CGA, VLFA and 3DPE. Our first baseline (A) includes none of the
three components but a standard cross-attention module to relate two
modalities using all local patches. Surprisingly, A obtains only a 1.6%
accuracy improvement over the query-only baseline, indicating its inability to
fully leverage the reference shape. Baseline B includes only the CGA component
with the top-$k$ sparse cross-attention, baseline C, contains the VLFA but
with a standard cross-attention. While baseline B does not show much
improvement over A, baseline C performs significantly better than A obtaining
a 5.2% accuracy gain. This clearly demonstrates the importance of the
auxiliary task where we learn matching correspondences for the AD task and
that the CGA fails to acquire meaningful correspondences in the absence of the
VLFA. Baseline D that employs both CGA and VLFA further boosts the performance
of C through its selective sparse attention mechanism. Finally, our model,
which includes all the components, outperforms D thanks to the introduction of
3DPE that facilitates corresponding matching across different views.

Ablation of loss functions. Tab. 3 reports an ablation study on the loss
functions used for learning the view-agnostic representation. Utilizing only
query-view alignment loss
($\mathcal{L}_{\textup{va}}(\hat{\mathcal{P}},\hat{\mathcal{N}})$) does not
yield any advantages (row 2) over not employing any alignment loss (row 1).
However, employing the view-view alignment
($\mathcal{L}_{\textup{va}}(\mathcal{P},\mathcal{N})$) alone leads to improved
results (row 3). The optimal result is achieved when both components are
combined (row 4).

Figure 7: (left) AUC score under different values of $k$. (right) Impact of
numbers of multi-view images during training and testing.

Sensitivity to $k$. We analyze the performance under different values of $k$
in Fig. 7 (left). Compared to the maximum possible $k_{\textup{max}}$ that is
$N\times 32\times 32$, we analyze significantly smaller $k$ values and, among
them, show $k=100$ yields the best result. Using all available tokens results
in deteriorated performance (shown as the dotted horizontal line) showing that
our top-$k$ sparse attention is effective in eliminating the noisy patches by
using only the $k$ top-related ones.

Sensitivity to $N$. Fig. 7 (right) depicts the analysis on the number of input
views for training and testing, $N_{\textup{train}}$ and $N_{\textup{test}}$
respectively. To this end, we train two separate CMT models with 5 and 10
views, and evaluate each using 5, 10, 15, and 20 views at test time. The plot
shows that, while increasing views in both training and testing helps,
training with few views and testing on more views can provide a good tradeoff
between training time and performance.

Figure 8: Evaluation on real data. The predicted anomalies are shown in the
blue bounding boxes.

Viewpoint prediction. As a side product of establishing the correspondences
across the query image and views in our model, we could estimate the camera
viewpoint in the query image w.r.t. a reference shape. To this end, we compute
dense correspondences between the query image and each view image, and then
calculate the distance between the pixel coordinates of each point in the
query image and their predicted correspondences in the multi-view images, and
choose the view with the lowest average distance as the approximate viewpoint.
As a baseline, we train a ResNet with the viewpoint supervision on the normal
images only, and evaluate it on the test normal query images. Our model,
trained with no viewpoint supervision, achieves a significantly better
accuracy (47% vs 89%) when predicting the closest view suggesting that our
model implicitly learns to relate the query image with the closest views.

Evaluation on real data. Here we apply our model, which is trained on the
synthetic _BrokenChairs-180K_ dataset, on a small set of real chair samples
that contain multiple pairs of the reference 3D shape, query images containing
either normal or irregular instances with broken, removed, or misaligned parts
from various viewpoints. Background pixels in query images are removed by
using a segmentation method [17] in a preprocessing step and also a synthetic
shadow was added to match the training images. To obtain the reference 3D
shapes, we take multiple photos of object instances while walking around them,
use the 3D reconstruction software [33], and finally apply Laplacian smoothing
to post-process it. Fig. 8 illustrates the results for two regular reference
shapes, each paired with three query images. In 5 out of 6 cases, our method
successfully classifies and localizes the anomalous parts, while in the
failure case, it incorrectly relates the self-occluded arm with an anomaly.

Anomaly localization. Here we adopt our model to localize anomalies in the
form of a bounding box, use a bounding box regression head (a 4-layer MLP),
and jointly train it with the other network parameters by using L1 regression
and generalized IoU loss [29]. This model achieves 56.5% average precision on
our dataset, outperforms a ViT baseline that is trained only on the query
images, and obtains 42.6%. Moreover, jointly learning the classification and
localization further boosts the classification performance to 85.9 (+1.2) AUC
and 77.3% (+1.9) accuracy.

User Perceptual Study. We also evaluated human performance in our task and
conducted a study with 100 participants. We presented each participant with 10
pairs of reference shapes and query images, with each pair randomly selected
from a random subset of 200. We observe a human accuracy of 70.6%, showing
that the proposed task is challenging, while our CMT obtains a superior
accuracy of 74.8% on the same subset.

##  6 Conclusion

In this paper, we have introduced a new AD task, a new benchmark, and a
customized solution inspired by quality control and inspection scenarios in
manufacturing. We showed that an accurate detection of fine-grained anomalies
in geometry requires a careful study of both modalities jointly. Our method
achieves this goal by learning dense correspondence across those modalities
from limited supervision. Our benchmark and method also have limitations. Due
to the difficulty and cost of obtaining real damaged objects, our dataset
contains only shapes and images of synthetic objects and currently is limited
to a single yet very diverse category of ‘chair’, presence of only one anomaly
in each query image, focusing only on shape anomalies excluding the appearance
based ones such as fading, discolor, texture anomaly. Moreover, our model
assumes that object instances are rigid, and cannot deal with articulations
and deformations, and requires an accurate reference 3D shape for accurate
detection.

## References

  * Ahmed and Courville [2020] Faruk Ahmed and Aaron Courville.  Detecting semantic anomalies.  In _AAAI_ , 2020.
  * Aubry et al. [2014] Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan C Russell, and Josef Sivic.  Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models.  In _CVPR_ , 2014.
  * Bergmann et al. [2019] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger.  Mvtec ad–a comprehensive real-world dataset for unsupervised anomaly detection.  In _CVPR_ , 2019.
  * Bergmann et al. [2021] Paul Bergmann, Kilian Batzner, Michael Fauser, David Sattlegger, and Carsten Steger.  The mvtec anomaly detection dataset: a comprehensive real-world dataset for unsupervised anomaly detection.  _IJCV_ , 2021.
  * Blum et al. [2021] Hermann Blum, Paul-Edouard Sarlin, Juan Nieto, Roland Siegwart, and Cesar Cadena.  The fishyscapes benchmark: Measuring blind spots in semantic segmentation.  _IJCV_ , 2021.
  * Carrera et al. [2016] Diego Carrera, Fabio Manganini, Giacomo Boracchi, and Ettore Lanzarone.  Defect detection in sem images of nanofibrous materials.  _IEEE Transactions on Industrial Informatics_ , 2016.
  * Chalapathy et al. [2018] Raghavendra Chalapathy, Aditya Krishna Menon, and Sanjay Chawla.  Anomaly detection using one-class neural networks.  _arXiv preprint arXiv:1802.06360_ , 2018.
  * Chan et al. [2021] Robin Chan, Krzysztof Lis, Svenja Uhlemeyer, Hermann Blum, Sina Honari, Roland Siegwart, Pascal Fua, Mathieu Salzmann, and Matthias Rottmann.  Segmentmeifyoucan: A benchmark for anomaly segmentation.  _arXiv preprint arXiv:2104.14812_ , 2021.
  * Chandola et al. [2009] Varun Chandola, Arindam Banerjee, and Vipin Kumar.  Anomaly detection: A survey.  _ACM computing surveys_ , 2009.
  * Deecke et al. [2021] Lucas Deecke, Lukas Ruff, Robert A Vandermeulen, and Hakan Bilen.  Transfer-based semantic anomaly detection.  In _ICML_ , 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Feng et al. [2019] Mengdan Feng, Sixing Hu, Marcelo H Ang, and Gim Hee Lee.  2d3d-matchnet: Learning to match keypoints across 2d image and 3d point cloud.  In _ICRA_ , 2019.
  * Grabner et al. [2018] Alexander Grabner, Peter M Roth, and Vincent Lepetit.  3d pose estimation and 3d model retrieval for objects in the wild.  In _CVPR_ , 2018.
  * Grabner et al. [2019] Alexander Grabner, Peter M Roth, and Vincent Lepetit.  Location field descriptors: Single image 3d model retrieval in the wild.  In _3DV_ , 2019.
  * Hejrati and Ramanan [2012] Mohsen Hejrati and Deva Ramanan.  Analyzing 3d objects in cluttered images.  _NeurIPS_ , 2012.
  * Hendrycks and Gimpel [2017] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _ICLR_ , 2017.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Krizhevsky and Hinton [2009] Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  2009\.
  * Lamb et al. [2022] Nikolas Lamb, Sean Banerjee, and Natasha Kholgade Banerjee.  Deepjoin: Learning a joint occupancy, signed distance, and normal field function for shape repair.  In _ACM TOG_ , 2022.
  * LeCun et al. [1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 1998.
  * Li et al. [2023] Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chenyang Zhu, Yulan Guo, and Kai Xu.  2d3d-matr: 2d-3d matching transformer for detection-free registration between images and point clouds.  In _ICCV_ , 2023.
  * Lim et al. [2013] Joseph J Lim, Hamed Pirsiavash, and Antonio Torralba.  Parsing ikea objects: Fine pose estimation.  In _ICCV_ , 2013.
  * Lin et al. [2021] Ming-Xian Lin, Jie Yang, He Wang, Yu-Kun Lai, Rongfei Jia, Binqiang Zhao, and Lin Gao.  Single image 3d shape retrieval via cross-modal instance and category contrastive learning.  In _ICCV_ , 2021.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _CVPR_ , 2017.
  * Mo et al. [2019] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su.  Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding.  In _CVPR_ , 2019.
  * Pang et al. [2021] Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel.  Deep learning for anomaly detection: A review.  _ACM computing surveys_ , 2021.
  * Park et al. [2018] Keunhong Park, Konstantinos Rematas, Ali Farhadi, and Steven M Seitz.  Photoshape: Photorealistic materials for large-scale shape collections.  In _SIGGRAPH Asia_ , 2018.
  * Pham et al. [2020] Quang-Hieu Pham, Mikaela Angelina Uy, Binh-Son Hua, Duc Thanh Nguyen, Gemma Roig, and Sai-Kit Yeung.  Lcd: Learned cross-domain descriptors for 2d-3d matching.  In _AAAI_ , 2020.
  * Rezatofighi et al. [2019] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In _CVPR_ , 2019.
  * Ruff et al. [2018] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft.  Deep one-class classification.  In _ICML_ , 2018.
  * Saleh et al. [2013] Babak Saleh, Ali Farhadi, and Ahmed Elgammal.  Object-centric anomaly detection by attribute-based reasoning.  In _CVPR_ , 2013.
  * Song et al. [2007] Xiuyao Song, Mingxi Wu, Christopher Jermaine, and Sanjay Ranka.  Conditional anomaly detection.  _IEEE TKDE_ , 2007.
  * Tancik et al. [2023] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al.  Nerfstudio: A modular framework for neural radiance field development.  In _ACM SIGGRAPH_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NeurIPS_ , 2017.
  * Wang et al. [2022] Pichao Wang, Xue Wang, Fan Wang, Ming Lin, Shuning Chang, Hao Li, and Rong Jin.  Kvt: k-nn attention for boosting vision transformers.  In _ECCV_ , 2022.
  * Wang et al. [2023] Shuzhe Wang, Juho Kannala, and Daniel Barath.  Dgc-gnn: Descriptor-free geometric-color graph neural network for 2d-3d matching.  _arXiv preprint arXiv:2306.12547_ , 2023.
  * Xiao et al. [2012] Jianxiong Xiao, Bryan Russell, and Antonio Torralba.  Localizing 3d cuboids in single-view images.  _NeurIPS_ , 2012.
  * Zhou et al. [2024] Qiang Zhou, Weize Li, Lihan Jiang, Guoliang Wang, Guyue Zhou, Shanghang Zhang, and Hao Zhao.  Pad: A dataset and benchmark for pose-agnostic anomaly detection.  In _NeurIPS_ , 2024.