sOmm#1 . #3 |_#4 #3#2|_#4

111This paper was accepted for publication in the Journal of Transportation
Engineering, Part B: Pavements. DOI: 10.1061/JPEODX.PVENG-1401.

# SIFT-Aided Rectified 2D-DIC for Displacement and Strain Measurements in
Asphalt Concrete Testing

Zehui Zhu  Imad L. Al-Qadi  Department of Civil and Environmental Engineering,
University of Illinois Urbana-Champaign. Email<EMAIL_ADDRESS>
###### Abstract

Two-dimensional digital image correlation (2D-DIC) is a widely used optical
technique to measure displacement and strain during asphalt concrete (AC)
testing. An accurate 2-D DIC measurement can only be achieved when the
camera’s principal axis is perpendicular to the planar specimen surface.
However, this requirement may not be met during testing due to device
constraints. This paper proposes a simple and reliable method to correct
errors induced by non-perpendicularity. The method is based on image feature
matching and rectification. No additional equipment is needed. A theoretical
error analysis was conducted to quantify the effect of a non-perpendicular
camera alignment on measurement accuracy. The proposed method was validated
numerically using synthetic images and experimentally in an AC fracture test.
It achieved relatively high accuracy, even under considerable camera rotation
angle and large deformation. As a pre-processing technique, the proposed
method showed promising performance in assisting the recently developed
CrackPropNet for automated crack propagation measurement under a non-
perpendicular camera alignment.

##  1 Introduction

Digital image correlation (DIC) is an optical displacement and strain
measurement technique. Since its introduction for asphalt concrete (AC)
application, DIC has become a vital tool in evaluating AC material properties,
validating test protocols, and verifying theoretical models [Chehab et al.
(2007), Birgisson et al. (2008), Birgisson et al. (2009), Safavizadeh et al.
(2017), Rivera-Pérez et al. (2021)]. The most popular DIC algorithms include
the two-dimensional (2D)-DIC and stereo-DIC. The 2D-DIC uses a single fixed
camera to measure the in-plane displacement of nominal planar objects.
However, the 2D-DIC method is not applicable when the test specimen surface is
non-planar, or the out-of-plane displacement is non-negligible. On the other
hand, the stereo-DIC method, which uses two synchronized cameras, was proposed
to measure 3D displacement based on the binocular stereo-vision principle [Pan
et al. (2009)].

The 2D-DIC remains one of the most popular optical technologies because of the
following advantages:

  1. 1.

_Simple experimental setup_ : For 2D-DIC, only a single charge-coupled device
(CCD) camera is needed. In contrast, the stereo-DIC requires two synchronized
cameras [Pan (2018)].

  2. 2.

_Simple algorithm_ : The computation of the displacement field in 2D-DIC
solely relies on the correlation between deformed images captured after
deformation and a reference image captured before deformation. On the other
hand, stereo-DIC entails a complex procedure that includes stereo-calibration
of the two-camera unit, stereo matching (i.e., cross-camera matching),
temporal matching, and triangulation for reconstructing 3D coordinates [Lin et
al. (2022)]. Hence, the stereo-DIC has more uncertainty due to the different
algorithms chosen in each step, thereby impacting the ultimate precision of
measurements [Balcaen et al. (2017), Zhong et al. (2019)].

However, an accurate 2-D DIC measurement can only be achieved when the
following requirements are met [Sutton et al. (2009)]:

  1. 1.

Specimen surface is planar.

  2. 2.

Out-of-plane deformation of the specimen is small enough.

  3. 3.

Imaging system does not suffer from geometric distortion.

  4. 4.

Charged-couple device sensor is parallel to the specimen surface (i.e., the
camera’s principal axis is perpendicular to the specimen surface).

The first three requirements can be easily satisfied in most AC laboratory
tests. However, equipment limitations often make the last requirement
difficult to meet. For example, as shown in Fig.1, only one of the three
testing devices for the Illinois-flexibility index test (I-FIT) offers open
space to satisfy the perpendicularity requirement [Ozer et al. (2017)].

Figure 1: I-FIT testing devices.

In the past decade, researchers have proposed a few methods to alleviate the
effect of non-perpendicularity on 2D-DIC. They can be categorized into two
groups based on their fundamental principles. The first is modifying
experimental setup or transforming images. For example, pan2013high proposed
changes to the imaging system. A bilateral telecentric lens is recommended for
its relatively high sensitivity to in-plane displacement and insensitivity to
out-of-plane displacement. Moreover, pan2013high1 suggested attaching a non-
deformable compensation specimen onto the testing surface. During loading, the
compensation specimen moves rigidly with the testing specimen. The measurement
error can be corrected by applying a parametric model fitted using the
compensation specimen’s displacement field. The second, quite differently,
applies rectification to transform images collected under a non-perpendicular
camera setting [Lava et al. (2011)]. The standard camera calibration procedure
was followed using a regular grid pattern. The camera’s extrinsic and
intrinsic parameters were determined and used to conduct the rectification.
DIC analysis was conducted on the rectified images, which can be collected
under a perpendicular camera setting.

While the mentioned approaches effectively correct errors caused by non-
perpendicularity, the simplicity of 2D-DIC is compromised by the requirement
of additional equipment, such as a special lens, compensation specimen, or
grid pattern. To fully leverage the benefits of 2D-DIC, this paper proposes a
simple and reliable method with no additional equipment needed. The proposed
method was validated numerically using synthetic images and experimentally in
an I-FIT. In addition, as a pre-processing technique, the proposed method was
applied to assist CrackPropNet in measuring crack propagation on an I-FIT
specimen surface.

##  2 Two-Dimensional DIC Principle

This section reviews the fundamentals of 2D-DIC. Fig.2 shows a 2-D DIC system
comprises of a camera, a light source, and a computer.

Figure 2: 2-D DIC setup.

The DIC works by tracking pixel movement in a series of images. It is achieved
by establishing gray value correspondences. For example, as shown in Fig.3, a
match between a square reference subset and a deformed subset is established.
The matching is attained by searching for a correlation coefficient extremum.
Equation 1 lists commonly used correlation criteria [Pan et al. (2009)].
Compared to cross-correlation (CC) and sum-of-squared differences (SSD), the
zero-normalized cross-correlation (ZNCC) and zero-normalized sum-of-squared
differences (ZNSSD) offer better performance against noise. They are less
insensitive to lighting fluctuations (e.g., offset and linear scale).

$\displaystyle CC$
$\displaystyle:\sum_{i=-M}^{M}\sum_{j=-M}^{M}f(x_{i},y_{j})g(x_{i}^{\prime},y_{j}^{\prime})$
(1) $\displaystyle ZNCC$
$\displaystyle:\frac{\sum_{i=-M}^{M}\sum_{j=-M}^{M}[f(x_{i},y_{j})-f_{m}]\times[g(x_{i}^{\prime},y_{j}^{\prime})-g_{m}]}{\sqrt{\sum_{i=-M}^{M}\sum_{j=-M}^{M}[f(x_{i},y_{j})-f_{m}]^{2}}\sqrt{\sum_{i=-M}^{M}\sum_{j=-M}^{M}[g(x_{i}^{\prime},y_{j}^{\prime})-g_{m}]^{2}}}$
$\displaystyle SSD$
$\displaystyle:\sum_{i=-M}^{M}\sum_{j=-M}^{M}[f(x_{i},y_{j})-g(x_{i}^{\prime},y_{j}^{\prime})]^{2}$
$\displaystyle ZNSSD$
$\displaystyle:\sum_{i=-M}^{M}\sum_{j=-M}^{M}[\frac{f(x_{i},y_{j})-f_{m}}{\sqrt{\sum_{i=-M}^{M}\sum_{j=-M}^{M}[f(x_{i},y_{j})-f_{m}]^{2}}}-\frac{g(x_{i}^{\prime},y_{j}^{\prime})-g_{m}}{\sqrt{\sum_{i=-M}^{M}\sum_{j=-M}^{M}[g(x_{i}^{\prime},y_{j}^{\prime})-g_{m}]^{2}}}]^{2}$

where $f(x_{i},y_{j})$ is gray value at $(x_{i},y_{j})$ in the reference
subset; $g(x_{i}^{\prime},y_{j}^{\prime})$ is gray value at
$(x_{i}^{\prime},y_{j}^{\prime})$ in the deformed subset; $f_{m}$ and $g_{m}$
are mean gray values of the reference and deformed subset, respectively.

Figure 3: Establish correspondences between a reference and a deformed images
[Pan et al. (2009)].

Once the matching is established, the in-plane displacement vector of point
$P$ can be calculated based on coordinate difference. Subsequently,
displacement of other points in the same subset (e.g., $Q$) can be computed
using shape functions $\gamma$ and $\delta$ in Eq.2. A commonly used first-
order shape function enables translation, shear, rotation, and combinations.

$\begin{cases}x_{i}^{\prime}=x_{i}+\gamma(x_{i},y_{i})\\\
y_{j}^{\prime}=y_{j}+\delta(x_{i},y_{i})\end{cases}$ (2)

The above only discussed calculating the displacement vector for a single
point. The reliability-guided (RG)-DIC algorithm, adopted by open-source
software Ncorr, is often used to attain full-field measurement [Pan et al.
(2009), Blaber et al. (2015)]. First, the algorithm finds a reasonable initial
guess of the displacement vector ($\mathbf{p}_{0}=(\gamma(x,y),\delta(x,y))$)
for a seed point. The seed point’s correlation coefficient is also calculated.
Second, the displacement vectors of the four neighboring points of the seed
point are calculated using $\mathbf{p}_{0}$ as their initial guess. Their
correlation coefficients are inserted into a max heap binary tree. Third, the
tree’s root with the highest correlation coefficient is popped, and its
$\mathbf{p}$ is used as the initial guess to calculate displacement vectors of
the uncomputed neighboring points. The third step is repeated until all points
in the region of interest (ROI) are calculated.

##  3 Research Objective

This paper addresses 2D-DIC measurement errors caused by non-perpendicularity.

A theoretical analysis was conducted to understand the displacement error
quantitatively. A simple and reliable method was proposed to correct such
errors. The proposed method was validated numerically using synthetic images
and experimentally in an I-FIT. Moreover, the proposed method was applied as a
pre-processing technique to assist CrackPropNet in measuring crack propagation
on an I-FIT specimen surface.

##  4 Theoretical Error Analysis of A Non-Perpendicular Camera Alignment

Perpendicularity is crucial in using 2D-DIC. If the camera’s principal axis is
not perpendicular to the planar specimen surface, measurement errors are
unavoidable. This section presents a theoretical study of the significant
factors affecting these errors.

As shown in Fig.4, a pinhole camera’s principal axis is rotated $\theta$ with
respect to the $y$-axis. Displacement error was assessed by assuming a point
$A(x_{A},y_{A})$ moved to $A^{\prime}(x_{A}+\Delta x,y_{A}+\Delta y)$.

Figure 4: Camera’s principal axis is not perpendicular to the planar specimen
surface.

Fig.5 shows the top view of Fig.4. $f$ refers to the focal length, the
distance between the pinhole plane and the image plane. $S$ is the distance
between the pinhole plane and the planar object surface.

Figure 5: Top view of the pinhole camera model under non-perpendicular and
perpendicular camera settings.

Under perpendicular setting (Fig.5 _(b)_), projected displacement $\Delta x$
on the image plane is presented as follows:

$\Delta x^{\prime}=\frac{\Delta x}{S}f$ (3)

Meanwhile, under non-perpendicular setting (Fig.5 _(a)_), by applying the
concept of the similar triangle, projected displacement on the image plane was
derived:

$\Delta x^{\prime\prime}=f[\frac{(x_{A}+\Delta x)\cos\theta}{S-(x_{A}+\Delta
x)\sin\theta}-\frac{x_{A}\cos\theta}{S-x_{A}\sin\theta}]$ (4)

Eq.5 shows the absolute error of projected displacement $\Delta x$ on the
image plane. The magnitude of error depends on $f$, $S$, $\theta$, $\Delta x$,
and $x_{A}$. The error increases as the focal length ($f$) increases. Fig.6
illustrates the effect of the remaining parameters. The parametric values
considered here are typical in AC laboratory tests. Due to the high-
dimensional nature of Eq.5, Fig.6 illustrates partial dependencies by fixing
other parameters. The error increases exponentially with a larger camera
rotation angle ($\theta$) and a smaller pinhole-object distance ($S$).
Moreover, a more significant displacement $\Delta_{x}$ causes a larger error.
However, the relationship between $x_{A}$ and $|Error_{\Delta x}|$ is non-
monotonic.

$|Error_{\Delta x}|=f|\frac{\Delta x}{S}-\frac{(x_{A}+\Delta
x)\cos\theta}{S-(x_{A}+\Delta
x)\sin\theta}+\frac{x_{A}\cos\theta}{S-x_{A}\sin\theta}|$ (5)

Figure 6: Effect of $\Delta x$, $S$, $x_{A}$, and $\theta$ on $|Error_{\Delta
x}|$.

Fig.7 shows the front view of Fig.4.

Figure 7: Front view of the pinhole camera model under non-perpendicular and
perpendicular camera settings.

Under perpendicular setting (Fig.7 _(b)_), the projected displacement $\Delta
y$ on the image plane is given by Eq.6:

$\Delta y^{\prime}=\frac{\Delta y}{S}f$ (6)

Meanwhile, under the non-perpendicular setting (Fig.7 _(a)_), the projected
displacement $\Delta y$ on the image plane can be obtained using the concept
of similar triangles:

$\Delta y^{\prime\prime}=f(\frac{y_{A}+\Delta y}{S-(x_{A}+\Delta
x)\sin\theta}-\frac{y_{A}}{S-x_{A}\sin\theta})$ (7)

Eq.8 presents the absolute error of projected displacement $\Delta y$ on the
image plane, which depends on $f$, $S$, $\theta$, $\Delta x$, $\Delta y$,
$x_{A}$, and $y_{A}$. The error increases with increasing focal length ($f$)
and decreasing $\Delta x$. Fig. 8 demonstrates the impact of the remaining
parameters, highlighting partial dependencies while keeping other parameters
fixed. The error generally increases with a larger camera rotation angle
($\theta$) and a smaller pinhole-object distance ($S$). However, the effect of
$x_{A}$, $y_{A}$, and $\Delta y$ is not monotonic.

$|Error_{\Delta y}|=f|\frac{\Delta y}{S}-\frac{y_{A}+\Delta y}{S-(x_{A}+\Delta
x)\sin\theta}+\frac{y_{A}}{S-x_{A}\sin\theta}|$ (8)

Figure 8: Effect of $\Delta y$, $S$, $y_{A}$, $x_{A}$, $\Delta x$, and
$\theta$ on $|Error_{\Delta y}|$.

##  5 SIFT-Aided Rectified 2D-DIC

This paper proposes a new method to compensate for measurement errors induced
by non-perpendicular camera settings. The method, namely, scale-invariant
feature transform (SIFT)-aided rectified 2D-DIC, consists of six steps:

  1. 1.

Image acquisition.

  2. 2.

Key point extraction.

  3. 3.

Key point matching.

  4. 4.

Homography matrix estimation.

  5. 5.

Image rectification.

  6. 6.

2D-DIC analysis.

This section discusses the details of each step. An example is also presented
for further explanation.

###  5.1 Image Acquisition

When a non-perpendicular camera alignment is unavoidable in a 2D-DIC
measurement, several calibration images of the specimen are collected before
testing in a perpendicular setting, as shown in Fig.9 _(a)_. Next, reference
and deformed images are acquired using the same camera under the non-
perpendicular setting during the test, as shown in Fig.9 _(b)_. It is
recommended not to change the camera settings (e.g., aperture, ISO, shutter
speed) during this procedure. Changing camera settings would require intrinsic
camera calibration. The camera settings must be carefully tuned such that all
acquired images are in focus.

Figure 9: Image acquisition.

###  5.2 Key Point Extraction

Key points and their descriptors are extracted from the calibration and the
reference images using the SIFT algorithm [Lowe (2004)]:

  * •

_Build a difference-of-Gaussian (DoG) pyramid._ Image ($I(x,y)$) is repeatedly
convolved with 2D Gaussian filters ($G(x,y,\sigma_{i})$) with increasing
$\sigma_{i}$, creating Gaussian images ($L(x,y,\sigma_{i})$) in Fig.10(_a_).
DoG images are produced by subtracting adjacent Gaussian images. (Eq.9).

$D(x,y,\sigma_{i})=L(x,y,\sigma_{i})-L(x,y,\sigma_{i-1})$ (9)

  * •

_Detect key points._ As shown in Fig.10(_b_), a key point is located by
comparing a pixel with its neighbors in a $3\times 3\times 3$ cubic. Local
maxima or minima are considered key points.

  * •

_Refine key point locations into the sub-pixel level._ Eq.10 shows the Taylor
expansion of $D(x,y,\sigma)$ up to the quadratic terms [Brown and Lowe
(2002)]. Assume $\mathbf{x_{0}}$ is a key point location found in the previous
step. By taking the first derivative of $D$ and setting it to zero, offset
$\mathbf{h}$ is obtained (Eq.11). $\mathbf{x_{0}}+\mathbf{h}$ is the refined
location of the key point.

$D(\mathbf{x_{0}}+\mathbf{h})\approx D(\mathbf{x_{0}})+\evalat{(\frac{\partial
D}{\partial\mathbf{x}})^{T}}{\mathbf{x}=\mathbf{x_{0}}}\mathbf{h}+\frac{1}{2}\mathbf{h}^{T}\frac{\partial^{2}D}{\partial^{2}\mathbf{x}}\mathbf{h}$
(10)

$\mathbf{h}=-(\frac{\partial^{2}D}{\partial^{2}\mathbf{x}})^{-1}(\frac{\partial
D}{\partial\mathbf{x}})^{T}$ (11)

  * •

_Assign a dominant orientation to each key point._ Each key point is described
concerning its dominant orientation to achieve rotation invariance. A
histogram of intensity gradient orientations is created for each key point’s
neighborhood, using 36 bins with 10∘ coverage per bin. The peak orientation in
the histogram is considered the dominant orientation and assigned to the key
point.

  * •

_Construct key point descriptors._ As shown in Fig.10 _(c)_ , a 16x16
neighborhood around each key point is used to construct its descriptor. The
gradient direction and magnitude are computed at each pixel. Next, the
magnitude is weighted by a Gaussian window based on the pixel’s distance to
the key point. Then, the neighborhood is divided into sixteen 4$\times$4 sub-
regions. For each sub-region, an eight-bin orientation histogram is created
(Fig.10 _(c)_). Hence, a vector of size 128 is formed as the descriptor of
each key point.

Figure 10: Schematic of SIFT algorithm.

For example, as shown in Fig.11 _(a)_ and _(b)_ , the above-described
procedure was followed to extract key points and their descriptors on the
calibration and the reference image. The images were collected from an I-FIT
specimen surface using a Point Grey Gazelle 4.1MP Mono camera with a
resolution of $2048\times 2048$. The calibration and the reference images
identified 152,150 and 24,151 key points, respectively. Only one percent of
the key points were marked to provide better visualization.

Figure 11: Key points are matched between calibration and reference images.

###  5.3 Key Point Matching

A one-to-one key point match is established by finding the shortest euclidean
distance between the 128-sized descriptor vectors. To filter out false
matches, if the second-shortest distance is close to the shortest distance
(i.e., $\frac{d_{closest}}{d_{next\;closest}}\geq\delta$), the match will be
discarded. The threshold ($\delta$) is user-defined and typically ranges from
0.3 to 0.8.

For example, 1,946 key point matches were identified between the calibration
and the reference image. A $\delta$ of 0.55 was used because it eliminated
most false matches while preserving enough correct matches. Fig.11 _(c)_ only
shows 0.5% of the matches for better visualization purposes.

###  5.4 Homography Matrix Estimation

Under the pinhole camera model, a homography matrix $H$ relates any two images
of the same planar surface. For example, because the I-FIT specimen surface is
planar, the calibration image is related to the reference image by
$\mathbf{H}$. In Fig.12, $\mathbf{x}([x,y,1]^{T})$ refers to the projected
coordinates of point $X$ in the reference coordinate system, and
$\mathbf{x}^{\prime}([x^{\prime},y^{\prime},1]^{T})$ represents the projected
coordinates of point $X$ in the calibration coordinate system. The
relationship between $\mathbf{x}$ and $\mathbf{x}^{\prime}$ is described by
$\mathbf{H}$ (Eq. 12).

$\lambda\mathbf{x}^{\prime}=\mathbf{H}\mathbf{x}$ (12)

where $\lambda$ is a scaling factor; $\mathbf{H}$ has eight degrees of
freedom, as shown in Eq.13.

$\mathbf{H}=\begin{bmatrix}h_{11}&h_{12}&h_{13}\\\ h_{21}&h_{22}&h_{23}\\\
h_{31}&h_{32}&1\end{bmatrix}$ (13)

Figure 12: Illustration of homography.

Each key point match provides two linearly independent equations, as shown in
Eq.14. Thus, to estimate $\mathbf{H}$, at least four matches are needed.

$\displaystyle\lambda\begin{bmatrix}x_{i}^{\prime}\\\ y_{i}^{\prime}\\\
1\end{bmatrix}=\begin{bmatrix}h_{11}&h_{12}&h_{13}\\\ h_{21}&h_{22}&h_{23}\\\
h_{31}&h_{32}&1\end{bmatrix}\begin{bmatrix}x_{i}\\\ y_{i}\\\ 1\end{bmatrix}$
(14) $\displaystyle\Rightarrow\begin{bmatrix}x_{i}^{\prime}\\\
y_{i}^{\prime}\\\
1\end{bmatrix}\times\begin{bmatrix}\mathbf{h_{1}}^{T}\mathbf{x}_{i}\\\
\mathbf{h_{2}}^{T}\mathbf{x}_{i}\\\
\mathbf{h_{3}}^{T}\mathbf{x}_{i}\end{bmatrix}=\begin{bmatrix}y_{i}^{\prime}\mathbf{h_{3}}^{T}\mathbf{x}_{i}-\mathbf{h_{2}}^{T}\mathbf{x}_{i}\\\
\mathbf{h_{1}}^{T}\mathbf{x}_{i}-x_{i}^{\prime}\mathbf{h_{3}}^{T}\mathbf{x}_{i}\\\
x_{i}^{\prime}\mathbf{h_{2}}^{T}\mathbf{x}_{i}-y_{i}^{\prime}\mathbf{h_{1}}^{T}\mathbf{x}_{i}\end{bmatrix}$
$\displaystyle\Rightarrow\begin{bmatrix}\mathbf{0}^{T}&-\mathbf{x}_{i}^{T}&y_{i}^{\prime}\mathbf{x}_{i}^{T}\\\
\mathbf{x}_{i}^{T}&\mathbf{0}^{T}&-x_{i}^{\prime}\mathbf{x}_{i}^{T}\\\
-y_{i}^{\prime}\mathbf{x}_{i}^{T}&x_{i}^{\prime}\mathbf{x}_{i}^{T}&\mathbf{0}^{T}\end{bmatrix}\begin{bmatrix}\mathbf{h_{1}}\\\
\mathbf{h_{2}}\\\ \mathbf{h_{3}}\end{bmatrix}=\mathbf{0}$

Given the matching key points, the random sample consensus (RANSAC) algorithm
was used to estimate the homography matrix as below [Fischler and Bolles
(1981)]:

  1. 1.

Randomly sample four key point matches.

  2. 2.

Solve for the eight unknown parameters in $\mathbf{H}$.

  3. 3.

Determine number of matches that fit the solved $\hat{\mathbf{H}}$ according
to Eq.15. $\epsilon$ is often set between 1 to 10 pixels. An $\epsilon$ of 5
was used in this paper.

$||x^{\prime}-\hat{\mathbf{H}}x||_{2}\leq\epsilon$ (15)

  4. 4.

Steps 1 to 3 are repeated 2000 times. Each time, an estimated
$\hat{\mathbf{H}}$ is rejected if there are too few fitting matches or is kept
if the consensus set is larger than the previously saved solution.

The RANSAC algorithm was chosen because of its robustness against noise and
outliers. To enhance the reliability of the estimated homography matrix, it is
suggested to repeat the above procedure on all calibration images and take the
mean. For example, steps 5.2 through 5.4 were repeated for each of the five
calibration images, and the mean estimated homography matrix follows:

$\hat{\bar{\mathbf{H}}}=\begin{bmatrix}1.0640e+00&6.4001e-04&-2.1895e+02\\\
-1.4929e-02&9.3542e-01&2.7591e+01\\\
-2.6070e-05&4.2958e-07&1.0000e+00\end{bmatrix}$ (16)

###  5.5 Image Rectification

The reference and deformed images collected under a non-perpendicular camera
setting are rectified according to Eq.17. Because calculated coordinates on
the right-hand side may not be integers, a bilinear interpolation was used.

$R(x,y)=I(\frac{\hat{\bar{\mathbf{H}}}_{11}x+\hat{\bar{\mathbf{H}}}_{12}y+\hat{\bar{\mathbf{H}}}_{13}}{\hat{\bar{\mathbf{H}}}_{31}x+\hat{\bar{\mathbf{H}}}_{32}y+\hat{\bar{\mathbf{H}}}_{33}},\frac{\hat{\bar{\mathbf{H}}}_{21}x+\hat{\bar{\mathbf{H}}}_{22}y+\hat{\bar{\mathbf{H}}}_{23}}{\hat{\bar{\mathbf{H}}}_{31}x+\hat{\bar{\mathbf{H}}}_{32}y+\hat{\bar{\mathbf{H}}}_{33}})$
(17)

where $R(x,y)$ is gray value at $(x,y)$ in the rectified image;
$\hat{\bar{\mathbf{H}}}$ is estimated homography matrix; $I$ refers to raw
image being rectified.

###  5.6 2D-DIC Analysis

DIC analysis should be conducted on rectified images as they can be considered
acquired under the perpendicular setting.

Fig.13(_c_) shows the DIC-measured vertical displacement field ($\mathbf{v}$)
using raw and rectified deformed images. The raw images were collected under a
non-perpendicular camera setting while conducting I-FIT, as shown in
Fig.13(_a_). The I-FIT is a fracture test that has been widely used to
characterize the cracking potential of AC. More information can be found
elsewhere [Ozer et al. (2016a), Ozer et al. (2016b), Zhu et al. (2020)].
Significant discrepancies existed in the measured displacement fields. The
accuracy will be discussed in the next section.

Figure 13: DIC-measured vertical displacement field ($\mathbf{v}$) on raw and
rectified deformed images.

##  6 Method Validation

The proposed method was validated numerically using synthetic images and
experimentally on an I-FIT. Numerical validation offers precise control and
reliability due to the known displacement field. Experimental validation is
crucial to verify real-world applicability.

###  6.1 Numerical Validation

####  6.1.1 Synthetic Images

First, a series of static images were collected on an I-FIT specimen surface
using a Point Grey Gazelle 4.1MP Mono camera with a resolution of $2048\times
2048$. The imaging system was carefully set up to ensure the camera’s
principal axis was perpendicular to the specimen surface.

Second, the displacement field shown in Fig.14 was used to generate deformed
images. It involves rotation concerning point $(x_{0},0)$ only. The
displacement vector $(u,v)$ at each pixel can be calculated using Eq.18.
Counterclockwise rotation was defined as positive [Zhu and Al-Qadi (2023b)].
As shown in Fig.15 _(b)_ , ten deformed images were generated with
$\theta={1^{\circ},2^{\circ},\dots,10^{\circ}}$.

Figure 14: Displacement fields used to generate deformed images.

$\begin{cases}u=\sin(\theta+\tan^{-1}(\frac{x-x_{0}}{y}))\sqrt{(x-x_{0})^{2}+y^{2}}+x_{0}-x\\\
v=\cos(\theta+\tan^{-1}(\frac{x-x_{0}}{y}))\sqrt{(x-x_{0})^{2}+y^{2}}-y\end{cases}$
(18)

Third, the reference and synthetic deformed images were numerically rotated to
simulate non-perpendicular camera settings. Only rotation with respect to
$z$-axis was considered (Fig.15 _(a)_). Four rotating angles
$\alpha={10^{\circ},20^{\circ},30^{\circ},40^{\circ}}$ were used, resulting in
four series of images, as shown in Fig.15 _(c-f)_. The rotation matrix was
given by Eq.19 [Goldstein et al. (2002)].

$\displaystyle\mathbf{R}$
$\displaystyle=\mathbf{R}_{z}(\alpha)\mathbf{R}_{y}(\beta)\mathbf{R}_{x}(\gamma)$
(19) $\displaystyle=\begin{bmatrix}\cos\alpha&-\sin\alpha&0\\\
\sin\alpha&\cos\alpha&0\\\
0&0&1\end{bmatrix}\begin{bmatrix}\cos\beta&0&\sin\beta\\\ 0&1&0\\\
-\sin\beta&0&\cos\beta\end{bmatrix}\begin{bmatrix}1&0&0\\\
0&\cos\gamma&-\sin\gamma\\\ 0&\sin\gamma&\cos\gamma\end{bmatrix}$
$\displaystyle=\begin{bmatrix}\cos\alpha\cos\beta&\cos\alpha\sin\beta\sin\gamma-\sin\alpha\cos\gamma&\cos\alpha\sin\beta\cos\gamma+\sin\alpha\sin\gamma\\\
\sin\alpha\cos\beta&\sin\alpha\sin\beta\sin\gamma+\cos\alpha\cos\gamma&\sin\alpha\sin\beta\cos\gamma-\cos\alpha\sin\gamma\\\
-\sin\beta&\cos\beta\sin\gamma&\cos\beta\cos\gamma\end{bmatrix}$

where $\alpha$, $\beta$, and $\gamma$ are Euler angles around $z$-, $y$-, and
$x$-axis, respectively (Fig.15 _(a)_).

Figure 15: Euler rotation schema and synthetic images.

####  6.1.2 Experimental Setup

First, DIC analysis was conducted on raw image series (Fig.15 _(b)_) acquired
under a perpendicular camera setting. The results were referred to as
_’Perpendicular’_ in Figs.16 and 17.

Second, two analyses were conducted on each series of numerically rotated
images(Fig.15 _(c-f)_), which simulate non-perpendicular camera settings.

  * •

DIC analysis was conducted directly on numerically rotated images (referred to
as _’Non-Perpendicular’_ in Figs.16 and 17).

  * •

SIFT-aided rectified 2D-DIC was conducted. The numerically rotated images were
first rectified before undergoing the DIC analysis (referred to as _’Rectified
Non-Perpendicular’_ in Figs.16 and 17). The raw reference image was used as
the calibration image.

All analyses were on the same ROI. A subset size of $23\times 23$ was chosen
through an iterative process.

####  6.1.3 Results

Fig.16 compares the DIC-measured displacement fields. Due to space
limitations, only the fifth and the tenth deformed images of image series
Fig.15(_(f)_) are shown herein. The non-perpendicular camera setting induced
significant errors. Meanwhile, the SIFT-aided rectified 2D-DIC method
generated visually identical displacement fields as the perpendicular images.

Figure 16: DIC-measured displacement fields on perpendicular, non-
perpendicular, and rectified images.

The DIC-measured displacement fields were compared with the ground truth to
assess the error quantitatively. The mean absolute error (MAE) was used as the
evaluation criterion (Eq.20).

$\textnormal{MAE}=\frac{\sum_{i=1}^{W}\sum_{j=1}^{H}|u_{ij}^{\prime}-u_{ij}|}{W\times
H}$ (20)

where $W$ and $H$ are numbers of correlation points in $x$\- and
$y$-direction, respectively; $u_{ij}^{\prime}$ is DIC-measured displacement;
$u_{ij}$ is actual displacement obtained from Eq.18.

Fig.17 compares the MAE of horizontal ($u$) and vertical ($v$) displacements.
The findings are summarized below:

  * •

The perpendicular camera setting achieved the smallest error. The MAE was
consistently smaller than 0.1 pixels, even under large deformations.

  * •

The highest error was observed on numerically rotated images simulating non-
perpendicular camera settings. However, the proposed method significantly
reduced the error. The MAE of $u$ and $v$ measured on rectified images
remained below 0.6 pixels. This level of accuracy is suitable for typical AC
lab testing applications, where the spatial resolution is usually finer than
50$\mu$m/pixel.

  * •

The magnitude of deformation and camera rotation angle had a limited impact on
the accuracy of SIFT-aided rectified 2D-DIC. The MAE of $u$ and $v$ measured
on non-perpendicular images increased as camera rotation angle ($\alpha$) and
deformation increased. However, the magnitude of $\alpha$ and deformation had
a negligible effect on perpendicular and rectified non-perpendicular images.

Figure 17: MAE of DIC-measured displacements on the perpendicular, non-
perpendicular, and rectified images.

###  6.2 Experimental Validation

####  6.2.1 Experimental Setup

As shown in Fig.18 _(a)_ , raw images were collected using two synchronized
cameras while conducting the I-FIT at 25°C with a 10 mm/min loading rate. A
low loading rate was used to minimize the imperfect synchronization effect. A
more ductile behavior is expected under a lower loading rate [Ozer et al.
(2016a)]. Fig.18 _(b)_ shows the load-displacement curve. The test specimen’s
surface was painted with a random black pattern on top of a layer of white
paint. Camera A (an iPhone 12 Pro Max’s telephoto camera ($1920\times 1080$
pixels)) was positioned perpendicularly to the specimen surface. Camera B (a
Point Grey Gazelle 4.1MP Mono ($2048\times 2048$ pixels)) was positioned non-
perpendicularly. Camera B’s principal axis was rotated $\alpha$ with respect
to the z-axis. Before the test, five calibration images were collected in a
perpendicular setting using Camera B.

Fig.18 _(c)_ and _(d)_ show examples of images collected by cameras A and B,
respectively. Moreover, the proposed method was followed to rectify images
collected by camera B (Fig.18 _(e)_). Each image series consists of a
reference and 60 deformed images, acquired with an 0.2s time interval. DIC
analysis was conducted on the three series of images.

Figure 18: Experimental setup.

####  6.2.2 Results

Because the actual displacement was unknown, the DIC-measured displacement
field on images collected by camera A was used as the baseline. The
displacement fields obtained from non-perpendicular and rectified images were
compared with the baseline. The mean (Eq.20) and standard deviation (Eq.21) of
the absolute error of $u$ and $v$ were used as the evaluation criteria.

$\textnormal{SDAE}=\sqrt{\frac{\sum_{i=1}^{W}\sum_{j=1}^{H}(|u_{ij}^{\prime}-u_{ij}|-\textnormal{MAE}_{u})^{2}}{W\times
H}}$ (21)

where $W$ and $H$ are numbers of correlation points in $x$\- and
$y$-direction, respectively; $u_{ij}^{\prime}$ is measured displacement;
$u_{ij}$ is baseline displacement attained from perpendicular images.

Fig.19 shows the MAE and SDAE of $u$ and $v$ on non-perpendicular and
rectified images. The findings are summarized below:

  * •

The error induced by non-perpendicularity was significant. However, the error
was reduced substantially by employing the SIFT-aided rectified 2D-DIC. The
MAE and SDAE of both $u$ and $v$ on rectified images were mostly smaller than
0.01 mm. Such accuracy is adequate in an I-FIT application.

  * •

The error on non-perpendicular images increased as the deformation increased.
However, such an effect was much less significant for rectified images.

Figure 19: MAE and SDAE of $u$ and $v$ on non-perpendicular images and
rectified images.

##  7 Automated Crack Propagation Measurement with A Non-Perpendicular Camera
Alignment

zhu2022auto recently developed an optical-flow-based deep neural network,
CrackPropNet, to measure crack propagation on AC specimen surfaces during
cracking tests. The input of CrackPropNet consists of a reference and a
deformed image, and it outputs a crack-edge probability map. CrackPropNet
locates displacement field discontinuities (i.e., cracks) by matching features
at various locations in the reference and deformed image. The network showed
promising performance in measuring crack propagation. However, significant
errors happened when the camera’s principal axis was not perpendicular to the
AC specimen surface. The method proposed in this paper can compensate for such
errors.

As shown in Fig.20, CrackPropNet was applied to measure crack propagation on
the three series of images collected in Fig.18. Because cameras A and B had
different fields of view, the crack maps were carefully adjusted to reflect
the same square area (Fig.20 _(b)_). Fig.20 _(c)_ shows that non-
perpendicularity caused non-negligible measurement errors. Due to the rotation
of camera B’s principal axis around the z-axis, the cracks experienced
compression in the horizontal direction, leading to the loss of crack shape
details. It is expected that CrackPropNet would fail more drastically if the
camera rotation angle is larger. Conversely, the rectification procedure
effectively restored most crack shape details, resulting in crack measurements
similar to those obtained from the perpendicular images, with only minor
differences.

Figure 20: CrackPropNet-measured crack propagation on the perpendicular, non-
perpendicular, and rectified non-perpendicular images.

To quantitatively evaluate the accuracy of the proposed method, an experiment
was conducted on the testing dataset developed by zhu2022auto. The dataset
comprises 188 images collected while conducting the I-FIT on various AC mixes.
It covers the diversified AC cracking behavior. Cameras were positioned
perpendicular to I-FIT specimen surfaces to collect these images.

For example, Fig.21 _(c)_ shows a reference image and its related deformed
images, which correspond to points 1, 2, and 3 in the load-displacement curve.
To simulate a non-perpendicular camera setting, raw images were numerically
rotated $40^{\circ}$ around the z-axis (Fig. 21 _(d)_). These numerically
rotated images were then rectified using the SIFT-aided rectification
procedure (Fig. 21 _(e)_). The raw, numerically rotated, and rectified images
were subsequently processed by CrackPropNet for crack propagation measurement.
Notably, Fig. 21 _(d)_ demonstrates that non-perpendicularity introduced
significant errors, with cracks measured from numerically rotated images
missing crucial details. Conversely, Fig. 21 _(e)_ showcases the effectiveness
of the proposed method, which successfully recovered crack shape details and
produced nearly identical results to those obtained from perpendicular images.

Figure 21: CrackPropNet-measured crack propagation on raw, numerically
rotated, and rectified images.

The above procedure was repeated for the entire dataset. The measured cracks
were compared with the ground-truth crack labels provided by zhu2022auto. F-1
($\frac{2\cdot Precision\cdot Recall}{Precision+Recall}$) was used as the
evaluation criterion, where a higher F-1 indicates a more accurate
measurement. To generate the crack map based on the CrackPropNet’s probability
map output, two commonly used strategies were employed: optimal dataset scale
(ODS) and optimal image scale (OIS). The former employs a fixed threshold for
all images, while the latter finds an optimal threshold for each image. The
CrackPropNet achieved an ODS F-1 of 0.748 and an OIS F-1 of 0.778 on rectified
images, while it achieved an ODS F-1 of 0.755 and an OIS F-1 of 0.781 on
perpendicular images.

It is safe to conclude that the proposed method can efficiently compensate for
crack propagation measurement errors caused by non-perpendicularity.

##  8 Summary and Conclusions

This paper proposes the SIFT-aided rectification method to compensate for
2D-DIC measurement errors caused by non-perpendicular camera alignment. The
proposed method is simple and easy-to-use because no additional equipment is
needed. A rigorous experimental program was conducted to demonstrate its
accuracy and reliability.

First, a theoretical error analysis was conducted to quantify the effects of a
non-perpendicular camera alignment on displacement measurement. Under the
pinhole camera model, focal length ($f$), pinhole-object distance ($S$),
camera rotation angle ($\theta$), displacement ($\Delta x$ & $\Delta y$), and
pixel location ($x_{A}$ & $y_{A}$) significantly affect the absolute error of
projected displacement on the image plane.

Second, the proposed method was validated numerically using synthetic images.
The MAE of horizontal and vertical displacements on rectified images was
consistently smaller than 0.6 pixels, even under considerable camera rotation
(up to 40∘) and large deformation. Such accuracy is adequate in typical AC lab
testing applications, where spatial resolution is normally finer than
50$\mu$m/pixel.

Third, the proposed method was experimentally verified. Raw images were
collected using two synchronized cameras while conducting the I-FIT. DIC-
measured displacement field on images collected under a perpendicular setting
was used as the baseline. The SIFT-aided rectified 2D-DIC efficiently
compensated for errors induced by non-perpendicular camera settings. The MAE
and SDAE of horizontal and vertical displacement were predominantly below 0.01
mm.

Fourth, the SIFT-aided rectification method was validated as a viable approach
to support CrackPropNet in measuring crack propagation under non-perpendicular
camera settings. Both qualitative and quantitative evaluations confirmed that
the rectification procedure effectively restored most crack shape details from
non-perpendicular images, leading to crack measurements that closely aligned
with those obtained from perpendicular images.

The following conclusions are drawn from this study:

  * •

Non-perpendicular camera alignment significantly impacts 2D-DIC measurement
accuracy during asphalt concrete testing.

  * •

The proposed SIFT-Aided Rectified 2D-DIC method significantly reduces
measurement error caused by a non-perpendicular camera alignment, even with
substantial camera rotation and relatively large deformation.

  * •

The SIFT-aided rectification method could be applied to assist CrackPropNet in
measuring crack propagation under non-perpendicular camera settings.

##  9 Limitations and Recommendations

The followings are the limitations of this study and relative suggestions for
future research:

  * •

The proposed method is applicable to planar specimen surfaces. A stereo-DIC is
needed for non-planar objects, such as a cylindrical dynamic modulus specimen.

  * •

Other key point detection and matching algorithms, such as speeded-up robust
features (SURF) and deep-learning-based SuperGlue may be considered in the
future. An end-to-end deep neural network insensitive to non-perpendicular
camera alignment may be developed by stacking SuperGlue and CrackPropNet.

##  10 Data Availability Statement

All data, models, or code supporting this study’s findings are available from
the corresponding author upon reasonable request.

##  11 Acknowledgment

The authors thank the research engineers and students at the Illinois Center
for Transportation: Greg Renshaw, Uthman Mohamed Ali, and Jose Julian Rivera-
Perez. The authors hold responsibility for the accuracy and factual
information presented in the contents.

## References

  * Balcaen et al. (2017) Balcaen, R., Reu, P., Lava, P., and Debruyne, D. (2017).  “Stereo-dic uncertainty quantification based on simulated images.” Experimental Mechanics, 57, 939–951.
  * Birgisson et al. (2008) Birgisson, B., Montepara, A., Romeo, E., Roncella, R., Napier, J., and Tebaldi, G. (2008).  “Determination and prediction of crack patterns in hot mix asphalt (hma) mixtures.” Engineering Fracture Mechanics, 75(3-4), 664–673.
  * Birgisson et al. (2009) Birgisson, B., Montepara, A., Romeo, E., Roncella, R., Roque, R., and Tebaldi, G. (2009).  “An optical strain measurement system for asphalt mixtures.” Materials and Structures, 42(4), 427–441.
  * Blaber et al. (2015) Blaber, J., Adair, B., and Antoniou, A. (2015).  “Ncorr: open-source 2d digital image correlation matlab software.” Experimental Mechanics, 55(6), 1105–1122.
  * Brown and Lowe (2002) Brown, M. and Lowe, D. G. (2002).  “Invariant features from interest point groups..” Bmvc, Vol. 4, 398–410.
  * Chehab et al. (2007) Chehab, G. R., Seo, Y., and Kim, Y. R. (2007).  “Viscoelastoplastic damage characterization of asphalt–aggregate mixtures using digital image correlation.” International Journal of Geomechanics, 7(2), 111–118.
  * Fischler and Bolles (1981) Fischler, M. A. and Bolles, R. C. (1981).  “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.” Communications of the ACM, 24(6), 381–395.
  * Goldstein et al. (2002) Goldstein, H., Poole, C., and Safko, J. (2002).  Classical Mechanics.  Pearson Prentice Hall.
  * Lava et al. (2011) Lava, P., Coppieters, S., Wang, Y., Van Houtte, P., and Debruyne, D. (2011).  “Error estimation in measuring strain fields with dic on planar sheet metal specimens with a non-perpendicular camera alignment.” Optics and Lasers in Engineering, 49(1), 57–65.
  * Lin et al. (2022) Lin, A., Li, R., Jiang, Z., Dong, S., Liu, Y., Liu, Z., Zhou, L., and Tang, L. (2022).  “Path independent stereo digital image correlation with high speed and analysis resolution.” Optics and Lasers in Engineering, 149, 106812.
  * Lowe (2004) Lowe, D. G. (2004).  “Distinctive image features from scale-invariant keypoints.” International journal of computer vision, 60(2), 91–110.
  * Ozer et al. (2017) Ozer, H., Al-Qadi, I. L., Barber, E., Okte, E., Zhu, Z., Wu, S., et al. (2017).  “Evaluation of i-fit results and machine variability using mnroad test track mixtures..” Report no., Illinois Center for Transportation.
  * Ozer et al. (2016a) Ozer, H., Al-Qadi, I. L., Lambros, J., El-Khatib, A., Singhvi, P., and Doll, B. (2016a).  “Development of the fracture-based flexibility index for asphalt concrete cracking potential using modified semi-circle bending test parameters.” Construction and Building Materials, 115, 390–401.
  * Ozer et al. (2016b) Ozer, H., Al-Qadi, I. L., Singhvi, P., Khan, T., Rivera-Perez, J., and El-Khatib, A. (2016b).  “Fracture characterization of asphalt mixtures with high recycled content using illinois semicircular bending test method and flexibility index.” Transportation Research Record, 2575(1), 130–137.
  * Pan (2018) Pan, B. (2018).  “Digital image correlation for surface deformation measurement: historical developments, recent advances and future goals.” Measurement Science and Technology, 29(8), 082001.
  * Pan et al. (2009) Pan, B., Qian, K., Xie, H., and Asundi, A. (2009).  “Two-dimensional digital image correlation for in-plane displacement and strain measurement: a review.” Measurement science and technology, 20(6), 062001.
  * Pan et al. (2013a) Pan, B., Yu, L., and Wu, D. (2013a).  “High-accuracy 2d digital image correlation measurements using low-cost imaging lenses: implementation of a generalized compensation method.” Measurement Science and Technology, 25(2), 025001.
  * Pan et al. (2013b) Pan, B., Yu, L., and Wu, D. (2013b).  “High-accuracy 2d digital image correlation measurements with bilateral telecentric lenses: error analysis and experimental verification.” Experimental Mechanics, 53(9), 1719–1733.
  * Rivera-Pérez et al. (2021) Rivera-Pérez, J., Ozer, H., Lambros, J., and Al-Qadi, I. L. (2021).  “Illinois flexibility index test: effect of specimen geometry and test configuration on the asphalt concrete damage zone.” Journal of Transportation Engineering, Part B: Pavements, 147(1), 04020085.
  * Safavizadeh et al. (2017) Safavizadeh, S., Wargo, A., and Kim, Y. R. (2017).  “Utilizing digital image correlation (dic) in asphalt pavement testing.” Journal of Testing and Evaluation, 46(3), 984–998.
  * Sutton et al. (2009) Sutton, M. A., Orteu, J. J., and Schreier, H. (2009).  Image correlation for shape, motion and deformation measurements: basic concepts, theory and applications.  Springer Science & Business Media.
  * Zhong et al. (2019) Zhong, F., Shao, X., and Quan, C. (2019).  “A comparative study of 3d reconstruction methods in stereo digital image correlation.” Optics and Lasers in Engineering, 122, 142–150.
  * Zhu and Al-Qadi (2023a) Zhu, Z. and Al-Qadi, I. L. (2023a).  “Automated crack propagation measurement on asphalt concrete specimens using an optical flow-based deep neural network.” International Journal of Pavement Engineering, 24(1), 2186407.
  * Zhu and Al-Qadi (2023b) Zhu, Z. and Al-Qadi, I. L. (2023b).  “Crack detection of asphalt concrete using combined fracture mechanics and digital image correlation.” Journal of Transportation Engineering, Part B: Pavements, 149(3), 04023012.
  * Zhu et al. (2020) Zhu, Z., Singhvi, P., Ali, U. M., Ozer, H., and Al-Qadi, I. L. (2020).  “Quantification of the effect of binder source on flexibility of long-term aged asphalt concrete.” Transportation Research Record, 2674(9), 605–616.


# Why Out-of-distribution Detection in CNNs Does Not Like Mahalanobis - and
What to Use Instead

Kamil Szyc
Wrocław University of Science and Technology
11/17 Z. Janiszewskiego Str.
50-372 Wrocław, Poland
<EMAIL_ADDRESS>
Tomasz Walkowiak
Wrocław University of Science and Technology
11/17 Z. Janiszewskiego Str.
50-372 Wrocław, Poland
<EMAIL_ADDRESS>
Henryk Maciejewski
Wrocław University of Science and Technology
11/17 Z. Janiszewskiego Str.
50-372 Wrocław, Poland
<EMAIL_ADDRESS>

###### Abstract

Convolutional neural networks applied for real-world classification tasks need
to recognize inputs that are far or out-of-distribution (OoD) with respect to
the known or training data. To achieve this, many methods estimate class-
conditional posterior probabilities and use confidence scores obtained from
the posterior distributions. Recent works propose to use multivariate Gaussian
distributions as models of posterior distributions at different layers of the
CNN (i.e., for low- and upper-level features), which leads to the confidence
scores based on the Mahalanobis distance. However, this procedure involves
estimating probability density in high dimensional data using the insufficient
number of observations (e.g. the dimensionality of features at the last two
layers in the ResNet-101 model are 2048 and 1024, with ca. 1000 observations
per class used to estimate density). In this work, we want to address this
problem. We show that in many OoD studies in high-dimensional data, LOF-based
(Local Outlierness-Factor) methods outperform the parametric, Mahalanobis
distance-based methods. This motivates us to propose the nonparametric, LOF-
based method of generating the confidence scores for CNNs. We performed
several feasibility studies involving ResNet-101 and EffcientNet-B3, based on
CIFAR-10 and ImageNet (as known data), and CIFAR-100, SVHN, ImageNet2010,
Places365, or ImageNet-O (as outliers). We demonstrated that nonparametric
LOF-based confidence estimation can improve current Mahalanobis-based SOTA or
obtain similar performance in a simpler way. 111All results are fully
reproducible, and the code is available on GitHub upon request.

##  1 Introduction

Figure 1: The idea of the proposed method. a) Method by Lee et al. [12], which
uses the input pre-processing [13], with Mahalanobis distance-based confidence
score $M_{\ell}$ from feature ensemble. For hyperparameter tuning, some OoD
examples are needed. b) We show that non-parametric LOF outperforms
Mahalanobis as an OoD detector in high-dimensional data. c) We propose to
compute confidence scores using LOF in place of Mahalanobis distance. d) We
propose a modified version of LOF - LOF_D, which achieves good results for
some CNN models. e) Simplified version of our method: OoD detection only using
the last layer of DNN - successful on many datasets, hyperparameter free, and
do not use any OoD data.

Deep neural networks deployed for real-world recognition tasks need to
recognize inputs far from the known or training data. However, despite high
classification accuracy on the benchmark datasets (e.g., top-1 accuracy on the
ImageNet equal 84.4% for EfficientNet-B7 [23], and 90.2% for one of the latest
models by Pham et al. [17]), deep networks are vulnerable to out-of-
distribution (OoD) or adversarial inputs that are easily recognized by humans
[3], [10], [25], [15]. Recognition of out of distribution examples is
essential in safety-critical applications or incremental-learning systems
[14], [5], [21], [4].

Many current approaches recognize inputs that are unlike the known data by
estimating class-conditional posterior probabilities. Posterior distributions
are then used to obtain confidence scores of prediction. Some authors propose
to use multivariate Gaussian distributions as models of class-conditional
distributions [1], [12], [20], which leads to estimating the uncertainty of
prediction using Mahalanobis distance. Alternative methods attempt to
recognize out of distribution examples by quantifying outlierness $d(x,X_{i})$
of input $x$ with regard to the known training data $X_{i}$ related to the
class $c_{i}$ maximizing the posterior probability $p(c|x)$, ie.
$c_{i}=\operatorname*{arg\,max}_{c\in C}P(c|x)$. The score $d$ is estimated as
distance, density, or some measure of outlierness, such as the Local
Outlierness Factor (LOF) [2]. A recent comprehensive survey of these
approaches is given in [6].

Many recent methods of detection of OoD inputs in deep neural networks are
based on confidence scores from posterior distributions. The baseline method
by Hendrycks and Gimpel [9] is based on the maximum value of posterior
distribution obtained as the softmax score. The ODIN method by Liang et al.
[13] uses input preprocessing and temperature scaling in the softmax function
to increase separability between in- and out-of-distribution samples. The
unified framework proposed by Lee et al. [12] uses the Mahalanobis distance-
based score aggregated from many layers of the DNN (ie. for lower and upper-
level features). While these methods use standard representations learned to
discriminate within in-distribution classes, some recent works e.g., [20],
[22] propose to use contrastive learning-based representations which aim to
discriminate in- and OoD samples.

Some authors propose to model posterior distributions using multivariate
Gaussian distribution, which leads to confidence scores based on Mahalanobis
distance. However, these procedures require estimating probability densities
in high dimensional representations generated by the DNN (e.g. the
dimensionality of features at the last layer of the ResNet-101[8] is 2048, and
of EfficientNet-B3 is 1536), commonly using ca. 1000-2000 observations per
class. This problem motivated this work. We argue that the quality of such
parametric density estimates is questionable and that more reliable confidence
scores for OoD detection can be proposed using the nonparametric LOF method.

Our main contributions are the following. We show that the LOF-based method
outperforms the Mahalanobis distance-based method in the task of OoD detection
in high-dimensional data (section 2.1). Motivated by this, we propose a simple
method for detecting OoD inputs, inspired by the unified framework by Lee et
al. [12], with confidence scores obtained using the LOF instead of the
Mahalanobis distance. Similar to [12], our method uses standard
representations generated by any modern neural network architecture and does
not require re-training the model. The idea of the proposed method is
summarized in Figure 1. We performed several feasibility studies using
ResNet-101 and EffcientNet-B3, trained on CIFAR-10 and ImageNet, with
CIFAR-100, SVHN, ImageNet2010, Places365, or ImageNet-O used as outliers. We
demonstrate that nonparametric, LOF-based OoD detection can improve
Mahalonobis-based results, or on some datasets obtain similar performance in a
simpler way, ie. with fewer (or no) hyperparameters, and with no OoD samples
needed for hyperparameter tuning.

##  2 Estimation of confidence scores using LOF-based outlierness measure

###  2.1 Mahalanobis distance-based vs nonparametric outlierness factor-based
OoD in high dimensional data

Since the estimation of density in high dimensional data is generally
considered unattractive [7], we want to empirically show confidence scores
obtained from density estimates lead to the limited performance of OoD
detection. On the other hand, outlier or out-of-distribution detection in high
dimensional data can be performed reasonably well using scores obtained from
the Local Outlierness Factor method Breunig et al. [2].

We performed a simple simulation study in which we compared the performance of
OoD detection based on confidence scores obtained using the Mahalanobis
distance vs confidence scores obtained with the LOF algorithm. As in-
distribution (known) data we generated two clusters from the MVN distribution
in $d$ dimensions, with the mean at $[0]_{d}$ and $[-1]_{d}$ and uncorrelated
variables with variance 1. As OoD we used a cluster with mean at
$[\frac{r}{\sqrt{d}}]_{d}$, uncorrelated, with variance 1. Confidence scores
were calculated as the Mahalanobis distance between a test sample and the
closest class-conditional Gaussian distribution, which can be interpreted as
the log of the probability density of the test sample. In the alternative
approach, confidence scores were obtained as local outlierness factors (LOF)
calculated for test samples with respect to the closest cluster of known data
(see section 2.2 for technical details of LOF).

$0$$200$$400$$600$$800$$1{,}000$$0$$20$$40$$60$$80$$100$diemnsionTNR at TPR
95%$0$$200$$400$$600$$800$$1{,}000$$60$$70$$80$$90$$100$dimensionAUCROCMahalanobisLOF
Figure 2:  Comparison of the TNR at TPR 95 % and the AUC for Mahalonobis and
LOF on simulation data. By increasing the complexity of the problem by
expanding the input data dimension, the LOF method is much more stable and
achieves better results. The dimension above 1,000 is common in the lasts
layers of CNNs.

In Figure 2, we summarize performance of Mahalanobis distance-based and LOF-
based OoD detectors as a function of dimensionality $d$. We observe that for
$d=1000$ with 1000 training points per class, the Mahalanobis procedure no
longer detects outliers (AUCROC $\approx$ 50%), while LOF is more reliable
(AUCROC > 80%). The same effect is observed if the probability density of the
known data is modeled using the Weibull distribution fitted to data by the EVM
Rudd et al. [19] algorithm (AUCROC $\approx$ 50% for $d=1000$).

We conclude that in high dimensional data ($d=1000\dots 2000$) OoD detection
can be done more reliably is the LOF-based score is used in place of the
scores obtained from parametric density estimates, such as Mahalanobis
distance-based.

###  2.2 LOF-based outlierness measures

The Local Outlier Factor [2] (LOF) is based on an analysis of the local
density of points. It works by calculating the so-called local reachability
density $LRD_{k}(x,X_{i})$ of input $x$ with regard to the known dataset
$X_{i}$. $LRD$ is defined as an inverse of an average reachability distance
between a given point, its $k$-neighbors, and their neighbors (for details
refer to [2]). $K$-neighbors ($N_{k}(x,X_{i})$) includes a set of points that
lie in the circle of radius $k$-distance, where $k$-distance is the distance
between the point, and it’s the farthest $k^{th}$ nearest neighbor
($||N_{k}(x,X_{i})||>=k$). The local outlier factor (LOF) is formally defined
as the ratio of the average $LRD$ of the $k$-neighbors of the point $x$ to the
$LRD$ of the point.

$d_{LOF}(x,X_{i})=\frac{\sum_{y\in{N_{k}(x,X_{i})}}{LRD_{k}(y,X_{i})}}{||N_{k}(x,X_{i})||LRD_{k}(x,X_{i})}$
(1)

Intuitively, if the point is an inlier, the ratio of average $LRD$ of
neighbors is similar to the $LRD$ of the point. So the LOF is around 1. For
outliers it should be above 1 since the density of an outlier is smaller than
its neighbor densities.

Original LOF[2] was used for outlier detection inside a data set but it can be
easily extended to out-of-distribution recognition. We can calculate local
reachability densities for all training data and treat them and all training
data as the model. During outlierness quantification of input $x$, we have to
find its nearest neighbors among all training data, calculate the local
reachability density for $x$ and divide the average $LRD$ of neighbors by $x$
$LRD$. Original LOF[2] is based on Euclidean distance and we will use this
distance in all experiments marked as LOF. Moreover, we modified the original
LOF by using the cosine metric and building a separate LOF model for each in-
distribution class (see Figure 1 d)). For OoD detection/uncertainty
quantification, we use the model LOF that corresponds to the closest class
identified by the CNN. We refer to this approach as LOF_D. Because the LOF_D
method builds separate models for each class, the time of determining the
confidence score is at least $n$ times shorter than in the case of the LOF
method, where $n$ is the number of classes. The use of LOF or LOF_D requires
specifying the number of analyzed neighbors $k$. The impact of this value on
the effectiveness of OoD detection for relatively large data sets that are
analyzed in this work is negligible. Therefore, in all reported experiments
$k=20$ is assumed (which is the default value in the LOF implementation
222https://scikit-
learn.org/stable/modules/generated/sklearn.neighbors.LocalOutlierFactor.html).

###  2.3 OoD in CNN using LOF-based outlierness measure

The Simple Unified Framework by Lee et al. [12] is based on three techniques
to detect OoD in CNNs (see Figure 1 a)), i.e. the Mahalanobis distance-based
confidence scores, ensembles of scores from each layer of the CNN, and the
input pre-processing by adding small perturbations (ODIN technique proposed by
Liang et al. [13]). We propose to replace the Mahalanobis distance-based
confidence scores by LOF-based outlierness scores - Eq. 1, and omit the ODIN-
like input preprocessing (see Figure 1 c)). The first modification is inspired
by the simulation experiments described in Section 2.1 (see Figure 1 b)). The
second modification is motivated by the fact that input perturbations are
computationally complex and require parameter-tuning (the magnitude of
perturbation). In our experiments, we found that the ODIN-like input
perturbations have negligible (or even negative) impact on separating the in-
and out-of-distribution samples for LOF-based methods. For example, for
experiments with CIFAR-10 (see Section 3.1) perturbation equal zero was always
selected as the best on validation data (we used the same fine-tuning method
as proposed in Lee et al. [12]). In contrast to this observation, the
Mahalanobis distance-based technique for OoD detection by Lee et al. [12] is
sensitive to the correct choice of the input perturbation strength.
Apparently, the Mahalanobis distance-based method requires better separation
of the in- and out-of-distribution samples compared with the non-parametric
LOF outlierness detector.

In the feasibility studies in Section 3.2, we also report the performance of
the simplified version of the proposed method (see Figure 1 e)), with the
estimation of the outlierness using only the last (perultimate) layer of the
deep model. We found that for some simple OoD detection problems, the
simplified method performs reasonably well, which yields a hyperparameter-free
OoD detection method.

##  3 Experimental results

In this section, we demonstrate the performance of the proposed OoD detection
using popular CNN architectures ResNet-101 and EfficientNet-B3 on several
image recognition datasets. The experiments are reproducible using the code on
GitHub.

###  3.1 Experiment organization

We decided to use low- and high-resolution datasets in our experiments. That
allows comprehensive evaluation of the methods. For the low-resolution
problem, we decided to use the CIFAR-10 trained by us on the ResNet-101 model.
We achieved 94.79% accuracy. Next, we chose two types of out-of-distribution
datasets - the SVHN and the CIFAR-100. For the high-resolution problem, we
decided to use ImageNet as known data, working with two architectures, the
ResNet-101 and EfficientNet-B3. We used publicly available pre-trained
versions for these models. We used three out-of-distribution datasets:
ImageNet-2010, the Places365, and ImageNet-O [10]. The ImageNet-2010 dataset
includes images from ILSVRC 2010, with classes unknown in the ImageNet
dataset.

We used the training partitions of the in-distribution datasets to build the
OoD models. To evaluate the OoD methods, we used the testing partitions of the
in-distribution datasets and the given out-of-distribution dataset, with a 1:1
proportion of known and unknown samples. To simplify the calculations for the
ImageNet variants, in most experiments we used only the 50 first classes from
the ImageNet as known data. We note that the model was trained on all 1000
classes.

We use the standard metrics for evaluation of OoD performance: TNR at TPR 95%,
AUROC, detection accuracy, and AUPR.

Since on most benchmarks, the methods compared achieve high performance, we
consider these OoD problems simple. We note that the most challenging problems
reported in this study are the CIFAR-10 (as the known data) vs the CIFAR-100
(as the unknown data) and the ImageNet vs. the ImageNet-2010.

In table 1 we present a comparison of the original Mahalanobis distance-based
method by Lee et al. [12] with our LOF-based methods. Our approaches do not
use the input pre-processing technique and use the LOF-based method for the
computing confidence score - instead of the parametric approach based on
Mahalanobis distance. In the study, we evaluated the simplified method, based
only on the last layer of the CNN. This approach is hyperparameter-free and
does not use OoD examples for model training. In table 2, we presented results
for this simplified version, with results for the ImageNet dataset.

The important part of all OoD detection methods is the features extraction
technique. The standard approach is the Global Average Pooling (gap). We
believe that the impact of the feature generation method on OoD detection is
worth in-depth testing. We used the classic gap method for the results
presented in table 1, and we tested other variants for our simplified sLOF and
sLOF_D methods presented in table 2. There are well-known methods like Global
Maximum Pooling (gap), Cross-dimensional weighting (CroW)[11], Deep Local
Feature (DeLF)[16], Selective Convolutional Descriptor Aggregation (SCDA)[24]
or Generalized Mean Pooling (GeM)[18]. We do not recommend any specific method
- the effectiveness of the methods in the context of OoD detection is the
field for further research.

###  3.2 Results and discussion

In table 1, we summarize the effectiveness of the proposed methods in
comparison with the Mahanalobis distance-based method proposed by Lee et al.
[12].

Table 1: The comparison of Lee et al. [12] and our methods. In contrast to [12], we do not use the input preprocessing and use LOF or LOF_D for computing confidence scores. For the two most challenging problems, the CIFAR-10 vs. the CIFAR-100 and the ImageNet vs. the ImageNet-2010 for the ResNet-101, our approach outperforms [12]. For the rest simpler tasks, we achieved similar results. | In-dist
---
(Model)
OOD | Method |  | TNR at
---
TPR 95%
AUROC |  | DTACC
---
AUPR
CIFAR-10 (ResNet-101) | CIFAR-100 | Mahal. | 53.77 | 90.47 | 83.18 | 90.12
LOF | 63.91 | 92.41 | 85.24 | 92.31
LOF_D | 65.92 | 93.16 | 85.89 | 93.11
SVHN | Mahal. | 99.02 | 99.46 | 97.14 | 99.40
LOF | 99.97 | 99.95 | 99.25 | 99.94
LOF_D | 99.74 | 99.9 | 98.77 | 99.9
ImageNet (ResNet-101) |  ImageNet -2010 | Mahal. | 71.60 | 93.70 | 87.75 | 93.24
LOF | 74.50 | 94.23 | 87.28 | 94.16
LOF_D | 66.15 | 92.31 | 84.45 | 92.23
ImageNet -O | Mahal. | 99.93 | 99.88 | 98.73 | 99.85
LOF | 99.67 | 99.91 | 98.77 | 99.88
LOF_D | 99.33 | 99.77 | 98.40 | 99.74
Places365 | Mahal. | 99.40 | 99.64 | 97.75 | 99.62
LOF | 98.65 | 99.69 | 97.32 | 99.68
LOF_D | 98.90 | 99.66 | 97.45 | 99.65
ImageNet (EfficientNet -B3) |  ImageNet -2010 | Mahal. | 91.75 | 98.11 | 93.65 | 97.98
LOF | 91.05 | 98.08 | 93.35 | 98.03
LOF_D | 77.75 | 95.44 | 88.75 | 95.48
ImageNet -O | Mahal. | 99.93 | 99.97 | 99.27 | 99.93
LOF | 99.94 | 99.98 | 99.63 | 99.95
LOF_D | 99.25 | 99.87 | 98.62 | 99.85
Places365 | Mahal. | 100.0 | 99.97 | 99.48 | 99.95
LOF | 99.95 | 99.99 | 99.55 | 99.96
LOF_D | 99.85 | 99.89 | 99.03 | 99.87
Table 2: The comparison of Lee et al. [12] and our simplified methods. In contrast to [12], we do not use the input preprocessing, use LOF or LOF_D for computing confidence score, but this time using only global features, our methods are hyperparameters free and do not use any OoD data. The features extraction method can be changed from the classic gap approach. We achieve comparable results for most tasks. For the ImageNet vs. the ImageNet-2010 with the ResNet-101, our methods perform better results, and for the EfficientNet-B3 worst. | In-dist
---
(Model)
OOD | Method |  | TNR at
---
TPR 95%
AUROC | ATACC | AUPR
ImageNet (ResNet-101) |  ImageNet -2010 | Mahal. | 71.60 | 93.70 | 87.75 | 93.24
sLOF | 56.36 | 88.40 | 80.82 | 87.57
sLOF_D | 85.40 | 92.35 | 90.32 | 90.91
ImageNet -O | Mahal. | 99.93 | 99.88 | 98.73 | 99.85
sLOF | 92.40 | 98.46 | 93.91 | 98.45
sLOF_D | 99.21 | 99.75 | 96.87 | 96.84
Places365 | Mahal. | 99.40 | 99.64 | 99.75 | 99.62
sLOF | 91.40 | 97.87 | 93.18 | 97.88
sLOF_D | 99.68 | 97.64 | 97.56 | 99.58
ImageNet (EfficientNet-B3) |  ImageNet -2010 | Mahal. | 91.75 | 98.11 | 93.65 | 97.98
sLOF | 69.27 | 89.24 | 83.94 | 86.07
sLOF_D | 84.61 | 87.64 | 90.07 | 80.95
ImageNet -O | Mahal. | 99.93 | 99.97 | 99.27 | 99.93
sLOF | 97.82 | 99.46 | 97.27 | 99.44
sLOF_D | 99.06 | 99.17 | 97.32 | 98.59
Places365 | Mahal. | 100.0 | 99.97 | 99.48 | 99.95
sLOF | 98.20 | 99.51 | 96.90 | 99.48
sLOF_D | 99.64 | 98.31 | 98.02 | 97.44

We observe that on difficult OoD problems (ie. CIFAR-10 as in-distribution vs
CIFAR-100 as OoD; ImageNet as in-distribution vs ImageNet-2010 as OoD –
difficult problems for ResNet-101), our LoF-based procedure outperforms the
Mahalanobis distance-based method. It is worth noting that this is done with
fewer hyperparameters: the Mahalanobis method needs finetuning the input
perturbation hyperparameter and the weights ($\alpha$ weights in Figure 1)
used to aggregate the per-layer confidence scores (both require OoD samples
for parameter finetuning). Our method uses only the latter hyperparameter (the
$\alpha$ weights). We also observe that the optimal value of the input
perturbation is study/data-dependent, in table 1 we report results with the
perturbation parameter fine-tuning for this particular study. Our method is
more robust in this respect.

Secondly, we observe that for the OoD benchmarks which seem easier, ie. the
AUROC obtained by the methods 98% - 99%, (these benchmarks include: CIFAR-10
vs SVHN, ImageNet vs ImageNet-O or Places365, ImageNet vs ImageNet-2010 (for
EfficientNet-B3)), our method achieves similar results, again with no input
perturbation hyperparameter, fine-tuned for a specific study in the
Mahalanobis method.

Finally, we found that for these simple OoD benchmarks, the LOF-based score
allows us to successfully detect OoD inputs using only the last layer of the
network. In Table 2, we report these results. We note that this simplified
LOF-based OoD detector requires no hyperparameters, and hence it does not
require OoD samples for hyperparameter tuning. The method by Lee et al. [12]
uses OoD samples to fine-tune both the input perturbation and the $\alpha$
weights hyperparameters.

We observe that the effectiveness of the LOF-based OoD detector using only the
last layer method depends on the feature extraction method. For the ResNet-101
model and the sLOF, we used the Euclidean distance and CroW [11] method, and
for the sLOF_D cosine distance and Global Maximum Pooling (gmp) strategy. For
the EfficientNet-B3 model, we used the exact distances strategy. Moreover, we
used Global Average Pooling(gap) for the sLOF and concatenated features gap
and gmp for sLOF_D. In-depth analysis of the effect of the feature generation
method which aims to both in-distribution data discrimination and in-
distribution vs out-of-distribution discrimination remains as the area of
further research.

##  4 Conclusion

We proposed a simple method for detecting out-of-distribution inputs in a DNN.
The method uses standard representations learned by modern neural network
architectures and does not require network retraining. Our method uses LOF-
based scores to quantify predictive uncertainty for test samples, in place of
Mahalanobis distance-based confidence scores used in SOTA OoD detectors which
rely on the standard representation. In several feasibility studies, we showed
that this modification leads to improved performance with regard to
Manalanobis-based OoD detectors, or realizes similar performance in a simpler
way. Our method uses fewer hyperparameters (on none in a simplified version):
it does not rely on the input perturbation hyperparameter used by current
methods such as [13] or [12]. We showed that for simple OoD problems, a
simplified LOF-based quantification of outlierness of inputs using only the
last layer features can realize similar performance as complex,
hyperparameter-dependent Mahalanobis distance-based SOTA methods such as [12].
We note that this simplified approach is entirely hyperparameter-free, and as
such does not require OoD samples for hyperparameter tuning, contrary to
Mahalanobis SOTA approaches. Further improvement of our method may come from
in-depth analysis of the effect of feature generation procedures (such as gmp,
gap, CroW, DeLF, SCDA, GeM, etc.) at different layers of the network (of at
the last layer when using the simplified method). Since our method performs
OoD detection using standard representations learned to maximize in-
distribution discrimination, an interesting question remains which methods of
feature extraction are most effective if we want to achieve successful in-
distribution discrimination as well as in- vs out-of-distribution
discrimination.

Limitations. We are aware of the following limitations of the proposed method.

Although the effectiveness of OoD detection using LOF-based scores depends on
the method of feature generation from the subsequent layer of the network, we
do not provide a systematic analysis of this effect. We believe the gap
(global average pooling) method yields reasonable performance, based on
results reported in Table 1. However, OoD detection based on the last layer
features seems more sensitive to the choice of feature generation, and the
choice may be data-dependent. This limitation motivates further research.

We do not report variability of results in tables 1 and 2 with respect to
random train/validation/test partitioning of the data. We work with relatively
large, class-balanced datasets, with ca. 1000-5000 points per class, hence we
believe the variability is not significant (we have some point observations of
e.g., TNR changing ca. 0.1-1 percentage points with different partitions).

## References

  * Bendale and Boult [2015] Abhijit Bendale and Terrance Boult.  Towards open world recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1893–1902, 2015.
  * Breunig et al. [2000] Markus M. Breunig, Hans-Peter Kriegel, Raymond T. Ng, and Jörg Sander.  Lof: Identifying density-based local outliers.  _SIGMOD Rec._ , 29(2):93–104, May 2000.  ISSN 0163-5808.  doi: 10.1145/335191.335388.  URL https://doi.org/10.1145/335191.335388.
  * Chakraborty et al. [2021] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay.  A survey on adversarial attacks and defences.  _CAAI Transactions on Intelligence Technology_ , 6(1):25–45, 2021.  URL https://doi.org/10.1049/cit2.12028.
  * Eykholt et al. [2018] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song.  Robust physical-world attacks on deep learning visual classification.  In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_ , pages 1625–1634. IEEE Computer Society, 2018.  doi: 10.1109/CVPR.2018.00175.  URL http://openaccess.thecvf.com/content_cvpr_2018/html/Eykholt_Robust_Physical-World_Attacks_CVPR_2018_paper.html.
  * Feng et al. [2018] Di Feng, Lars Rosenbaum, and Klaus Dietmayer.  Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection.  In _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_ , pages 3266–3273. IEEE, 2018.
  * Geng et al. [2020] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen.  Recent advances in open set recognition: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * Hastie et al. [2009] Trevor Hastie, Robert Tibshirani, and Jerome Friedman.  _The elements of statistical learning: data mining, inference, and prediction_.  Springer Science & Business Media, 2009.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hendrycks and Gimpel [2017] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=Hkg4TI9xl.
  * Hendrycks et al. [2019] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  _arXiv preprint arXiv:1907.07174_ , 2019.
  * Kalantidis et al. [2016] Yannis Kalantidis, Clayton Mellina, and Simon Osindero.  Cross-dimensional weighting for aggregated deep convolutional features.  In _European conference on computer vision_ , pages 685–701. Springer, 2016.
  * Lee et al. [2018] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , NIPS’18, page 7167–7177, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * Liang et al. [2018] Shiyu Liang, Yixuan Li, and R. Srikant.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=H1VGkIxRZ.
  * McAllister et al. [2017] Rowan McAllister, Yarin Gal, Alex Kendall, Mark Van Der Wilk, Amar Shah, Roberto Cipolla, and Adrian Weller.  Concrete problems for autonomous vehicle safety: Advantages of bayesian deep learning.  In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_ , IJCAI’17, page 4745–4753. AAAI Press, 2017.  ISBN 9780999241103.
  * Nguyen et al. [2015] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 427–436, 2015.
  * Noh et al. [2017] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han.  Large-scale image retrieval with attentive deep local features.  In _Proceedings of the IEEE international conference on computer vision_ , pages 3456–3465, 2017.
  * Pham et al. [2021] Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, and Quoc V. Le.  Meta pseudo labels.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2021\.  URL https://arxiv.org/abs/2003.10580.
  * Radenović et al. [2018] Filip Radenović, Giorgos Tolias, and Ondřej Chum.  Fine-tuning cnn image retrieval with no human annotation.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(7):1655–1668, 2018.
  * Rudd et al. [2017] Ethan M Rudd, Lalit P Jain, Walter J Scheirer, and Terrance E Boult.  The extreme value machine.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(3):762–768, 2017.
  * Sehwag et al. [2021] Vikash Sehwag, Mung Chiang, and Prateek Mittal.  Ssd: A unified framework for self-supervised outlier detection.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=v5gjXpmR8J.
  * Sharif et al. [2016] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter.  Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition.  In _Proceedings of the 2016 acm sigsac conference on computer and communications security_ , pages 1528–1540, 2016.
  * Tack et al. [2020] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 11839–11852. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/8965f76632d7672e7d3cf29c87ecaa0c-Paper.pdf.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  EfficientNet: Rethinking model scaling for convolutional neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 6105–6114. PMLR, 09–15 Jun 2019.  URL http://proceedings.mlr.press/v97/tan19a.html.
  * Wei et al. [2017] Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou.  Selective convolutional descriptor aggregation for fine-grained image retrieval.  _IEEE Transactions on Image Processing_ , 26(6):2868–2881, 2017.
  * Zhou and Firestone [2019] Zhenglong Zhou and Chaz Firestone.  Humans can decipher adversarial images.  _Nature communications_ , 10(1):1–9, 2019.