# Visual Explanation for Deep Metric Learning

Sijie Zhu, Taojiannan Yang, and Chen Chen S. Zhu, T. Yang, and C. Chen are
with the Department of Electrical and Computer Engineering, University of
North Carolina at Charlotte, Charlotte, NC, 28223 USA e-mails:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This work explores the visual explanation for deep metric learning and its
applications. As an important problem for learning representation, metric
learning has attracted much attention recently, while the interpretation of
the metric learning model is not as well-studied as classification. To this
end, we propose an intuitive idea to show where contributes the most to the
overall similarity of two input images by decomposing the final activation.
Instead of only providing the overall activation map of each image, we propose
to generate point-to-point activation intensity between two images so that the
relationship between different regions is uncovered. We show that the proposed
framework can be directly applied to a wide range of metric learning
applications and provides valuable information for model understanding. Both
theoretical and empirical analyses are provided to demonstrate the superiority
of the proposed overall activation map over existing methods. Furthermore, our
experiments validate the effectiveness of the proposed point-specific
activation map on two applications, i.e. cross-view pattern discovery and
interactive retrieval. Code is available at https://github.com/Jeff-
Zilence/Explain_Metric_Learning

###### Index Terms:

Deep metric learning, visual explanation, convolutional neural networks,
activation decomposition

##  I Introduction

Learning the similarity metrics between arbitrary images is a fundamental
problem for a variety of tasks, such as image retrieval [1], verification [2,
3], localization [4], video tracking [5], etc. Recently the deep Siamese
network [6] based framework has become a standard architecture for metric
learning and achieves exciting results on a wide range of applications [7].
However, there are surprisingly few works conducting visual analysis to
explain why the learned similarity of a given image pair is high or low.
Specifically, which part contributes the most to the similarity is a
straightforward question and the answer can reveal important hidden
information about the model as well as the data.

Previous visual explanation works mainly focus on the interpretation of deep
neural network for classification [8, 9, 10, 11]. Guided back propagation
(guided BP) [8] has been used for explanation by generating the gradient from
prediction to input, which shows how much the output will change with a little
change in each dimension of the input. Another representative visual
explanation approach, class activation map (CAM) [9], generates the heatmap of
discriminative regions corresponding to a specific class based on the
linearity of global average pooling (GAP) and fully connected (FC) layer.
However, the original method only works on this specific architecture
configuration (i.e. GAP+FC) and needs re-training for visualizing other
applications. Based on the gradient of the last convolutional layer instead of
the input, Grad-CAM [12] is proposed to generate activation maps for all
convolutional neural network (CNN) architectures. Besides, other existing
methods explore network ablation [10], the winner-take-all strategy [13],
inversion [14], and perturbation [11] for visual explanation.

Since verification applications like person re-identification (re-id) [3]
usually train metric learning models along with classification, recent work
[15] leverages the classification activation map to improve the performance,
but the activation map of metric learning is still not well explored. For two
given images, a variant of Grad-CAM has been used for visualization of image
retrieval [16] by computing the gradient from the cosine similarity of the
embedding features to the last convolutional layers of both images. However,
Grad-CAM only provides the overall highlighted regions of two input images,
the relationship between each activated region of two images is yet to be
uncovered. Since the similarity is calculated from two images and possibly
based on several similar patterns between them, the relationship between these
patterns is critical for understanding the model.

Figure 1: An overview of activation decomposition. The overall activation map
(the second column) on each image highlights the regions contributing the most
to the similarity. The point-specific activation map (the third column)
highlights the regions in one image that have large activation responses on a
specific position, e.g. mouth or eye, in the other image.

In this paper, we propose an activation decomposition framework for visual
explanation of deep metric learning and explore the relationship between each
activated region by point-to-point activation response between two images. As
shown in Fig. 1, the overall activation maps (the second column) of two input
images are generated by decomposing their similarity (score) along each image.
In this example, the query image (A) has a high/strong activation on both the
eyes and mouth areas, but the overall map of the retrieved image (B) only
highlights the eyes. Therefore, it is actually hard to understand how the
model works only based on the overall maps. For image B, the mouth region
(green point) has a low activation which means the activation between the
mouth and the whole image A is low compared to the overall activation
(similarity). However, by further decomposing this activation (green point)
along image A, that is to compute the activation between the mouth region of
image B and each position in image A, the resulting point-specific activation
map (green box) reveals that the mouth region of image B still has a high
response on the mouth region of image A. This point-specific activation map
can be generated for each pixel, which encodes the point-to-point activation
intensity for representing the relationship between regions in both images,
e.g. eye-to-nose or mouth-to-mouth. Compared with the overall activation map,
the point-specific activation map provides much more refined information about
the model which is crucial for explanation.

The main contributions of this paper are summarized as follows.

  * •

We propose a novel explanation framework for deep metric learning
architectures based on activation decomposition. It can be served as a white-
box explanation tool to better understand the metric learning models without
modifying the model architectures, and is applicable to a host of metric
learning applications, e.g. face recognition, person re-identification, image
retrieval, etc.

  * •

The proposed point-specific activation map uncovers the point-to-point
activation intensity which, to the best of our knowledge, has not been
explored by existing methods. Our experiments further show the importance of
the point-specific activation map on two practical applications, i.e. cross-
view pattern discovery and interactive retrieval.

  * •

We provide both theoretical and empirical analysis to show the superiority of
the proposed overall activation map for deep metric learning over the popular
Grad-CAM algorithm.

The remainder of this paper is organized as follows. Section II provides a
brief review on existing visual interpretation methods for classification and
metric learning. Section III introduces the idea of activation decomposition
for visual interpretation on a simple network architecture. Section IV further
extends the framework to more complex architectures for metric learning as a
unified solution. Section V presents a detailed comparison between the
proposed method and Grad-CAM. Section VI validates the effectiveness of the
overall activation map of our method, and Section VII demonstrates the unique
advantages of the point-specific activation map on two practical applications.
Finally, Section VIII concludes the paper.

##  II Related Work

###  II-A Interpretation for Classification

As a default setting, most existing interpretation methods [9, 8, 13, 11] are
designed in the classification context where the output score is generated
from one input image. Among the approaches that do not require architecture
change, one intuitive idea [9, 13] is to look into the model and see where
contributes the most to the final prediction. CAM [9] and its variant Grad-CAM
[12] are among the most widely used approaches, but recent works [11, 12, 17]
simply consider CAM as a heuristic linear combination of convolutional feature
maps which is limited to the original architecture, global average pooling
(GAP) and one FC layer [9, 12, 17]. Specifically, to generate CAM for standard
VGG [18] with global max pooling (GMP) and three FC layers, [9, 17] have to
replace this architecture with GAP+FC and re-train the model. In this paper,
we show that CAM is actually a special case of activation decomposition on the
specific architecture, and the activation decomposition idea applies to more
complex architectures without re-training, even beyond the classification
problem.

Grad-CAM [12] is considered as a generalization of CAM for arbitrary CNN
architectures, but the original paper only provides the proof on CAM’s
architecture. We show that Grad-CAM is not equivalent to activation
decomposition for some architectures (Section V).

Another direction of explanation is to check the black box model by modifying
the input and observing the response in the prediction. A representative
method [11] aims to optimize the blurred region and see which region has the
strongest response on the output signal when it is blurred. Perturbation
optimization is applicable for any black box model, but the optimization is
computationally expensive.

Figure 2: Activation decomposition on a simple architecture for deep metric
learning.

###  II-B Interpretation for Metric Learning

As ignored by most existing methods, the explanation for metric learning is
important for a wide range of applications [19, 20], e.g. weakly supervised
localization. In this work, we further shows its potential significance on
diagnosis of metric learning losses and new applications including cross-view
pattern discovery and interactive retrieval for person re-identification and
face recognition. Gradient-based methods, e.g. guided BP [8] can be adopted to
metric learning, but recent works [21, 11] claim that gradient can be
irrelevant to the model and guided-BP fails on the sanity check [21]. Among
methods that pass the sanity check, Grad-CAM has been used for visualization
of image retrieval in [16], but no quantitative result is provided in the
paper. Instead of adding more gradient heuristics like Grad-CAM++ [22], we
propose to explain the result of Grad-CAM under the decomposition framework.
As shown in Section V, by removing the heuristic step in Grad-CAM, the meaning
of the generated activation map can be clearly explained using the proposed
framework.

Recent work [19] also explores visualization techniques on metric learning,
but it only provides qualitative results on several specific architectures.
Another variant [20] replaces the classification score with triplet loss in
Grad-CAM framework and deploys weights transfer from the training data in the
inference phase. The method requires three samples to form the triplet and its
usage in the inference phase is dependent upon the specifically designed
transfer procedure, thus is not a general method given only the pretrained
model. Moreover, all the aforementioned methods only focus on the overall
activation map which is not satisfactory for understanding the model as
discussed in Section I. Apart from white-box explanation, black-box
optimization method, such as perturbation[11], needs reformulation for metric
learning and can be more computationally expensive if the point-to-point
activation map is desired.

##  III Activation Decomposition on A Simple Architecture

To better introduce our method, we first review the formulation of CAM and
illustrate our idea based on a simple architecture (CNN+GAP in Fig. 2) for
metric learning. Note that the proposed method is not limited to this
architecture, the general formulation will be presented in Section IV. As
shown in Eq. 1, CAM is actually a spatial decomposition of the prediction
score of each class, and the original method only applies for CNN with GAP and
one FC layer without bias. The decomposition clearly shows how much each part
of the input image contributes to the overall prediction of one class and
provides valuable information about how the decision is made inside the
classification model. The idea is based on the linearity of GAP:

$\small
S_{c}=\sum_{k}\omega_{k,c}\left(\frac{1}{Z}\sum_{i,j}A_{i,j,k}\right)=\frac{1}{Z}\sum_{i,j}\left(\sum_{k}\omega_{k,c}A_{i,j,k}\right).$
(1)

$S_{c}$ denotes the overall score (before softmax) of class $c$ and
$\omega_{k,c}$ is the FC layer parameter for the $k$-th channel of class $c$.
$A_{i,j,k}$ denotes the feature map of the last convolutional layer at
position $(i,j)$, and $Z$ is the normalization term of GAP. In fact, the
result $\sum_{k}\omega_{k,c}A_{i,j,k}$ can be considered as a decomposition of
$S_{c}$ along $(i,j)$. For a two-stream Siamese-style architecture (see Fig.
2) with GAP and the cosine similarity metric ($S$) for metric learning
applications (e.g. image retrieval), we propose to do decomposition along
$(i,j,x,y)$ so that the relationship between different parts of two images are
uncovered:

$\begin{split}S&=\sum_{k}GAP(A^{q}_{k})GAP(A^{r}_{k})\\\
&=\frac{1}{Z}\sum_{k}\left(\sum_{i,j}A^{q}_{i,j,k}\sum_{x,y}A^{r}_{x,y,k}\right)\\\
&=\frac{1}{Z}\sum_{i,j,x,y}\left(\sum_{k}A^{q}_{i,j,k}A^{r}_{x,y,k}\right).\end{split}$
(2)

Here the normalization terms of cosine similarity (L2 norm) and GAP are
included in $Z$. We use $(i,j)$ and $(x,y)$ for different streams because some
applications such as cross-view image retrieval [4] may have different image
sizes for two streams. $A$ denotes the feature map of the last convolutional
layer. The superscripts $q$ and $r$ respectively denote the query and
retrieved images in this paper. For each query point $(i,j)$ in the query
image, the corresponding point-specific activation map in the retrieved image
is given by $\sum_{k}A^{q}_{i,j,k}A^{r}_{x,y,k}$ which is the contribution of
features at $(i,j,x,y)$ to the overall cosine similarity. Like CAM and Grad-
CAM, bilinear interpolation is implemented to generate the contribution of
each pixel pair and the full resolution map. The overall activation maps of
two images are generated by a simple summation along $(i,j)$ or $(x,y)$, e.g.
$\sum_{i,j}(\sum_{k}A^{q}_{i,j,k}A^{r}_{x,y,k})$. Although we only show the
positive activation in the map, the negative value is also available for
counterfactual explanation like [12].

Recent works [23, 24] have highlighted the superiority of L2 normalization, we
therefore adopt the cosine similarity $S$ as the default metric. With L2
normalization, the squared Euclidean distance $D$ equals to $2-2S$ so that $S$
and $D$ are equivalent as metrics. Although there are still a number of
methods [3, 25] utilizing the Euclidean distance without L2 normalization, [3]
shows that the cosine similarity performs well as the evaluation metric. We
further empirically show that the cosine similarity works well for explanation
in this case (Section VII-B).

##  IV Generalization for Complex Architectures

Recent metric learning approaches usually leverage more complex architectures
to improve performance, e.g. adding several FC layers after the flattened
feature or global pooling. Although different metric learning applications
have different head architectures (e.g. GAP+FC) on CNN, the basic components
are highly similar. Given the introduction of activation decomposition on a
simple architecture in the previous section, we present a unified extension to
make our method applicable to most existing state-of-the-art CNN architectures
for various applications, including image retrieval [7], face recognition
[26], person re-identification [3] and image geo-localization [4].
Specifically, in Section IV-A, we address linear components by considering
them together as one linear transformation. Then Section IV-B focuses on
transforming nonlinear components to linear ones in the inference phase.

###  IV-A Linear Component

For the last convolutional layer feature $A\in\mathbb{R}^{m\times n\times p}$
($m$, $n$ and $p$ denote the width, height, and number of channels), its GAP
(global average pooling) is equivalent to the flattened feature
$\hat{A}\in\mathbb{R}^{mnp}$ multiplied by a transformation matrix
$T_{GAP}\in\mathbb{R}^{p\times mnp}$:

$GAP(A)=\frac{1}{mn}\sum_{i,j}A_{i,j}=T_{GAP}\hat{A}.$ (3)

Here $(i,j)$ denotes the spatial coordinates. By reshaping $T_{GAP}$ to
$p\times m\times n\times p$ as $T^{*}_{GAP}$, the matrix is given by:

$T_{GAP}^{*}(k^{\prime},i,j,k)=\left\\{\begin{aligned}
\frac{1}{mn}&&k^{\prime}=k\\\ 0&&k^{\prime}\neq k\end{aligned}\right.$ (4)

The matrix $T_{GAP}$ is simply calculated by reshaping $T^{*}_{GAP}$ to
$p\times mnp$. Therefore, the GAP can be considered as a special case of
flattened layer multiplied by a transformation matrix. Without loss of
generality, we consider a two-stream framework with flattened layer followed
by one FC layer with weights $\hat{W}\in\mathbb{R}^{l\times mnp}$ and biases
$B\in\mathbb{R}^{l}$ ($l$ denotes the length of the feature embedding vector).
Since the visual explanation is generated in the inference phase when typical
components such as FC layer and batch normalization (BN) are linear, all the
linear components together are formulated as one linear transformation
$g(\hat{A})=\hat{W}\hat{A}+B=\sum_{i,j}W_{i,j}A_{i,j}+B$ in the FC layer. Here
$W_{i,j}\in\mathbb{R}^{l\times p}$ and $A_{i,j}\in\mathbb{R}^{p}$ denote the
weights matrix and feature vector corresponding to position $(i,j)$. Although
$B$ is ignored in CAM, we keep it as a residual term in the decomposition.
Then Eq. 2 is re-formulated as:

$\small\begin{split}SZ&=g^{q}(A^{q})\cdot g^{r}(A^{r})\\\
&=\left(\sum_{i,j}W^{q}_{i,j}A^{q}_{i,j}+B^{q}\right)\cdot\left(\sum_{x,y}W_{x,y}^{r}A^{r}_{x,y}+B^{r}\right)\\\
&=\sum_{i,j,x,y}\underbrace{(W^{q}_{i,j}A^{q}_{i,j})\cdot(W^{r}_{x,y}A_{x,y}^{r})}_{\textbf{point-
to-point activation}}+\sum_{i,j}(W^{q}_{i,j}A^{q}_{i,j}\cdot B^{r})\\\
&\quad+\sum_{x,y}(W^{r}_{x,y}A^{r}_{x,y}\cdot B^{q})+B^{q}\cdot
B^{r}.\end{split}$ (5)

Here $Z$ denotes the normalization term for cosine similarity (L2 norm), and
$\cdot$ is the inner product. As can be seen from Eq. 5, the decomposition of
$S$ has four terms, and the last three terms contain the bias $B$. The first
term clearly shows the activation response for location pair $(i,j,x,y)$ and
the point-specific map for query position $(i,j)$ is given by
$I(x,y)=(W^{q}_{i,j}A^{q}_{i,j})\cdot(W^{r}_{x,y}A_{x,y}^{r})$. The second and
third terms correspond to the activation between one image and the bias of the
other image. Although they can be considered as negligible bias terms, they
actually contribute to the overall activation map. For the overall activation
map of the query image, the second term can be included, because
$W^{q}_{i,j}A^{q}_{i,j}\cdot B^{r}$ varies at different $(i,j)$ positions
while the third and last terms stay unchanged. Similarly, the first and third
terms are considered when calculating the overall map for the retrieved image.
We investigate both settings about the bias term (i.e. w or w/o) on the
overall activation map and they are referred to as “Decomposition”:
$(W^{q}_{i,j}A^{q}_{i,j})\cdot(\sum_{x,y}W^{r}_{x,y}A^{r}_{x,y})$, and
“Decomposition+Bias” :
$(W^{q}_{i,j}A^{q}_{i,j})\cdot(\sum_{x,y}W^{r}_{x,y}A^{r}_{x,y}+B^{r})$ in
Section VI. The last term is pure bias which is the same for every input
image. Therefore, we ignore this term if not mentioned.

###  IV-B Non-linear Component

Non-linear transformation increases the difficulty of activation
decomposition, our idea is to find its approximate linear transformation which
can be directly integrated into the formulation of Section IV-A. For a non-
linear transformation $f(x)$ with its current input data
$x\in[x_{0}-\delta,x_{0}+\delta]$ ($\delta$ is a small number), the first two
terms of the Taylor expansion of $f(x)$ can be considered as its linear
transformation in the inference phase:

$\small f(x)=f(x_{0})+\frac{f^{\prime}(x_{0})}{1!}(x-x_{0})+...\\\ \approx
f(x_{0})+f^{\prime}(x_{0})(x-x_{0}).$ (6)

Therefore, our method can generalize to more complex architectures as long as
the network is differentiable, since the formulation of gradient
$f^{\prime}(x_{0})$ can be computed by chain rule.

In this paper, we shed light on the practical guidelines for deploying our
method on image retrieval, face recognition, person re-identification, and
cross-view geo-localization frameworks. For these popular applications, the
most widely used non-linear components – global maximum pooling (GMP) and
rectified linear unit (ReLU) – can be transformed as linear operations in the
inference phase by multiplying a mask111The mask is computed once for each
input image.. In the inference phase, GMP can be considered as a combination
of a maximum mask $M$ and GAP as shown in Fig. 3. The transformation matrix of
GMP is given by

$T^{*}_{GMP}=mn(T_{GAP}^{*}\odot M).$ (7)

$M\in R^{m\times n\times p}$ is the maximum matrix of $A$ where only the
maximum position of each channel has nonzero value as $1$. $T_{GMP}$ is
computed by reshaping $T^{*}_{GMP}$ to $p\times mnp$. The result of Hadamard
product ($M\odot A$) is considered as the new feature map which can be
directly applied to Eq. 5. Similarly, by adding a mask for ReLU, the FC layer
with ReLU can be included in $W$ and $B$ in the inference phase.

Figure 3: Maximum mask for GMP. $Z$ is the constant normalization term for GAP. TABLE I: Weakly supervised localization accuracy (IOU=0.5) with different thresholds on CUB validation set. Threshold | Grad-CAM | Grad-CAM (no norm) | Decomposition+Bias (ours) | Decomposition (ours)
---|---|---|---|---
0.15 |  16.71% | 16.83% | 23.49% | 17.38%
0.2 | 16.26% | 17.06% | 32.01% | 19.77%
0.3 | 14.48% | 21.59% |  44.94% | 34.92%
0.4 | 9.43% | 35.01% | 37.85% |  50.64%
0.5 | 4.43% | 47.00% | 21.99% | 45.78%
0.6 | 1.54% |  48.27% | 9.35% | 27.98%
0.7 | 0.27% | 20.90% | 2.25% | 9.11%

##  V Comparison with Grad-CAM

Apparently, the proposed method can generate point-specific activation map
which uncovers more fine-grained information than Grad-CAM. We will further
show the relationship between our overall activation map and Grad-CAM. As
another way for generating overall activation map on more complex
architectures, Grad-CAM follows a heuristic design without a clear meaning for
the generated map. When we calculate the activation map based on Grad-CAM,
what do we get? [12] has shown that Grad-CAM is equivalent to CAM on GAP based
architecture, while it is not true on other architectures. For a
classification architecture with flattened feature and one FC layer, the
prediction score for class $c$ is formulated as
$S_{c}=\sum_{i,j,k}W_{i,j,k,c}A_{i,j,k}$, where $W$ is the reshaped weights of
the FC layer following the shape of the last convolutional feature ($A$). From
our activation decomposition perspective, the contribution of each position
$(i,j)$ is clearly given by $\sum_{k}W_{i,j,k,c}A_{i,j,k}$, while the Grad-CAM
map for class $c$ at $(i,j)$ position is given by:

$\begin{split}GradCAM_{i,j,c}&=\sum_{k}A_{i,j,k}GAP\left(\frac{\partial
S_{c}}{\partial A_{k}}\right)\\\
&=\sum_{k}A_{i,j,k}GAP\left(W_{k}\right).\end{split}$ (8)

In this case, the result of Grad-CAM is different from activation
decomposition because of the GAP operation in Eq. 8. For architectures using
GMP, they are also different, because only the maximal value of each channel
contributes to the overall prediction score with the activation decomposition
framework, while Grad-CAM puts the same weight for features at all positions.
Concretely, for a classification architecture with GMP and one FC layer
($W\in\mathbb{R}^{p\times l}$ and no bias), the prediction score for class $c$
is given by
$S_{c}=\sum_{k}max_{i,j}(A_{:,:,k})W_{k,c}=\sum_{i,j,k}A_{i,j,k}M_{i,j,k}W_{k,c}$,
where $M$ is the maximum matrix in Eq. 7. The decomposition of position
$(i,j)$ is $\sum_{k}A_{i,j,k}M_{i,j,k}W_{k,c}$ and only the maximum position
of each channel has a nonzero value. However, for Grad-CAM, the activation map
is given by:

$GradCAM_{i,j,c}=GAP\left(\frac{\partial S_{c}}{\partial
A}\right)A_{i,j}=\frac{1}{mn}\sum_{k}A_{i,j,k}W_{k,c},$ (9)

where all $(i,j)$ positions have a nonzero weight resulting in a more
scattered activation map.

Although [12] empirically shows that the heuristic GAP step can help improve
the performance of object localization, this step makes the meaning of the
generated map unclear. By removing this step, which means combining the
gradient and feature directly as $A_{i,j,k}\frac{\partial S_{c}}{\partial
A_{i,j,k}}$, the modified version would generate the same result as activation
decomposition for flattened and GMP based architectures. As for the metric
learning architecture discussed in Section IV-A, the Grad-CAM map of query
image is written as (the detailed derivation is presented in Appendix A):

$\begin{split}GradCAM_{i,j}=&\frac{1}{Z}\left(\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}GAP(W^{q})A^{q}_{i,j}\right)\\\
&\cdot\underbrace{\left(\sum_{x,y}W^{r}_{x,y}A^{r}_{x,y}+B^{r}\right)}_{E^{r}}.\end{split}$
(10)

$E^{q}$ and $E^{r}$ denote the embedding vectors of the query and retrieved
images before L2 normalization and $Z$ is the normalization term. The gradient
term $\frac{\partial(E^{q}/|E^{q}|)}{\partial E^{q}}$, which comes from the L2
normalization, would put less weights on the dominant channels so that the
generated activation map becomes more scattered as shown in Fig. 4 (please
also refer to Appendix A for proof). It can be removed by calculating gradient
from $E^{q}\cdot E^{r}$ without L2 normalization, denoted as “Grad-CAM (no
norm)”. If we remove the gradient term as well as the GAP term, the result is
actually equivalent to the overall map of our “Decomposition+Bias” given by
the first two terms of Eq. 5 as
$(W^{q}_{i,j}A^{q}_{i,j})\cdot(\sum_{x,y}W^{r}_{x,y}A^{r}_{x,y}+B^{r})$.

TABLE II: Weakly supervised localization accuracy (IoU=0.5) of various methods on CUB validation set. | Method | Accuracy
---|---|---
Classification | CAM [9] | 41.0%
Metric Learning | Grad-CAM | 16.7%
Grad-CAM (no norm) | 48.3%
Decomposition+Bias (ours) | 44.9%
Decomposition (ours) | 50.6%
Figure 4: The activation maps of different approaches on an example image pair
from the CUB dataset.

##  VI Evaluating Overall Activation Map

To validate the advantage of the proposed method, we provide results of human
evaluation as well as several practical applications. We first verify the
effectiveness of the proposed overall activation map with weakly supervised
localization and human evaluation (Sections VI-A and VI-B). We also show how
the activation map brings potential insights on the generalization ability of
different metric learning models (Section VI-C).

Figure 5: Human evaluation interface for evaluating the overall activation
maps on CUB. Figure 6: Human evaluation results on the proposed method vs.
Grad-CAM variants.

###  VI-A Weakly Supervised Localization

Following previous works [9, 12] on visual explanation, we conduct weakly
supervised localization experiment in the context of metric learning. Among
the popular datasets for image retrieval, CUB [27] is a challenging dataset
with over 10,000 images from 200 bird species, and the bounding box annotation
is available for localization evaluation. We first train a metric learning
model on CUB using the state-of-the-art image retrieval method [7]. Then the
weakly supervised localization is conducted to show the differences between
the variants of the proposed visual explanation method. We first generate the
activation maps of different methods, then the mask is generated by segmenting
the heatmap with a threshold. Following previous works [9, 12], the bounding
box that covers the largest connected component in the mask is generated as
the prediction. A predicted bounding box is considered correct if the IoU
(Intersection over Union) between the bounding box and ground-truth is greater
than $0.5$. We evaluate multiple thresholds for different approaches and find
the results are sensitive to the threshold. Table I lists the localization
accuracy on CUB validation set under different thresholds. For convenience, we
report the best result of each method in Table II, including the result of CAM
(classification based method) adopted from the original paper [9].

In these two tables, “Grad-CAM (no norm)” means Grad-CAM with gradient
computed from the product of two embedding vectors without normalization and
“Decomposition+Bias” denotes the first two terms of Eq. 5. Since the
architecture of [7] is based on GMP, “Grad-CAM (no norm)” is not equivalent to
“Decomposition+Bias” as shown in Section V. The proposed framework outperforms
Grad-CAM for metric learning as well as CAM based on classification. As
justified in Section V, computing the gradients before normalization does
improve the performance of Grad-CAM by a large margin. The qualitative result
in Fig. 4 also illustrates the scattering effect of Grad-CAM with
normalization as demonstrated in our theoretical analysis (Appendix A).

###  VI-B Human Evaluation

Following the work of Grad-CAM [12], we also conduct human evaluation on the
widely used platform (AMT: Amazon Mechanical Turk) to further evaluate the
quality of overall activation maps generated by different methods. Based on
Table II, two comparison pairs are formed as “Decomposition” vs. “Grad-CAM”
and “Decomposition” vs. “Grad-CAM (no norm)”. For each comparison pair, the
AMT raters are asked to carefully check the visual explanation generated from
two different methods as shown in Fig. 5. Each visual explanation contains an
image pair with birds from the same species and the top activated regions
($2\%$ pixels) are highlighted based on the overall activation map of one
competing method. We denote these two methods as robot A and robot B in the
questionnaire. The raters will decide which robot they think is more
reasonable in a five-point Likert scale as shown in Fig. 5.

$720$ evaluations of comparison pairs are collected from at least $40$
different AMT workers for two comparison pairs (half for each) and the result
is consistent with the outcome of Section VI-A as shown in Fig. 6. Over $60\%$
evaluations support that the proposed method is clearly more reasonable than
the vanilla Grad-CAM. Removing the normalization for the gradient computing
does improve the quality of Grad-CAM’s activation map significantly, but the
proposed method still holds more supportive evaluations ($41.1\%$ vs.
$25.2\%$) in the AMT test.

Figure 7: Qualitative results (overall activation maps corresponding to
different metric learning losses) for model diagnosis. Although the metric
learning models with two different losses yield similar training accuracies
(82.69% and 82.98% in Table III), their overall activation maps generated by
the activation decomposition framework reveal great differences in the regions
to which the models pay attention. These activation maps also provide valuable
insights on the model generalization ability to the validation/test set.

###  VI-C Model Diagnosis

A large number of loss functions have been proposed for metric learning [2, 1,
7], while only the performance and embedding distribution are evaluated. We
show that the overall activation map can also help evaluate the generalization
ability of different metric learning methods.

TABLE III: Top-1 recall accuracy on CUB. Loss | Train | Validation
---|---|---
MS | 82.69% | 65.45%
Triplet | 82.98% | 60.55%
TABLE IV: Localization accuracy (IoU=0.5) on CUB training set. | Decomposition+Bias | Decomposition
---|---|---
MS | 52.10% | 58.99%
Triplet | 39.93% | 55.60%
Accuracy Drop | 12.17% | 3.39%

We follow the setting of [7] and train two metric learning models with
Multiple Similarity (MS) loss [7] and Triplet loss [2], respectively. As shown
in Table III, although they have almost the same accuracy on the training set,
there is a big gap between their generalization abilities on the validation
set. Before checking the result on the validation set, is there any clue in
the training set for such gap? Despite the similar training accuracies, the
predictions of two models are actually based on different regions from the
activation maps generated by our activation decomposition framework (see Fig.
7). The activation maps also provide valuable insights on the model
generalization ability to the validation/test set. It is evident from Fig. 7,
the model with MS loss is more focused on the salient object area than the one
with Triplet loss, which leads to better validation performance as confirmed
in Table III. The quantitative results of localization in Table IV also
support the fact that “Triplet” model is more likely to focus on the
background region rather than the object as we witness the localization
accuracy drop for Triplet loss. Although “Decomposition+Bias” can be more
sensitive to different loss functions, implying that the bias term does
provide valuable information on the overall map, both settings of the proposed
method have the same trend on accuracy. Therefore, our activation
decomposition framework can shed light on the generalization ability of loss
functions for metric learning, thereby providing an useful tool for model
diagnosis.

##  VII Evaluating Point-Specific Activation Map

The proposed activation decomposition framework not only generates the overall
activation map, but also can produce the point-specific activation map for
fine-grained visual explanation of metric learning. In this section, we
introduce two applications to demonstrate the importance of the proposed
point-specific activation map.

###  VII-A Application I: Cross-view Pattern Discovery

Figure 8: An example of illustrating cross-view pattern discovery – image
orientation estimation – by using the activation maps from the metric learning
(i.e. image matching/retrieval) model. Based on the overall activation maps
from two views, the estimated orientation angle between two views is
$211.9^{\circ}$. By generating the point-specific activation map of a maximum
activated region in the other view, the estimated orientation is much more
accurate, i.e. $31.9^{\circ}$, with only $1.9^{\circ}$ of angle difference as
compared with the ground truth in this example.

When metric learning is applied to cross-view applications, e.g. image
retrieval [28, 4, 29], the model is capable of learning similar patterns in
different views which may provide geometric information of the two views, e.g.
the camera pose or orientation information. We conduct orientation estimation
experiment to show the advantage of point-specific activation map compared
with the overall activation map on providing geometric information based on
cross-view patterns. In our experiment, we take street-to-aerial image geo-
localization [28, 4] as an example.

The objective of cross-view image geo-localization is to find the best matched
aerial-view image (with GPS information) in a reference dataset for a query
street-view image. We conduct the experiment on CVUSA [28], which is the most
popular benchmark for this problem containing 35,532 training pairs and 8,884
test pairs. Each pair consists of a query street image and the corresponding
orientation-aligned aerial image at the same GPS location. For example, as
shown in Fig. 8, the left two images (street-view panorama and aerial-view
images) are the original aligned image pair and the yellow line denotes the
$0^{\circ}$ which corresponds to the South direction ($180^{\circ}$
corresponds to the North direction). We use $[0,360]^{\circ}$ to denote
different angles as marked on those two images.

We train a simple Siamese-VGG [18] network following the modified triplet loss
in [4]. During the training, if the image pairs are not aligned (e.g. randomly
rotate the aerial view images), the activation map can be used for orientation
estimation. Specifically, we train the Siamese-VGG with randomly rotated
aerial images so that the model is approximately rotation-invariant and so is
the overall activation map for aerial image. In the example of Fig. 8, the
overall activation maps for the original aligned pair are first generated to
show the highlighted regions contributing to the similarity score, and both
views focus on the road areas in this example. The most activated regions are
highly relevant in both views and most of them are similar patterns. When we
randomly rotate the aerial image ($30^{\circ}$ in this example and is denoted
as the ground truth rotation angle in Fig. 8), the aerial view activation map
still focuses on the road area which is relevant to the street view.

Figure 9: The results of cross-view image orientation estimation (i.e. angle
error distributions) obtained by the overall and point-specific maps based on
the image matching experiment on the CVUSA dataset.

In this paper, we simply use the pixel with the maximum activation value for
orientation estimation, because the most activated areas (highlighted in cyan
boxes in Fig. 8) from two views are likely to be relevant. For the street-view
image, the selected pixel lies in the angle of $90.2^{\circ}$ (the cyan line)
based on the angle marks on the left aligned image. For the rotated aerial
view, the selected pixel lies in the angle of $302.1^{\circ}$ (the cyan line).
Since the overall map contains multiple activated regions, the selected
(maximum activation) pixels in both views actually do not correspond to the
same object, which reveals one disadvantage of using only the overall
activation map. In this example, the estimated angle is
$302.1^{\circ}-90.2^{\circ}=211.9^{\circ}$, which is not correct. However, for
the selected pixel (the one with the maximum activation value) in the street-
view image, if we generate its corresponding point-specific activation map on
the aerial-view image (as shown on the very right of Fig. 8), the new selected
pixel (maximum activation) on this point-specific activation map lies in the
angle of $122.1^{\circ}$ (the red line). Then, the estimated angle is
calculated as $122.1^{\circ}-90.2^{\circ}=31.9^{\circ}$, which is very close
to the ground truth ($30^{\circ}$). This demonstrates the advantage of the
point-specific activation decomposition for finding more fine-grained
information in this application.

Figure 10: Top retrieved images by interactive retrieval on face and re-id
datasets. The first column shows the query images and the red box on each of
them indicates the region of interest (RoI). In the last row, the blue
bounding box highlights a failure retrieval case, which is further analyzed in
Fig. 13.

We also present the quantitative results in Fig. 9. The angle error is
computed by error = ground truth - estimated angle. We then add or subtract
$360^{\circ}$ to the error to fit it in the range of $[-180,180]^{\circ}$ if
the absolute value of the error is greater than $180^{\circ}$. For example,
when the error is $359^{\circ}$, we subtract it by $360^{\circ}$ and get
$-1^{\circ}$ as the error. This is a reasonable setting adopted from the
previous work [28]. As evident from Fig. 9, the point-specific activation map
significantly outperforms the overall map for cross-view orientation
estimation. Point-specific activation based orientation estimation has over
$16\%$ samples (a total of 8,884 test sample pairs in the CVUSA dataset) with
angle error less than $\pm 3.5^{\circ}$ (the red bar at $0^{\circ}$ in Fig.
9), while overall activation map based method has less than $12\%$. The result
further validates the superiority of the point-specific map compared with the
overall map. Moreover, we also provide a demo222Please check the demo (the
“Geo-localization” section) at https://github.com/Jeff-
Zilence/Explain_Metric_Learning to show how the point-specific map changes
according to the query pixel.

###  VII-B Application II: Interactive Retrieval

Verification applications like face recognition and person re-id usually
retrieve the most similar images to the query image. However, we may be more
interested in a specific RoI (region of interest) than the overall image for
certain circumstances, for example finding people with a similar bag or pair
of shoes (i.e. RoI) in surveillance applications. Since our framework provides
the point-to-point activation map, it can be a reasonable tool for measuring
partial similarity to serve this purpose. With the partial similarity, we are
able to interactively retrieve images with only parts/regions similar to the
query image instead of the overall most similar images. As shown in Fig. 10,
the point-specific activation generated by the proposed method works well on
retrieving people with similar clothes and faces with similar beard. No
explicit supervision is adopted except the pre-trained retrieval model.

Specifically, we follow the pipeline of recent approaches on face recognition
[26] and person re-id [3]. For re-id, the model is trained and evaluated on
Market- 1501 [30] where some validation images only contain a small part of a
person. Although the model is trained with Euclidean distance without L2
normalization, cosine similarity still works well as the evaluation metric.
For face recognition, we take the trained model on CASIA-WebFace [31] and
evaluate it on FIW [32] where the face identification and kinship relationship
are available.

Figure 11: Human evaluation interface for interactive retrieval.

Figure 12: Human evaluation results on the point-specific (interactive) vs.
overall retrieval for face verification (first row) and person re-
identification (second row).

Figure 13: Explanation of the failure case by the point-specific activation
map.

The interactive search is conducted by simply matching the equivalent partial
feature in Eq. 5 ($W^{q}_{i,j}A^{q}_{i,j}$) with the reference embedding
features. We first compute $W^{q}_{i,j}A^{q}_{i,j}$ as the feature of each
position in the last convolutional layer, and a bilinear interpolation is
adopted to generate the feature for every pixel of the original image. For a
point of interest $(i,j)$, we compute the cosine similarity between the
calculated feature on $(i,j)$ and the embedding feature of the reference
images as the point-specific similarity (note that the equivalent feature is
normalized with the $l_{2}$ norm of the original embedding as $|E^{q}|$). In
this case, the embedding features of the reference dataset do not have to be
recomputed. This similarity can be also considered as the summation of the
values in the point-specific map corresponding to $(i,j)$. In the case of Eq.
2 (CNN+GAP), the similarity can be simplified as
$\sum_{x,y}(\sum_{k}A^{q}_{i,j,k}A^{r}_{x,y,k})/|E^{q}||E^{r}|$ ($E^{q}$ and
$E^{r}$ denote the original embedding features of the query and retrieved
images).

Apparently, the images retrieved by the original feature are likely to be from
the same identity as the query image, because the models are trained for
identification task. However, the proposed interactive retrieval is able to
retrieve different images based on the same query image with different RoIs as
shown in Fig. 10. To further quantitatively evaluate the advantage of the
proposed point-specific map, we conduct human evaluation on the overall
retrieval (based on the original feature) vs. the interactive retrieval (based
on the point-specific map). Similar to Section VI-B, the AMT workers are asked
to decide which retrieval result is better given a specific RoI (see Fig. 11).
We conduct experiments on both face verification and person re-id tasks. 360
evaluations of comparison pairs in total are collected from at least 40
different AMT workers. As shown in Fig. 13, the interactive retrieval holds
considerably more supportive evaluations than the overall retrieval ($57.5\%$
vs. $28.0\%$ on face, and $47.5\%$ vs. $31.2\%$ on person re-id), which
demonstrates the effectiveness of the proposed point-specific activation map.

However, we do find some failure cases as shown in the last row of Fig. 10
with the blue bounding box. There is no shoes in the failure image, but why is
this image retrieved with a top rank (i.e. $6$th rank)? We provide an in-depth
analysis from the perspectives of both overall and point-specific activation
maps in Fig. 13. In the overall activation map, three regions of the query
image have a high activation on the retrieved image. From the point-specific
activation maps, there is a high activation on the purse in the retrieved
image corresponding to the shoes in the query image. This might be because of
their similar red color and the arm of the retrieved person may appear like a
leg from a specific viewpoint. This example also validates the importance of
the point-specific activation map generated by our framework for explanation.

Comparison with Cropping/Masking. One may argue that another possible way to
achieve interactive retrieval is to crop or mask out a specific RoI. Then we
can generate the feature with the specific part of image and use this feature
as query to retrieve images. However, generally speaking, this operation will
cause the feature to be very different from features of the original images,
thus would not generate satisfactory retrieval results. And we validate this
by providing the qualitative comparison of retrieval results in Figs. 14 and
15.

Figure 14: Comparison between our method and cropping/masking for interactive
retrieval on face verification. Query image is in the first column. Figure 15:
Comparison between our method and cropping/masking for interactive retrieval
on person re-identification. Query image is in the first column.

For face verification, flattened features followed by FC layers are usually
used, because the input image size is relatively small and flattened features
reserve all the information in the last convolutional layer. However, this
architecture does not allow change in the input size, so one may need to
resize the image to the original size after cropping a specific RoI. Another
way is to assign $0$ for all pixels outside the RoI. The comparison in Fig. 14
shows that cropping with resizing or masking are not able to generate
satisfactory results.

For person re-identification, GAP is usually used so that we do not need to
resize the cropped image. The results in Fig. 15 also show the superiority of
the proposed method over cropping/masking.

##  VIII Conclusion

We propose a simple yet effective framework for visual explanation of deep
metric learning based on the idea of activation decomposition. The framework
is applicable to a host of applications, e.g. image retrieval, face
recognition, person re-id, geo-localization, etc. Experiments show the
importance of visual explanation for metric learning as well as the
superiority of both the overall and point-specific activation maps generated
by the proposed method. Furthermore, we introduce two applications, i.e.
cross-view pattern discovery and interactive retrieval, which reveal the
importance of the point-specific activation map for explanation. Our work also
points to interesting directions in exploring the point-specific activation
maps for fine-grained information discovery and analysis.

## References

  * [1] H. Oh Song, Y. Xiang, S. Jegelka, and S. Savarese, “Deep metric learning via lifted structured feature embedding,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [2] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015.
  * [3] H. Luo, Y. Gu, X. Liao, S. Lai, and W. Jiang, “Bag of tricks and a strong baseline for deep person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2019, pp. 0–0.
  * [4] S. Hu, M. Feng, R. M. Nguyen, and G. Hee Lee, “Cvm-net: Cross-view matching network for image-based ground-to-aerial geo-localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7258–7267.
  * [5] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. Torr, “Fully-convolutional siamese networks for object tracking,” in _European conference on computer vision_. Springer, 2016, pp. 850–865.
  * [6] S. Chopra, R. Hadsell, Y. LeCun _et al._ , “Learning a similarity metric discriminatively, with application to face verification,” in _CVPR (1)_ , 2005, pp. 539–546.
  * [7] X. Wang, X. Han, W. Huang, D. Dong, and M. R. Scott, “Multi-similarity loss with general pair weighting for deep metric learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 5022–5030.
  * [8] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all convolutional net,” _arXiv preprint arXiv:1412.6806_ , 2014.
  * [9] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Learning deep features for discriminative localization,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [10] B. Zhou, D. Bau, A. Oliva, and A. Torralba, “Interpreting deep visual representations via network dissection,” _IEEE transactions on pattern analysis and machine intelligence_ , 2018.
  * [11] R. C. Fong and A. Vedaldi, “Interpretable explanations of black boxes by meaningful perturbation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 3429–3437.
  * [12] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-cam: Visual explanations from deep networks via gradient-based localization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 618–626.
  * [13] J. Zhang, S. A. Bargal, Z. Lin, J. Brandt, X. Shen, and S. Sclaroff, “Top-down neural attention by excitation backprop,” _International Journal of Computer Vision_ , vol. 126, no. 10, pp. 1084–1102, 2018.
  * [14] A. Mahendran and A. Vedaldi, “Understanding deep image representations by inverting them,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 5188–5196.
  * [15] W. Yang, H. Huang, Z. Zhang, X. Chen, K. Huang, and S. Zhang, “Towards rich feature discovery with class activation maps augmentation for person re-identification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1389–1398.
  * [16] A. Gordo and D. Larlus, “Beyond instance-level image retrieval: Leveraging captions to learn a global visual representation for semantic retrieval,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6589–6598.
  * [17] B. Zhou, Y. Sun, D. Bau, and A. Torralba, “Interpretable basis decomposition for visual explanation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 119–134.
  * [18] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [19] A. Stylianou, R. Souvenir, and R. Pless, “Visualizing deep similarity networks,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 2029–2037.
  * [20] L. Chen, J. Chen, H. Hajimirsadeghi, and G. Mori, “Adapting grad-cam for embedding networks,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 2794–2803.
  * [21] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim, “Sanity checks for saliency maps,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 9505–9515.
  * [22] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in _2018 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2018, pp. 839–847.
  * [23] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “Sphereface: Deep hypersphere embedding for face recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 212–220.
  * [24] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “Cosface: Large margin cosine loss for deep face recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5265–5274.
  * [25] E. Ristani and C. Tomasi, “Features for multi-target multi-camera tracking and re-identification,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.
  * [26] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [27] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona, “Caltech-UCSD Birds 200,” California Institute of Technology, Tech. Rep. CNS-TR-2010-001, 2010.
  * [28] M. Zhai, Z. Bessinger, S. Workman, and N. Jacobs, “Predicting ground-level scene layout from aerial imagery,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 867–875.
  * [29] Y. Tian, C. Chen, and M. Shah, “Cross-view image matching for geo-localization in urban environments,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3608–3616.
  * [30] L. Zheng, L. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian, “Scalable person re-identification: A benchmark,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1116–1124.
  * [31] D. Yi, Z. Lei, S. Liao, and S. Z. Li, “Learning face representation from scratch,” _arXiv preprint arXiv:1411.7923_ , 2014.
  * [32] J. P. Robinson, M. Shao, Y. Wu, H. Liu, T. Gillis, and Y. Fu, “Visual kinship recognition of families in the wild,” in _IEEE Transactions on pattern analysis and machine intelligence_ , 2018.

##  Appendix A Grad-CAM for Metric Learning

For metric learning architecture discussed in Section IV-A of the main paper,
the similarity is formulated as $S=\frac{E^{q}\cdot E^{r}}{|E^{q}||E^{r}|}$,
where $E^{q}\in\mathbb{R}^{l}$ and $E^{r}\in\mathbb{R}^{l}$ are the embedding
vectors of the query and retrieved images. $|x|$ denotes the L2 norm and
$a\cdot b$ is the inner product of $a$ and $b$. The Grad-CAM map of the query
image is given by:

$GradCAM_{i,j}=GAP\left(\frac{\partial S}{\partial A^{q}}\right)A^{q}_{i,j}$
(11)

With the gradient chain rule, the gradient is written as:

$\displaystyle\frac{\partial S}{\partial A^{q}}$
$\displaystyle=\frac{\partial\frac{(E^{r})^{T}E^{q}}{|E^{q}||E^{r}|}}{\partial
A^{q}}=\left(\frac{E^{r}}{|E^{r}|}\right)^{T}\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}\frac{\partial E^{q}}{\partial A^{q}}$ (12)
$\displaystyle=\left(\frac{E^{r}}{|E^{r}|}\right)^{T}\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}W^{q}$

By expanding $E^{r}$ as $\sum_{x,y}W^{r}_{x,y}A^{r}_{x,y}+B^{r}$ (Section
IV-A) and merging Eq. 11 with Eq. 12, the Grad-CAM map is reformulated as:

$\displaystyle GradCAM_{i,j}$
$\displaystyle=GAP\left(\left(\frac{E^{r}}{|E^{r}|}\right)^{T}\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}W^{q}\right)A^{q}_{i,j}$ (13)
$\displaystyle=\frac{1}{Z}\left(\sum_{i^{*},j^{*}}(E^{r})^{T}\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}W_{i^{*},j^{*}}^{q}\right)A^{q}_{i,j}$
$\displaystyle=\frac{1}{Z}(E^{r})^{T}\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}\left(\sum_{i^{*},j^{*}}W_{i^{*},j^{*}}^{q}\right)A^{q}_{i,j}$
$\displaystyle=\frac{1}{Z}\left(\sum_{x,y}W^{r}_{x,y}A^{r}_{x,y}+B^{r}\right)\cdot\left(\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}GAP(W^{q})A^{q}_{i,j}\right)$
$\displaystyle=\frac{1}{Z}\left(\frac{\partial(E^{q}/|E^{q}|)}{\partial
E^{q}}GAP(W^{q})A^{q}_{i,j}\right)\cdot\left(\sum_{x,y}W^{r}_{x,y}A^{r}_{x,y}+B^{r}\right)$

Here the $Z$ is the normalization term for simplicity. $\frac{\partial
E/|E|}{\partial E}$ is the $l\times l$ Jacobian matrix given by:

$\small\frac{\partial(E/|E|)}{\partial
E}=\Big{(}\frac{\partial(E_{i}/|E|)}{\partial
E_{j}}\Big{)}_{i,j}=\left\\{\begin{aligned}
\frac{1}{|E|}\left(1-\frac{E_{i}^{2}}{|E|^{2}}\right)&&i=j\\\
-\frac{E_{i}E_{j}}{|E|^{3}}&&i\neq j\end{aligned}\right.$ (14)

$\frac{1}{|E|}$ is the normalization term. The $\frac{\partial E/|E|}{\partial
E}$ term can be removed, if we compute the gradient from $E^{q}\cdot E^{r}$
instead of $\frac{E^{q}\cdot E^{r}}{|E^{q}||E^{r}|}$. For the dominant channel
$i$, the weight $(1-\frac{E^{2}_{i}}{|E|^{2}})$ is small resulting in a more
scattered activation map.

##  Appendix B Qualitative Results for Cross-view Pattern Discovery

Figure 16: Examples of cross-view pattern discovery, i.e., image orientation
estimation. (Best viewed in color)

##  Appendix C Qualitative Results for Interactive Retrieval

###  C-A Face Verification

Figure 17: Top retrieved images by interactive retrieval on face recognition.
Red boxes on the query images (first column) highlight the Region of Interest
(RoI).

###  C-B Person Re-ID

Figure 18: Top retrieved images by interactive retrieval on person re-
identification. Red boxes on the query images (first column) highlight the
Region of Interest (RoI).


# PRISM: A Design Framework for Open-Source Foundation Model Safety

Terrence Neumann
McCombs School of Business
University of Texas at Austin
Bryan Jones
School of Law
University of Texas at Austin

###### Abstract

The rapid advancement of open-source foundation models has brought
transparency and accessibility to this groundbreaking technology. However,
this openness has also enabled the development of highly-capable, unsafe
models, as exemplified by recent instances such as WormGPT and FraudGPT, which
are specifically designed to facilitate criminal activity. As the capabilities
of open foundation models continue to grow, potentially outpacing those of
closed-source models, the risk of misuse by bad actors poses an increasingly
serious threat to society. This paper addresses the critical question of how
open foundation model developers should approach model safety in light of
these challenges. Our analysis reveals that open-source foundation model
companies often provide less restrictive acceptable use policies (AUPs)
compared to their closed-source counterparts, likely due to the inherent
difficulties in enforcing such policies once the models are released. To
tackle this issue, we introduce PRISM, a design framework for open-source
foundation model safety that emphasizes Private, Robust, Independent Safety
measures, at Minimal marginal cost of compute. The PRISM framework proposes
the use of modular functions that moderate prompts and outputs independently
of the core language model, offering a more adaptable and resilient approach
to safety compared to the brittle reinforcement learning methods currently
used for value alignment. By focusing on identifying AUP violations and
engaging the developer community in establishing consensus around safety
design decisions, PRISM aims to create a safer open-source ecosystem that
maximizes the potential of these powerful technologies while minimizing the
risks to individuals and society as a whole.

## INTRODUCTION

The rapid advancement of open-source foundation models has led to the
widespread release of powerful tools that can be easily fine-tuned for various
purposes, both beneficial and malicious. While open-source development is
celebrated for its accessibility, transparency, and enhanced privacy for end-
users compared to API-restricted access, it also presents significant
challenges in terms of model safety. Recent examples like WormGPT and
FraudGPT, which facilitate cybercrime by fine-tuning open large language
models, underscore the urgent need for robust safety measures in open-source
AI development.

One of the primary issues is the difficulty open-source developers face in
monitoring and enforcing their acceptable use policies (AUPs) (Klyman 2024).
This lack of oversight makes the models vulnerable to misuse by malicious
actors, posing a considerable risk to society. Our research, which includes an
analysis of a dataset compiling AUPs from various foundation model companies,
reveals that open-source developers tend to provide fewer policies across many
categories of potential harms compared to their closed-model counterparts.
This discrepancy may stem from the inherent challenges in enforcing such
policies once the models are released or from the open-source philosophy that
values accessibility and innovation over strict limitations.

These enforcement challenges and the resulting policy gaps raise a critical
question: _how should open foundation model developers approach model safety_?
To address this, we introduce PRISM, a forward-looking framework designed to
guide open-source foundation model development. PRISM stands for Private,
Robust, Independent Safety at Minimal marginal cost of compute. This framework
emphasizes the importance of integrating robust safety measures that do not
compromise end-user privacy and are not dependent on the specific language
model architecture. By focusing on these principles, PRISM aims to create a
safer open-source environment without imposing significant additional
computational costs on developers or users.

In this paper, we begin by providing a background section on open and closed
foundation model development, covering key definitions, notions of model
safety and vulnerabilities, explaining the capability gaps between open and
closed foundation models, and discussing how AUPs are implemented and enforced
to self-regulate the use of foundation models. In the next section, we propose
the development of a large language model that embodies the PRISM principles.
By adopting the PRISM design principles, we argue that it is possible to
achieve utility improvements for both end-users and society-at-large.
Implementing privacy-preserving techniques, enhancing model robustness, and
ensuring that safety measures are model-independent and cost-effective can
help mitigate the risks associated with open foundation models. Our goal is to
foster a responsible open-source community that balances the benefits of
accessibility and transparency with the imperative of protecting against
misuse and harm.

## BACKGROUND & MOTIVATION

###  A Working Definition of Open and Closed Foundation Models

A foundation model is “any model that is trained on broad data (generally
using self-supervision at scale) that can be adapted (e.g., fine-tuned) to a
wide range of downstream tasks” (Bommasani . 2021, p.3). Open foundation model
development is an evolving concept, as many publications have described a
“gradient” of openness (Solaiman 2023), with various levels of transparency
into model design. For instance, foundation model developers can open-source
the model weights, but not the model code or training data, leaving questions
as to the ultimate transparency or “openness” of the foundation model.

We define an open foundation model as _a foundation model in which the pre-
trained weights are available for end-users to download_ , acknowledging that
some may define openness differently. Conversely, we define a closed
foundation model as _a foundation model in which direct end-user access to
model weights is restricted_ , with model access achieved via third-party API
requests.

###  Open and Closed Foundation Model Safety and Vulnerabilities

Foundation models, regardless of whether they are open or closed, present
challenges to society. Propaganda generated by large language models (LLMs)
can be more convincing than human-created content (Spitale . 2023), and
adversarial actors can save costs using LLMs for influence operations (Musser
2023). There is also a risk of proliferating child sexual abuse material via
image generation models (Thiel . 2023).

There has been significant debate about the safety of open foundation models
in particular, as their advanced capabilities and lack of centralized
governance present substantial risks of misuse. Some argue for more restricted
release, citing threats like influence operations, surveillance, scamming,
cyber-attacks, and the development of banned weapons (Seger . 2023). Others
argue that the threats are overblown and that open models offer societal
benefits such as accelerating science, allowing broader definitions of
acceptable behavior, and enhancing economic innovation (Kapoor . 2024).

To mitigate these threats, developers have attempted to align models with
human values through reinforcement learning (Bai . 2022). However, the learned
reward functions often fail to capture the subtlety of human values (Bommasani
. 2021), especially when values conflict. This alignment approach is
vulnerable to “jailbreak” attacks like prompt injection (Chao . 2023) and
malicious fine-tuning (Qi . 2023), which can drastically increase unsafe
outputs. The brittleness of reinforcement learning for safety is evident, as
only 3% of neurons are uniquely responsible for safe outputs in some models
(Wei . 2024). Real-world examples like WormGPT and
FraudGPT111https://thehackernews.com/2023/07/new-ai-tool-fraudgpt-emerges-
tailored.html222https://thehackernews.com/2023/07/wormgpt-new-ai-tool-
allows.html demonstrate this vulnerability. Such models have been fine-tuned
to provide criminal or otherwise harmful outputs, ranging from hacking
instructions, malicious codes, to detailed plans on how to conduct advanced
phishing schemes. Some have described “catastrophic robustness failures” of
model alignment and safety as among the greatest challenges facing open
foundation model developers (Bommasani . 2021).

This paper presents an alternative safety design that identifies unsafe
prompts or outputs via independent models, rather than relying on the complex
process of reinforcement learning to align with diverse human values.

###  The (Narrowing) Capability Gap Between Open and Closed Foundation Models

Figure 1: The evolving capabilities of open- and closed-source language models
over time. Open-source models are improving at a rate at least as fast as
closed-source models. Source: (Chiang . 2024)

The capabilities of open foundation models are advancing at a rate that may
outpace that of closed models, driven by the collective efforts of a global
community of researchers and developers. Chiang . (2024) have gathered
millions of rows of human feedback data in which model output from many models
are compared pairwise (e.g., “Do you prefer the output of Model A or Model
B?”) and models are ranked via a method that tabulates wins and losses similar
to how chess player rankings are calculated (Boubdir . 2023). By compiling
these scores for nearly 100 model releases over the past year and identifying
the models as either open- or closed-source, we can determine the rate of
improvement of both model types over time. From a simple regression analysis
of this data, we see that the coefficient on the daily rate of capability
improvement of open-source models is greater ($\beta_{open}=0.485$), but not
statistically different, than the daily rate of capability improvement of
closed-source models ($\beta_{closed}=0.347$). See Figure 1 and Table 1.

| Estimate (Std. Error)
---|---
(Intercept) | $1055\textsuperscript{***}$
| $(28.2)$
Release Date | $0.3470\textsuperscript{***}$
| $(0.080)$
Open-Source | $-149.9\textsuperscript{***}$
| $(32.5)$
Release Date:Open-Source | $0.1383$
| $(0.095)$
$R^{2}=0.698$
${}^{***}p<0.001$, ${}^{**}p<0.01$, ${}^{*}p<0.05$
Table 1: Results for regression of model Elo score on Release Date (indicating
the daily rate of capability improvement), whether they are open-source, and
the interaction between the two.

If this accelerated rate of improvement persists, open-source models may well
become the predominant mode of foundation model development and usage,
particularly among businesses with the resources to host and customize these
models. The cost-effectiveness of open-source models, which eliminates the
need for expensive per-inference fees, is a significant advantage for
businesses seeking to leverage advanced AI technologies.

The AI landscape is adversarial and dynamic, characterized by attackers intent
on causing societal harm and the constantly evolving capabilities of
foundation models, their defenses, and the attackers. Thus, as the
capabilities of open-source foundation models increase, so too does the
potential for misuse. The advancement in capabilities not only amplifies
existing threat vectors but also introduces new ones that may have been
previously unconsidered. This dynamic nature of emerging threats necessitates
a continuous and adaptable approach to safety and control measures. The open-
source nature of these models, while promoting innovation and accessibility,
simultaneously poses a challenge in maintaining robust oversight and ensuring
alignment with societal values. As such, the process of safeguarding these
models must be iterative, involving regular updates and enhancements to
security protocols, threat assessments, and ethical guidelines.

###  Self-Governance via Acceptable Use Policies of Foundation Models

While some government guidelines exist limiting or restricting the use of
foundation models to unsafe applications, in general, most end-user
restrictions are not comprehensively defined by governments at this point. For
instance, the FTC has issued warnings against the use of AI to make false or
unsubstantiated claims, emphasizing that companies must ensure their AI-
generated content is truthful and not misleading (Atleson 2023). However,
broader regulatory frameworks specifically addressing the deployment and use
of foundation models in various sectors, including the public sector (Jones
2023), are still evolving.

In response to potential misuse of their models, foundation model companies
often make end-users agree to specific acceptable use policies (AUPs) that
define restrictions on model usage. These AUPs cover often cover a wide range
of potential societal harms, ranging from the production of mis- and dis-
information, to privacy violations, to encouraging self-harm or other
dangerous behavior. However, enforcement of acceptable use policies is
difficult in practice. As recently as last year, researchers demonstrated that
there was little difference between closed- and open-source large language
models in their defenses against malicious fine-tuning attacks (Qi . 2023).
However, closed-model companies theoretically have greater centralized means
to dynamically update “content moderation” style filters as new threats
emerge, while open foundation model developers currently forgo monitoring and
updating the system once it is in the hands of end-users.

In order to analyze whether open- or closed-source models implement more
restrictive AUPs, we examine a new AUP dataset (Klyman 2024), adding a
variable to distinguish companies as either open-source or closed-source
developers based on publicly available information. We then compared the
number of individual policies addressing different types of harm to society
using the taxomy provided by Klyman (2024) as inspiration. Our analysis
reveals that, in most harm categories of AUPs, closed-source models have a
higher average number of policies, indicating more restrictive usage. For
instance, we see higher average values for the number of AUPs related to:
challenging types of content and expression ($\mu_{open}=2.61$;
$\mu_{closed}=2.88$); discrimination and bias ($\mu_{open}=1.69$;
$\mu_{closed}=2.00$); sensitive economic and justice applications
($\mu_{open}=1.46$; $\mu_{closed}=2.00$); misinformation and deception
($\mu_{open}=2.69$; $\mu_{closed}=3.41$); and malicious and illegal activities
($\mu_{open}=4.46$; $\mu_{closed}=5.23$). However, we see open source
foundation models having higher number of AUPs related to: encouraging self-
harm or other dangerous behavior ($\mu_{open}=0.85$; $\mu_{closed}=0.76$); and
sensitive and high-risk applications, which include numerous national security
threats ($\mu_{open}=2.23$; $\mu_{closed}=1.11$). See Figure 2 for a
visualization of this data.

Figure 2: This chart shows the mean number of acceptable use policies (AUPs)
for open- and closed-source foundation models across a variety of categories
of potential harm. In many cases, closed-source models seem to provide more
terms of use intended to mitigate societal harm.

The differences in AUPs between open-source and closed-source models may stem
from the inherent principles of openness and accessibility in open-source
development, which prioritize innovation and collaboration over stringent
usage limitations. Additionally, open-source developers recognize the
practical challenges of enforcing terms of use once models are released, given
their free accessibility and potential for modification. As a result, they may
opt for less comprehensive AUPs and rely on decentralized, voluntary self-
enforcement by the user community. In contrast, closed-source model developers
have more centralized control and resources to implement and enforce rigorous
usage restrictions. The disparity in AUP comprehensiveness reflects these
differing priorities and practicalities in model governance and safety
management between open-source and closed-source models.

## THE PRISM DESIGN FRAMEWORK FOR OPEN-SOURCE FOUNDATION MODEL SAFETY

The previous section highlighted critical elements in the open-source
foundation model safety debate. We noted that reinforcement learning, the
primary paradigm for safety alignment, is vulnerable to attacks optimized when
attackers access model weights. We also discussed the rapidly expanding
capabilities of open-source models, potentially outpacing closed-source
counterparts, which amplifies both their potential uses and misuses.
Furthermore, we showed that open-source models generally have fewer AUPs
across various societal harm categories, either due to their "openness" ethos
or the acknowledged challenges in monitoring model usage.

In response to these concerns, we propose an innovative open-source LLM that
tackles the practical challenges of open foundation model safety head-on. Our
groundbreaking model embodies three core principles: (1) privacy, (2) robust,
model-independent safety, and (3) minimizing the marginal cost of compute. In
the following sections, we will delve into how each component delivers utility
gains for end-users and society as a whole, paving the way for a safer and
more responsible open-source AI landscape.

###  Large Language Model Formulation

In this section, we formulate a safety mechanism for a language model $f$ that
embodies the PRISM principles. Let a pre-trained large language model be
denoted as a function $f$ which, during inference, takes an input sequence of
tokens (a prompt) $x=(x_{1},x_{2},\dots,x_{N})$ and has outputted tokens
$y_{1},y_{2},...,y_{j}$ so far. The function $f$ outputs a probability
distribution over the vocabulary $V$ for the next output token:
$f(x,y_{1},y_{2},\dots,y_{j})=P(y_{j+1}=v|x,y_{1},y_{2},\dots,y_{j})$ for
$v\in V$. By iteratively selecting the next token with the highest
probability, a completed output sequence $Y=(y_{1},y_{2},\dots,y_{J})$ is
produced.

Rather than directly modifying the probability distribution responsible for
producing the output sequence $Y$, which is the outcome of current
reinforcement learning approaches to safety alignment of foundation models, we
instead propose modular “interceptor” functions that moderate prompts and
outputs: $p(x)$ and $q(Y)$ respectively. Interceptor functions are language
models trained to enforce the AUPs of a given model. These functions take the
input sequence $x$ or the output sequence $Y$ as input and return a
probability of the next token in the prompt or output being “unsafe”:
$p(x)=P(x_{i+1}\text{ is ``unsafe''}|x_{1},x_{2},\dots,x_{i})$ and
$q(Y)=P(y_{j+1}\text{ is ``unsafe''}|y_{1},y_{2},\dots,y_{j})$. Then, a
“blocking” function returns a binary sequence indicating which tokens are to
be removed, given a probability threshold for being “unsafe”:
$b_{input}(x,p,\tau_{1})=(b_{1},b_{2},\dots,b_{N})$ where $b_{i}=1$ if
$p(x_{i})>\tau_{1}$ and $b_{i}=0$ otherwise for $i=1,2,\dots,N$, and
similarly, $b_{output}(Y,q,\tau_{2})=(b_{1},b_{2},\dots,b_{J})$ where
$b_{j}=1$ if $q(y_{j})>\tau_{2}$ and $b_{j}=0$ otherwise for $j=1,2,\dots,J$.
Lower values of $\tau$ indicate stricter moderation of prompts and outputs,
and the inclusion of criteria as “safe” or “unsafe” will impact the underlying
probability distribution.

Finally, numerous interceptor models may be capable of enforcing AUPs to an
acceptable standard, and let these sets be defined as $P$ and $Q$. Choosing an
inefficient or overly complex model can significantly add to computational
expense, making this a non-starter for many end-users. Thus the objective of
the system, holding the language model $f$ constant, could be characterized
as:

$\displaystyle\operatorname*{arg\,min}_{p\in P,q\in Q}\quad C(p,q)$ (1)
subject to: $\displaystyle\quad U(p,q,\tau_{1},\tau_{2})-U_{0}>0$

Where $C(p,q)$ represents the computational cost function associated with
additional models $p$ and $q$, $U(p,q,\tau_{1},\tau_{2})$ is the utility of
the entire system (however that is defined by the developer), holding $f$
fixed. $U_{0}$ is some target utility, perhaps defined by prior benchmarks.
Thus, the objective is to minimize the cost of compute associated with $p$ and
$q$ from the broader set of tested models $P$ and $Q$ subject to the
constraints that model utility does not fall below a desired threshold.

###  Private

$p$ and $q$ are not inherently privacy-preserving. In theory, end-user data
could be collected and used for further training the system. However, as
previously mentioned, a significant appeal of open foundation models is that
they can be stored on premises and used with sensitive data that contain legal
stipulations regarding user privacy. Therefore, training a robust $p$ and $q$
while maintaining end-user privacy is a challenge worth considering for open-
source foundation model developers.

One approach may be to host user hackathons and bounty programs, offering up
cash incentives for spotting model vulnerabilities or potential
attacks333https://bughunters.google.com/about/rules/google-
friends/5238081279623168/abuse-vulnerability-reward-program-rules.
Additionally, this serves as an opportunity to make a developer organization’s
choice of $p$ and $q$ transparent as well, since it will not be relying on
private end-user data. Because of the modular nature of the model, these can
be refined and updated quickly and with little cost. Community engagement can
help build consensus around challenging design decisions related to model
safety.

Further, it is possible to maintain privacy while also enforcing a policy
requiring end-users to maintain the most up-to-date interceptor models $p$ and
$q$. This could be done, for instance, through licensing. An API could
determine whether model weights for $p$ and $q$ have been updated in the
central model repository as new threats emerge. If they have been updated,
end-users would be required to update their local repository to reflect these
changes. Failure to do so would void their license, rendering model inference
impossible. Because $p$ and $q$ are independent from $f$, pushing updates to
these models should not result in system-wide issues, as fine-tuning only
affects $f$.

#### Utility Gains for End-Users

For end-users, keeping the core language model ($f$) on-premises and only
updating the smaller interceptor models ($p$ and $q$) as needed ensures that
sensitive data remains private and is not shared with model developers or
third parties. This is especially important for users in industries with
strict data privacy regulations, which may curtail their ability to interact
with closed-source model APIs (Sher Benchlouch 2023). Additionally, privacy-
as-a-policy helps mitigate many risks associated with data collection and
retention that individual users face (Shahriar . 2023).

#### Utility Gains for Society-at-Large

For society-at-large, the proposed model formulation reduces the risk of data
misuse or unintended biases being introduced into the language model by
keeping user data private and not relying on it for further training. This
helps maintain trust in the technology and its applications (Bansal . 2015).
The transparent development of interceptor models through user hackathons,
posted bounties, and community feedback encourages the establishment of widely
accepted safety standards and best practices, ensuring that the language model
is being used responsibly and in line with societal values.

###  Robust & Independent

As we previously mentioned, reinforcement learning is a particularly brittle
approach to safety (Wei . 2024). Therefore, catastrophic robustness failures
are among the greatest challenges facing open foundation model development
(Bommasani . 2021).

In order to improve safety robustness, we introduce modular $p$ and $q$
functions that moderate input and output, which should provide advantages
against both of the most common attacks - prompt injection and malicious fine-
tuning. For instance, if an attacker attempts a prompt injection attack, they
may try to access dangerous content by sneaking past defenses used when
interpreting the input prompt. However, in this context, even if a prompt
fools $p$, if the output is dangerous, it should be moderated by $q$, removing
the unsafe tokens. Relatedly, models can be fine-tuned, perhaps to be
malicious, but the prompts and outputs will still have to pass the scrutiny of
$p$ and $q$. Therefore, the safeguards that enforce the AUPs are protected
from malicious fine-tuning attempts.

Switching from reinforcement learning to a more modular approach to AI safety
may benefit the model’s overall capabilities. For instance, research has
demonstrated that a language models capabilities scale more linearly with
dataset size and number of parameters when they are optimized to predict the
next token (Kaplan . 2020). Thus, as the amount of training data and model
complexity increases, the model’s performance on language tasks improves in a
more predictable and consistent manner. By concentrating on this core
objective of language modeling, the modular approach can potentially achieve
better performance with less computational resources compared to reinforcement
learning-based methods.

#### Utility Gains for End-Users

For end-users, the introduction of modular interceptor functions $p$ and $q$
that moderate input and output improves safety robustness against common
attacks such as prompt injection and malicious fine-tuning. This can prove
extremely advantageous for businesses as end-users, who seek to limit their
liability for AI generated content. Researchers have demonstrated that
reinforcement learning approaches to model safety are so fragile that, with as
few as 10 non-malicious examples (i.e. standard business training data) (Qi .
2023), safety guardrails can accidentally be significantly disrupted. With
this modular approach, end-users could reap the scaling benefits of a pure
large language model (Kaplan . 2020), and know that the model safety is
robust.

#### Utility Gains for Society-at-Large

From a societal perspective, by providing a more resilient framework for
enforcing AUPs and mitigating the risks associated with common attacks, the
proposed formulation helps address public concerns about the potential misuse
of open-source foundation models, including for fraud and crime (Seger .
2023). The modular nature of the safety mechanisms also allows for the
incorporation of diverse perspectives and expertise in their development, such
as the use of custom dictionaries and subject-matter expert hand-crafted
datasets, further enhancing the robustness and adaptability of the system to
evolving safety challenges.

###  Minimal Cost of Compute

Given the choice between two models with the same capabilities, the end-user
is more likely to prefer a model with the smallest compute requirements. Less
computationally expensive models will be cheaper and faster to operate. Given
that the configuration of the PRISM large language model involves two
additional interceptor models $p$ and $q$, it is essential that $p$ and $q$
are significantly smaller in size than the underlying language model $f$.

The interceptor models $p$ and $q$ can be trained to learn and enforce AUPs
from the underlying large language model $f$ using a technique called
knowledge distillation (Hinton . 2015). Knowledge distillation is a process in
which a smaller model (the student) is trained to mimic the behavior of a
larger, more complex model (the teacher) by learning from the teacher’s
outputs (Hinton . 2015).

In this case, the large language model $f$ serves as the teacher, and the
interceptor models $p$ and $q$ act as the students. By exposing $f$ to a
diverse range of prompts and analyzing its outputs, the developers can
identify patterns and behaviors that align with or violate the desired AUPs.
This information can then be used to create a labeled dataset, along with
developer community feedback. The interceptor models $p$ and $q$ are trained
on this labeled dataset, learning to predict whether a given input or output
sequence is “unsafe”, or likely to violate the AUPs. By minimizing the
difference between their predictions and the labels derived from the large
language model’s behavior, the interceptor models effectively distill the
knowledge about AUPs from the teacher model into a more compact and
computationally efficient form.

#### Utility Gains for End-Users

When comparing a more efficient model to a less efficient model holding
utility constant, end-users directly benefit from cost savings for compute and
faster model performance. These two factors make minimizing the marginal
compute of safety mechanisms a worthwhile goal for model developers.

#### Utility Gains for Society-at-Large

Energy dedicated to running and processing foundation models is increasing
every day (Farahany 2024). Larger models consume more energy per inference
than smaller, more efficient models. Therefore, optimizing a model to be as
efficient offers huge potential for energy savings, especially considering
that the foundation model will be used by thousands, if not millions, of
developers throughout the world.

## DISCUSSION

###  Summary of Findings

#### Open Language Models are Improving at the Same or Greater Rate than
Closed Language Models

Open-source foundation models are rapidly advancing, with their capabilities
improving at a rate equal to or faster than their closed-source counterparts.
This accelerated progress, driven by the collective efforts of a global
research community, positions open-source models as a dominant force in the AI
landscape. As these models become more powerful and widely adopted, their
potential for both beneficial applications _and_ harmful misuse grows.

#### Open Foundation Model Companies Provide Less Restrictive AUPs than Closed
Model Companies.

Our analysis reveals that open-source foundation model companies often provide
less restrictive acceptable use policies (AUPs) compared to closed-source
model developers. This discrepancy may stem from the inherent challenges in
enforcing usage restrictions once the models are released or from the open-
source philosophy that values accessibility and innovation over strict
limitations. Consequently, open-source models may be more vulnerable to misuse
by malicious actors.

#### The PRISM Framework Offers a Solution to Vulnerabilities in Open
Foundation Models.

By introducing modular interceptor functions that independently moderate
prompts and outputs, PRISM provides a robust defense against common attacks
such as prompt injection and malicious fine-tuning. This adaptable approach
focuses on identifying AUP violations rather than directly modifying the
language model, ensuring a more resilient safety mechanism.

#### The PRISM Framework Offers Utility Gains for End-Users and Society.

For end-users, PRISM enables the use of sensitive data while maintaining
privacy and provides greater control over AUP enforcement. Minimizing the cost
of compute for the interceptor functions $p$ and $q$ makes these models more
desirable to end-users as well. On a societal level, the framework helps
address public concerns about the potential misuse of open-source foundation
models by incorporating diverse perspectives and ensuring responsible
development practices.

###  Limitations

While the PRISM framework offers a promising approach to open-source
foundation model safety, its real-world effectiveness remains to be
empirically tested. The proposed interceptor functions have not yet been
implemented and evaluated in practical scenarios, leaving questions about
their performance and reliability unanswered. Additionally, the study does not
provide a comprehensive analysis of the potential computational overhead
introduced by these interceptor functions.

Furthermore, the long-term adaptability of PRISM to the rapidly evolving AI
safety landscape requires further investigation. As new threats and challenges
emerge, the framework may need to be continually updated and refined to remain
effective. This adaptability is particularly crucial in light of the
increasing capabilities of open-source foundation models and the potential for
malicious actors to exploit vulnerabilities in novel ways.

Lastly, the study acknowledges that some exemptions from safety measures may
be necessary for research purposes. In order to foster innovation and progress
in the field of AI, researchers may require access to models without certain
restrictions to investigate novel approaches and test hypotheses.

###  Next Steps

In a future version of the paper, we will build a large language model using
the PRISM framework and empirically test the extent to which this model is (i)
more resistant to prompt injection and fine-tuning attacks, (ii) able to
remotely enforce AUPs in various contexts, and (iii) perceived as more useful
to end-users.

## References

  * Atleson (2023) atleson2023aiAtleson, M.  2023\.  Keep your AI claims in check. Keep your ai claims in check.  https://www.ftc.gov/business-guidance/blog/2023/02/keep-your-ai-claims-check.  Federal Trade Commission.  Accessed: 2024-06-03
  * Bai . (2022) bai2022trainingBai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N.others  2022\.  Training a helpful and harmless assistant with reinforcement learning from human feedback Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862.
  * Bansal . (2015) bansal2015roleBansal, G., Zahedi, M. Gefen, D.  2015\.  The role of privacy assurance mechanisms in building trust and the moderating role of privacy concern The role of privacy assurance mechanisms in building trust and the moderating role of privacy concern.  European Journal of Information Systems24624–644.
  * Bommasani . (2021) bommasani2021opportunitiesBommasani, R., Hudson, DA., Adeli, E., Altman, R., Arora, S., von Arx, S.others  2021\.  On the opportunities and risks of foundation models On the opportunities and risks of foundation models.  arXiv preprint arXiv:2108.07258.
  * Boubdir . (2023) boubdir2023eloBoubdir, M., Kim, E., Ermis, B., Hooker, S. Fadaee, M.  2023\.  Elo uncovered: Robustness and best practices in language model evaluation Elo uncovered: Robustness and best practices in language model evaluation.  arXiv preprint arXiv:2311.17295.
  * Chao . (2023) chao2023jailbreakingChao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, GJ. Wong, E.  2023\.  Jailbreaking Black Box Large Language Models in Twenty Queries. Jailbreaking black box large language models in twenty queries.
  * Chiang . (2024) chiang2024chatbotChiang, WL., Zheng, L., Sheng, Y., Angelopoulos, AN., Li, T., Li, D.Stoica, I.  2024\.  Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Chatbot arena: An open platform for evaluating llms by human preference.
  * Farahany (2024) wef2024manageFarahany, N.  2024\.  How to manage AI’s energy demand today, tomorrow and in the future. How to manage ai’s energy demand today, tomorrow and in the future.  https://www.weforum.org/agenda/2024/04/how-to-manage-ais-energy-demand-today-tomorrow-and-the-future/.  Accessed: 2024-06-06
  * Hinton . (2015) hinton2015distillingHinton, G., Vinyals, O. Dean, J.  2015\.  Distilling the Knowledge in a Neural Network. Distilling the knowledge in a neural network.
  * Jones (2023) adalovelace2023foundationJones, E.  2023\.  Foundation models in the public sector. Foundation models in the public sector.  https://www.adalovelaceinstitute.org/evidence-review/foundation-models-public-sector/ Accessed: 2024-06-03
  * Kaplan . (2020) kaplan2020scalingKaplan, J., McCandlish, S., Henighan, T., Brown, TB., Chess, B., Child, R.Amodei, D.  2020\.  Scaling Laws for Neural Language Models. Scaling laws for neural language models.
  * Kapoor . (2024) kapoor2024societalKapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P.others  2024\.  On the Societal Impact of Open Foundation Models On the societal impact of open foundation models.
  * Klyman (2024) klyman2024aupsKlyman, K.  202404\.  Acceptable Use Policies for Foundation Models: Considerations for Policymakers and Developers. Acceptable use policies for foundation models: Considerations for policymakers and developers.  Stanford Center for Research on Foundation Models.  https://crfm.stanford.edu/2024/04/08/aups.html
  * Musser (2023) musser2023costMusser, M.  2023\.  A Cost Analysis of Generative Language Models and Influence Operations. A cost analysis of generative language models and influence operations.
  * Qi . (2023) qi2023finetuningQi, X., Zeng, Y., Xie, T., Chen, PY., Jia, R., Mittal, P. Henderson, P.  2023\.  Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! Fine-tuning aligned language models compromises safety, even when users do not intend to!
  * Seger . (2023) seger2023opensourcingSeger, E., Dreksler, N., Moulange, R., Dardaman, E., Schuett, J., Wei, K.Gupta, A.  2023\.  Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives. Open-sourcing highly capable foundation models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives.
  * Shahriar . (2023) shahriar2023privacyShahriar, S., Allana, S., Hazratifard, SM. Dara, R.  2023\.  A Survey of Privacy Risks and Mitigation Strategies in the Artificial Intelligence Life Cycle A survey of privacy risks and mitigation strategies in the artificial intelligence life cycle.  IEEE Access1161829-61854.  10.1109/ACCESS.2023.3287195
  * Sher Benchlouch (2023) sher2023privacySher, G. Benchlouch, A.  2023\.  The Privacy Paradox with AI. The privacy paradox with ai.  https://www.reuters.com/legal/legalindustry/privacy-paradox-with-ai-2023-10-31/.  Accessed: 2024-05-30
  * Solaiman (2023) solaiman2023gradientSolaiman, I.  2023\.  The Gradient of Generative AI Release: Methods and Considerations. The gradient of generative ai release: Methods and considerations.  https://arxiv.org/pdf/2302.04844
  * Spitale . (2023) AIdisinfoSpitale2023Spitale, G., Biller-Andorno, N. Germani, F.  2023\.  AI model GPT-3 (dis)informs us better than humans Ai model gpt-3 (dis)informs us better than humans.  Science Advances926eadh1850.  https://www.science.org/doi/abs/10.1126/sciadv.adh1850 10.1126/sciadv.adh1850
  * Thiel . (2023) thiel2023generativeThiel, D., Stroebel, M. Portnoff, R.  2023\.  Generative ML and CSAM: Implications and Mitigations. Generative ml and csam: Implications and mitigations.  Stanford Digital Repository. https://doi. org/10.25740/jv206yg3793.
  * Wei . (2024) wei2024assessingWei, B., Huang, K., Huang, Y., Xie, T., Qi, X., Xia, M.Henderson, P.  2024\.  Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications Assessing the brittleness of safety alignment via pruning and low-rank modifications.  arXiv preprint arXiv:2402.05162.