# Temporal Graph Networks for Graph Anomaly Detection in Financial Networks

Yejin Kim1, Youngbin Lee111footnotemark: 1, Minyoung Choe2, Sungju Oh3,
Yongjae Lee1 These authors contributed equally.Corresponding author.

###### Abstract

This paper explores the utilization of Temporal Graph Networks (TGN) for
financial anomaly detection, a pressing need in the era of fintech and
digitized financial transactions. We present a comprehensive framework that
leverages TGN, capable of capturing dynamic changes in edges within financial
networks, for fraud detection. Our study compares TGN’s performance against
static Graph Neural Network (GNN) baselines, as well as cutting-edge
hypergraph neural network baselines using DGraph dataset for a realistic
financial context. Our results demonstrate that TGN significantly outperforms
other models in terms of AUC metrics. This superior performance underlines
TGN’s potential as an effective tool for detecting financial fraud, showcasing
its ability to adapt to the dynamic and complex nature of modern financial
systems. We also experimented with various graph embedding modules within the
TGN framework and compared the effectiveness of each module. In conclusion, we
demonstrated that, even with variations within TGN, it is possible to achieve
good performance in the anomaly detection task.

## Introduction

Detecting financial fraud is crucial in the evolving landscape of the
financial sector, especially as transactions become more digitized and
complex, particularly with the rise of fintech. The financial industry’s high
reliance on trust makes fraud detection vital for upholding system integrity
and preventing significant financial losses to individuals, businesses, and
financial institutions. Traditional methods like manual review, rule-based
systems, and auditing prove insufficient in addressing the diverse and
advanced methods employed in financial fraud (West and Bhattacharya 2016).
Consequently, there’s a notable shift towards leveraging AI and big data for
fraud detection. (Ngai et al. 2011)

The landscape of financial transactions has been profoundly transformed by the
rise of fintech, with a significant majority now being conducted via mobile
devices (Kim et al. 2016). This shift has not only increased the volume and
velocity of financial activities but also introduced complex challenges in
real-time fraud detection. In this context, real-time monitoring systems,
particularly those leveraging interaction graph databases, have become pivotal
in modeling the intricate relationships between entities such as users,
accounts, and transactions (Altman et al. 2023).

Figure 1: Visualization of temporally evolving edges in financial networks

Graph analysis, and more specifically Graph Anomaly Detection (GAD), plays a
crucial role in identifying irregular patterns or connections within these
networks (Akoglu, Tong, and Koutra 2015). The continuous evolution of
technology further amplifies the need for advanced research in financial fraud
detection. Effective fraud detection systems are essential not only for
mitigating financial losses but also for preserving the foundational trust
that sustains the financial system.

Consider a scenario where a customer applies for a loan, and over time, the
guarantor information linked to this customer changes. This situation,
representative of the dynamic nature of financial networks, can be visualized
as evolving connections in Figure 1. In such dynamic environments, it is
imperative for financial institutions to rapidly and accurately distinguish
between fraudulent and legitimate activities. Our paper addresses this need by
exploring the application of Temporal Graph Network (TGN) (Rossi et al. 2020)
in enhancing the efficacy of GAD in financial networks.

Figure 2: TGN architecture for graph anomaly detection in finance.

TGN has emerged as a promising model capable of capturing these dynamic
changes in nodes and edges effectively. TGN can learn from a graph that
evolves over time, making it potentially suitable for applications like real-
time fraud detection in financial transactions. However, the application of
TGN for GAD in the finance domain is still not well-established. Addressing
this gap, our research presents a comprehensive framework for utilizing TGN in
the financial domain, specifically for anomaly detection. We experimented with
various models within this framework, comparing their performance against
traditional static GNN models and hypergraph neural network baselines. Our
findings reveal that TGN, with its ability to learn from dynamic edges, shows
remarkable performance, demonstrating its potential as a powerful tool in
financial fraud detection. This research not only advances the field of GAD in
finance but also opens new avenues for the application of dynamic graph models
in real-world scenarios.

## Preliminaries

### Problem Definition

A graph anomaly detection can be described as node classification task. From
the financial network, we define a user node set
$U=\\{u_{1},u_{2},...u_{|U|}\\}$. The core challenge lies in discerning
whether a node $u$ is ‘normal’ user or ‘fraudster’ user. This is represented
by binary classification using node label $y_{u}$, where $y_{u}=0$ for normal
users and $y_{u}=1$ for fraudster users. Ultimately, the goal is to accurately
predict the state of $u$, enabling effective detection of anomalies of nodes.

### Continuous Time Dynamic Graph

We construct a dynamic graph with user interactions, changing its structure
over continuous time. We define our continuous-time dynamic graph as
$\mathcal{G}(T)=(U,\mathcal{E}_{T})$ where $\mathcal{E}_{T}$ denotes the
temporal set of edges. Each edge in $\mathcal{E}_{T}$ is characterized by a
tuple $e=(u_{s},u_{d},t)$, consisting of a source node $n_{s}$, an destination
node $n_{d}$, and a timestamp $t$.

## Methodology

### Temporal Graph Networks

In our discussion, nodes are indexed by $i$, and neighboring nodes are indexed
by $j$. The neighborhood set of node $i$, denoted as $n_{i}^{k}(t)$, refers to
the $k$-hop temporal neighbors connected up to time $t$. We follow notations
used in (Rossi et al. 2020).

We learn node embeddings from our dynamic graph, which are later used for a
downstream task, node classification. We build our model within the TGN
framework excluding the memory module. This was due to the very large size of
the data used in our experiments, leading to out-of-memory errors. In graph
embedding module, temporal embeddings for a dynamic graph are generated,
specifically creating embeddings for each node at time step $t$. We build
various functions $f$ to effectively learn the connectivity between nodes. A
node embedding can be represented as:

$\mathbf{z}_{i}(t)=\sum_{j\in
n_{i}^{k}(t)}f\left(\mathbf{v}_{i}(t),\mathbf{v}_{j}(t),\mathbf{e}_{ij}\right)$
(1)

where $\mathbf{v}_{i}(t)$ and $\mathbf{v}_{j}(t)$ represent the node features,
and $\mathbf{e}_{ij}$ is the edge feature.

####  Temporal Graph Attention (attn)

With function $f$ of attn, a node embedding $\mathbf{h}$ is initialized with
the node’s features as $\mathbf{h}_{i}^{(0)}(t)=\mathbf{v}_{i}(t)$. Then, it
learns the relationships between target nodes and neighboring nodes based on
attention as it passes through $l$ layers.

$\mathbf{h}_{i}^{(l)}(t)=\text{MLP}^{(l)}\left(\mathbf{h}_{i}^{(l-1)}(t)\|\tilde{\mathbf{h}}_{i}^{(l)}(t)\right)$
(2)

$\tilde{\mathbf{h}}_{i}^{(l)}(t)=\text{ MultiHeadAttention
}{}^{(l)}\left(\mathbf{Q}^{(l)}(t),\mathbf{K}^{(l)}(t),\mathbf{V}^{(l)}(t)\right)$

(3)

where the the query $\mathbf{Q}$, key $\mathbf{K}$, and value $\mathbf{V}$ are
defined as in TGAT (Veličković et al. 2017).

####  Temporal Graph Sum (sum)

With function $f$ of sum, a simpler aggregation over the graph is conducted.
That is,

$\mathbf{h}_{i}^{(l)}(t)=\mathbf{W}_{2}^{(l)}\left(\mathbf{h}_{i}^{(l-1)}(t)\|\tilde{\mathbf{h}}_{i}^{(l)}(t)\right)$
(4)

$\tilde{\mathbf{h}}_{i}^{(l)}(t)=\text{ReLU}\left(\sum_{j\in
n_{i}([0,t])}\mathbf{W}_{1}^{(l)}\left(\mathbf{h}_{j}^{(l-1)}(t)\left\|\mathbf{e}_{ij}\right\|\boldsymbol{\phi}\left(t-t_{j}\right)\right)\right)$

(5)

where $\boldsymbol{\phi}$ is a time encoding (Veličković et al. 2017).

In addition to the two graph embedding methods proposed in (Rossi et al.
2020), we further experimented with two additional methods to explore the
effectiveness of various graph embedding techniques.

####  Temporal Graph Mean (mean)

Similar to sum, we can take mean of neighbor embeddings when aggregating
information.

$\tilde{\mathbf{h}}_{i}^{(l)}(t)=\text{ReLU}\left(\frac{1}{|n_{i}([0,t])|}\sum_{j\in
n_{i}([0,t])}\mathbf{W}_{1}^{(l)}\left(\mathbf{h}_{j}^{(l-1)}(t)\left\|\mathbf{e}_{ij}\right\|\boldsymbol{\phi}\left(t-t_{j}\right)\right)\right)$

(6)

where $|n_{i}([0,t])|$ is the number of neightbor of $i$.

####  Temporal Graph Convolution (conv)

Alternatively, we can distill contribution of neighbor embeddings by
introduting a linear layer.

$\tilde{\mathbf{h}}_{i}^{(l)}(t)=\text{ReLU}\left(\mathbf{W}_{2}^{(l)}\sum_{j\in
n_{i}([0,t])}\mathbf{W}_{1}^{(l)}\left(\mathbf{h}_{j}^{(l-1)}(t)\left\|\mathbf{e}_{ij}\right\|\boldsymbol{\phi}\left(t-t_{j}\right)\right)\right)$

(7)

For every graph embedding modules, the final node embedding is obtained from
the last layer: $\mathbf{z}_{i}(t)=\mathbf{h}_{i}^{(L)}(t)$.

### Pre-training and Downstream task

To conduct graph anomaly detection, we initially learn node embeddings through
pre-training with the edge prediction task using TGN as an encoder.
Subsequently, we train a separate MLP as a decoder for node classification as
a downstream task by inputting the embeddings obtained from the encoder.

#### Pre-training

In this phase, we focus on a task that involves predicting connections between
nodes. We posit that if a node $u$ is connected to node $i$ at time $t$, this
scenario constitutes a positive pair. Conversely, a negative pair is formed by
randomly sampling a node from the entire set of nodes. We denote such a
negatively sampled node as $k$. To refine the model’s ability to differentiate
between these pairs, we utilize the Binary Cross-Entropy Loss (BCELoss): The
objective is to increase the probability of an edge existing in positive pairs
while decreasing it for negative pairs. Mathematically, this can be
represented as follows:

$\mathbf{L}_{1}=-\sum_{(u,i,k,t)\in
D}[\log(z_{u}(t)^{T}z_{i}(t))+\log(1-z_{u}(t)^{T}z_{k}(t))]$

(8)

where $D$ denotes the edge set, which is derived from $\mathcal{E}_{T}$ with
additional negative sampling, $u$ is the source node, $i$ is positive
destination node, $k$ is negative destination node, and $t$ represents time.

#### Downstream task

In downstream phase, the objective shifts to predicting the class of each
node. The predicted class for node $u$ is expressed as:

$y_{u}^{\text{pred}}=\text{MLP}(\mathbf{z}_{u})$ (9)

where $\mathbf{z}_{u}$ is the node embedding at its final time. For the loss
function in this phase, we employed the BCE loss:

$\mathbf{L}_{2}=-\sum_{|U|}[y_{u}^{true}\log(y_{u}^{\text{pred}})+(1-y_{u}^{true})\log(1-y_{u}^{\text{pred}})]$
(10)

where $y_{u}^{true}$ represents the true class of the nodes $u$.

## Experiment

### Experimental Settings

#### Dataset

In our study, we used the DGraph dataset (Huang et al. 2022), designed for
fraud detection in real-world financial scenarios. Provided by Finvolution
Group, a fintech platform, DGraph is built on interactions where the users
fill in emergency contact of other users. It consists of 3,700,550 user nodes
and 4,300,999 directed edges, making it a challenging GAD dataset with the
extreme imbalanced classes.

#### Baseline

In our research, we employed a suite of baseline methods that are renowned in
the graph neural network (GNN) domain, specifically chosen for their
capability to handle the intricate structures and features of the DGraph
dataset. Our baselines comprised MLP for its foundational neural network
approach; Graph Convolutional Network (GCN) (Kipf and Welling 2016), which
excels in aggregating neighbor information within graph structures; GraphSAGE
(Hamilton, Ying, and Leskovec 2017), known for generating node embeddings by
sampling and aggregating from a node’s local neighborhood.

In addition to the aforementioned models, our suite of baseline methods
encompasses the hypergraph neural networks. A hypergraph, defined by nodes and
hyperedges, represents each hyperedge as a set of nodes, allowing it to
capture higher-order relationships rather than pairwise relationships. We
construct hyperedges by grouping destination nodes from each source node. In
this setting, we implemented four distinct hypergraph neural networks.
Hypergraph Neural Networks (HGNN) (Feng et al. 2019) applies graph
convolutions by transforming a given hypergraph into an ordinary graph.
Hypergraph Networks with Hyperedge Neurons (HNHN) (Dong, Sawin, and Bengio
2020) first updates hyperedge embeddings using their including nodes and then
updates node embeddings using their incident hyperedges. Hypergraph
Convolution and Hypergraph Attention (HCHA) (Bai, Zhang, and Torr 2021)
utilizes parameters for the weights to the incident node or hyperedge
embeddings during aggregation. AllSetTransformer (Chien et al. 2021) comprises
two multiset functions with SetTransformer(Lee et al. 2019) for aggregating
node or hyperedge embeddings.

#### Hyperparameter

We conducted experiments with TGN models using the following configurations: a
node embedding dimension of 128, a batch size of 200, and a learning rate of
0.0001 for pre-training. For the downstream task, we used a batch size of 100,
and a learning rate of 0.0003. For the graph baseline models, we consistently
used a batch size of 1024. For hypergraph baselines, we performed a grid
search within the following ranges: learning rate in 0.01, 0.02, 0.03, 0.04,
0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.2, 0.3, 0.4, 0.5, and 0.6, hidden
dimension in 16, 32, 64, and 128, number of layer in 1 and 2. All models were
trained with 10 epochs.

#### Metrics and Evaluation

Our research employed one primary metric for model evaluation: Area Under the
Curve (AUC). AUC is a widely recognized measure for the capability of a model
to distinguish between different classes, in our case, between normal and
fraudsters users. This metric was chosen for its robustness and widespread
acceptance in evaluating the performance of classification models.

### Results and Discussion

Model | Valid AUC | Test AUC
---|---|---
MLP | 0.6507 | 0.6465
GCN | 0.6087 | 0.6135
GraphSAGE | 0.6344 | 0.6344
AllSetTransformer | 0.6779 | 0.6829
HNHN | 0.6775 | 0.6845
HGNN | 0.6429 | 0.6450
HCHA | 0.6289 | 0.6318
TGN(Sum) | 0.7517 | 0.7640
TGN(Mean) | 0.7569 | 0.7747
TGN(GCN) | 0.7581 | 0.7716
TGN(GAT) | 0.7569 | 0.7691
improv. | 11.83% | 13.18%
Table 1: AUC comparison between the models: the best-performing TGN model is
highlighted in bold, and the top baseline model is underlined.

As shown in Table 1, TGNs exhibit significantly better performance, confirming
their effectiveness in aggregating information from a node’s neighbor history.
This enhanced ability contributes to superior sensitivity and specificity,
particularly in identifying patterns indicative of anomalous behavior,
compared to baseline models.

Furthermore, we investigate various graph embedding modules within the TGN
framework. While variants exist within TGN, the results consistently affirm
the model’s ability to achieve remarkable performance in the anomaly detection
task. This contributes to a deeper understanding of the factors influencing
TGNs’ effectiveness in handling the complexity of emergency contact
interactions.

Delving deeper into the baseline comparisons, it’s noteworthy that MLPs
exhibited better performance than static graph models like GCN and GraphSAGE.
This outcome suggests that approaches which view interactions in an
independent and static manner may not be as effective. The dynamic and
interconnected nature of financial transactions poses a challenge to these
static models, highlighting the need for more adaptive and dynamic approaches.

Additionally, the hypergraph baselines have demonstrated remarkable
performance, offering a key insight: viewing interactions dependently is
crucial in anomaly detection. This finding emphasizes the importance of
considering the complex interdependencies inherent in financial transactions,
which hypergraphs capture more effectively than traditional graph models.

## Conclusion

In conclusion, the experimental results solidify the position of TGNs as a
robust and promising approach for financial anomaly detection. The superior
performance over baseline models, coupled with the adaptability demonstrated
in capturing dynamic patterns, positions TGNs as a valuable asset in the
arsenal against financial fraud. These findings contribute not only to the
academic discourse on anomaly detection but also hold practical implications
for enhancing fraud detection measures in fintech applications.

Interestingly, the hypergraph baselines also exhibited commendable
performance, suggesting its potential utility in anomaly detection tasks.
Therefore, as a future work, we propose experimenting with a hybrid model that
combines the strengths of TGN and hypergraph. This combination could
potentially harness the temporal sensitivity of TGN and the intricate node
relationship mapping of hypergraphs, offering a more holistic approach to
anomaly detection.

## Acknowledgments

This work was supported by Institute of Information & communications
Technology Planning & Evaluation (IITP) grant funded by the Korea
government(MSIT) (RS-2022-00143911, AI Excellence Global Innovative Leader
Education Program)

## References

  * Akoglu, Tong, and Koutra (2015) Akoglu, L.; Tong, H.; and Koutra, D. 2015.  Graph based anomaly detection and description: a survey.  _Data mining and knowledge discovery_ , 29: 626–688.
  * Altman et al. (2023) Altman, E.; Egressy, B.; Blanuša, J.; and Atasu, K. 2023.  Realistic Synthetic Financial Transactions for Anti-Money Laundering Models.  _arXiv preprint arXiv:2306.16424_.
  * Bai, Zhang, and Torr (2021) Bai, S.; Zhang, F.; and Torr, P. H. 2021.  Hypergraph convolution and hypergraph attention.  _Pattern Recognition_ , 110: 107637.
  * Chien et al. (2021) Chien, E.; Pan, C.; Peng, J.; and Milenkovic, O. 2021.  You are allset: A multiset function framework for hypergraph neural networks.  _arXiv preprint arXiv:2106.13264_.
  * Dong, Sawin, and Bengio (2020) Dong, Y.; Sawin, W.; and Bengio, Y. 2020.  Hnhn: Hypergraph networks with hyperedge neurons.  _arXiv preprint arXiv:2006.12278_.
  * Feng et al. (2019) Feng, Y.; You, H.; Zhang, Z.; Ji, R.; and Gao, Y. 2019.  Hypergraph neural networks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, 3558–3565.
  * Hamilton, Ying, and Leskovec (2017) Hamilton, W.; Ying, Z.; and Leskovec, J. 2017.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30.
  * Huang et al. (2022) Huang, X.; Yang, Y.; Wang, Y.; Wang, C.; Zhang, Z.; Xu, J.; Chen, L.; and Vazirgiannis, M. 2022.  Dgraph: A large-scale financial dataset for graph anomaly detection.  _Advances in Neural Information Processing Systems_ , 35: 22765–22777.
  * Kim et al. (2016) Kim, Y.; Choi, J.; Park, Y.-J.; and Yeon, J. 2016.  The adoption of mobile payment services for “Fintech”.  _International Journal of Applied Engineering Research_ , 11(2): 1058–1061.
  * Kipf and Welling (2016) Kipf, T. N.; and Welling, M. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_.
  * Lee et al. (2019) Lee, J.; Lee, Y.; Kim, J.; Kosiorek, A.; Choi, S.; and Teh, Y. W. 2019.  Set transformer: A framework for attention-based permutation-invariant neural networks.  In _International conference on machine learning_ , 3744–3753. PMLR.
  * Ngai et al. (2011) Ngai, E. W.; Hu, Y.; Wong, Y. H.; Chen, Y.; and Sun, X. 2011.  The application of data mining techniques in financial fraud detection: A classification framework and an academic review of literature.  _Decision support systems_ , 50(3): 559–569.
  * Rossi et al. (2020) Rossi, E.; Chamberlain, B.; Frasca, F.; Eynard, D.; Monti, F.; and Bronstein, M. 2020.  Temporal Graph Networks for Deep Learning on Dynamic Graphs.  arXiv:2006.10637.
  * Veličković et al. (2017) Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; and Bengio, Y. 2017.  Graph attention networks.  _arXiv preprint arXiv:1710.10903_.
  * West and Bhattacharya (2016) West, J.; and Bhattacharya, M. 2016.  Intelligent financial fraud detection: a comprehensive review.  _Computers & security_, 57: 47–66.


# MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

Tao Gong1∗ Chengqi Lyu1∗ Shilong Zhang2,1∗ Yudong Wang1,3∗ Miao Zheng1∗ Qian
Zhao1∗ Kuikun Liu1∗ Wenwei Zhang1∗ Ping Luo2,1 Kai
Chen${}^{1\textrm{\Letter}}$
∗equal contribution, in random order
1Shanghai AI Laboratory 2The University of Hong Kong
3School of Electrical and Information Engineering, Tianjin University
$\left\\{\texttt{gongtao, lvchengqi, zhangshilong, wangyudong,
<EMAIL_ADDRESS>
$\left\\{\texttt{zhaoqian, liukuikun, zhangwenwei,
<EMAIL_ADDRESS>
###### Abstract

We present a vision and language model named MultiModal-GPT to conduct multi-
round dialogue with humans. MultiModal-GPT is capable of following diverse
instructions, such as generating detailed captions, counting specific objects,
and addressing general inquiries posed by users. The model is efficiently
fine-tuned from OpenFlamingo, with Low-rank Adapter (LoRA) incorporated in
both the gated-cross-attention and self-attention components of the language
model. Our approach involves constructing instruction templates that
incorporate vision and language data for multi-modality instruction tuning,
enabling the model to comprehend and adhere to human directives. We observe
that the quality of training data is crucial for effective dialogue
performance, as a limited dataset with short responses may cause the model to
generate brief replies to any instruction. To further enhance MultiModal-GPT’s
conversational abilities, we employ language-only instruction-following data
for joint training alongside visual-language instructions. Utilizing the
_same_ instruction template for both types of data results in a significant
improvement in dialogue performance. Our experiments demonstrate MultiModal-
GPT’s proficiency in maintaining continuous dialogues with humans. The code,
dataset, and demo can be found at https://github.com/open-mmlab/Multimodal-
GPT.

##  1 Introduction

Humans interact with the world through multiple channels, including vision and
language, each of which has a unique advantage in representing and conveying
certain concepts of the world, thus contributing to a better understanding of
the world. A central objective of artificial intelligence research is to
create a versatile assistant capable of effectively following multimodal
vision-and-language instructions that align with human intentions, in order to
accomplish a diverse array of real-world tasks.

Recently, GPT-4 [11] has demonstrated remarkable proficiency in multi-modal
dialogues with humans. Although GPT-4’s [11] exceptional capabilities have
been observed, the mechanisms underpinning its outstanding performance remain
elusive. Studies such as Mini-GPT4 [18] and LLaVA [8] have sought to replicate
this performance by aligning visual representations with the input space of
LLM, subsequently utilizing the original self-attention in the LLM to process
visual information. However, incorporating such models with detailed or
spatiotemporal visual information can be computationally intensive due to the
potentially large number of image tokens. Furthermore, both models employ
vicuna [2], an open-source chatbot refined through fine-tuning LLaMA [16] on
user-generated conversations from ChatGPT, which omits the language
instruction tuning phase in their research.

To address these challenges, we build upon the open-source Flamingo framework
[1], a multimodal pre-trained model that deploys a perceiver resampler to
efficiently extract visual information from the vision encoder, while also
employing gated cross-attention layers for image-text interactions. This model
has been pre-trained on an extensive dataset of image-text pairs, showcasing
robust few-shot visual comprehension capabilities. Nevertheless, it lacks the
capacity to engage in zero-shot multi-turn image-text dialogues. As a result,
our goal is to fine-tune OpenFlamingo using comprehensive datasets of image
and text instructions, enabling the model to conduct conversations that more
closely align with human preferences. By capitalizing on OpenFlamingo’s
foundational strengths, we aspire to narrow the performance gap between the
model’s existing capabilities and the desired outcome of more accurate, human-
like interactions in multimodal dialogues. We have dubbed our multimodal
chatbot MultiModal-GPT.

We also use a unified instruction template for both language and visual
instruction data during model training. We first construct instruction
templates with vision and language data to train the MultiModal-GPT. We find
the training data is vital with respect to the performance of the MultiModal-
GPT. Some datasets, such as VQA v2.0 [3], OKVQA [9], GQA [5], CLEVR [6] and
NLVR [15] datasets, will degrade the dialogue performance of the MultiModal-
GPT, since the response in these datasets is restricted to one or two words
(e.g., yes/no). Consequently, when these datasets are incorporated into the
training process, the model exhibits a tendency to generate answers comprising
merely one or two words. This brevity is not conducive to user-friendliness.

To further enhance the ability of MultiModal-GPT to chat with people, we also
collect language data and define a unified instruction template to jointly
train the MultiModal-GPT. The joint training of language-only instructions and
visual and language instructions effectively improves the performance of the
model. We show various demos to show the ability of continuous dialogue of
MultiModal-GPT with humans.

##  2 Unified Instruction Template

We propose a unified template for the integration of unimodal linguistic data
and multimodal vision-and-language data, with the objective of effectively
training the MultiModal-GPT model in a synergistic manner. This unified
approach aims to enhance the model’s performance across diverse tasks by
leveraging the complementary strengths of both data modalities and fostering a
more profound understanding of the underlying concepts.

###  2.1 Language-only Instruction Template

<BOS> Below is an instruction that describes a task. Write a response that
appropriately completes the request ### Instruction: {instruction} ### Input:
{input} ### Response: {response} <EOS> Table 1: The input sequence of language
data used to train the model. The {instruction}, {input} and {response} are
texts from the source data. Only the {response} part and <EOS> token will be
calculated loss.

We employ the Dolly 15k and Alpaca GPT4 datasets [12] as resources for
assessing language-only instruction-following capabilities. These datasets
have been specifically designed to improve the performance of language models
in executing instruction-based tasks. To ensure consistent instruction-
following format, we utilize the prompt template presented in Table 1 for
structuring the dataset input.

###  2.2 Vision and Language Instruction Template

<BOS> Below is an instruction that describes a task. Write a response that
appropriately completes the request ### Image: <image_token> ### Instruction:
{question} ### Response: {response}<EOS> ### Instruction: {question} ###
Response: {response} <EOS> Table 2: The input sequence of vision and language
data used to train the model. The {question} and {response} are texts from the
source data. <image_token> is a token denoting the existence of image. Note
that there are multi-round dialogues if the dataset has. Only the {response}
part and <EOS> token will be calculated loss.

We utilize a diverse selection of vision and language instruction-following
datasets in our study, including LLaVA [8], Mini-GPT4 [18], A-OKVQA [14], COCO
Caption [7], and OCR VQA [10]. These datasets encompass a wide array of
applications and domains, thereby facilitating the comprehensive evolving of
our model’s performance.

In order to present the text in a consistent, instruction-following format, we
adopt the prompt delineated in Table 2 as a template for structuring these
datasets. By adhering to a standardized format, we ensure that our model is
better equipped to process the information and respond accordingly.

It is important to note that the COCO Caption dataset generally does not
include instructional content, as it predominantly consists of descriptive
captions. To overcome this limitation and incorporate instructional data, we
employ the GPT-4 [11] model to generate pertinent instructions for the COCO
Caption dataset. This integration of synthesized instructions enriches the
dataset, enabling our model to achieve a more robust capabilities in
processing and responding to human instructions.

Table 3 showcases a variety of examples illustrating the instructions
generated for the COCO Caption dataset, demonstrating the effectiveness of our
approach in adapting the dataset to better suit our research objectives.

• Can you describe the image? • Could you provide a description of the image?
• What do you see in this image? • Share your thoughts on the content of the
image. • Please narrate what’s happening in the picture. • Can you give a
brief explanation of the image? • Describe the main elements and details
present in the image. • In your own words, what is depicted in the image? •
How would you describe the image’s content in a caption? • Can you suggest an
insightful caption that highlights the underlying message of the image? Table
3: The list of instructions for image caption.

##  3 Method

###  3.1 Architecture

The proposed MultiModal-GPT is based on the open-flamingo model [1]. As shown
in Figure 1, MultiModal-GPT consists of a vision encoder from CLIP [13], a
perceiver resampler to receive the spatial features from the vision encoder,
and a language decoder LLaMA [16]. Note that the language decoder is
conditioned on the spatial features from the perceiver resampler by cross-
attention in order to encode the feature of vision into text. Please refer to
[1] for more details of the model architecture.

Figure 1: The overall framework of MultiModal-GPT. MultiModal-GPT consists of
a vision encoder, a perceiver resampler to receive the spatial features from
the vision encoder, and a language decoder which is conditioned on the spatial
features from the perceiver resampler by cross-attention in order to encode
the feature of vision into text. We freeze the whole open-flamingo model and
add LoRA to the self-attention part, the cross-attention part, and the FFN
part in the language decoder to finetune MultiModal-GPT.

###  3.2 Joint Training

We use both language-only instruction-following data and vision and language
instruction-following data to train the MultiModal-GPT jointly. As shown in
Fig.1, We freeze the whole open-flamingo model and add LoRA [4] to the self-
attention, cross-attention, and FFW part in the language decoder to finetune
MultiModal-GPT. The MultiModal-GPT is trained by predicting the next token of
the text, and only the {response} and <EOS> tokens in the input sequence are
involved in the loss calculation.

##  4 Experiments

###  4.1 Implementation Details

We jointly train the MultiModal-GPT model using a comprehensive mix of
language data and vision and language data sources to enhance its performance.
The language datasets include Dolly 15k and Alpaca GPT4 [12], while the vision
and language datasets encompass LLaVA [8], Mini-GPT4 [18], A-OKVQA [14], COCO
Caption [7], and OCR VQA [10]. Other vision and language instruction datasets,
such as MultiInstruct [17] can also be explored, and we left it as future
work. This combination of datasets aims to provide a diverse and rich training
environment for the MultiModal-GPT model.

To effectively train the model, we incorporate the entire text corpus from the
Dolly 15k and Alpaca GPT4 datasets. Similarly, we include all image-text pairs
available from the LLaVA and Mini-GPT4 datasets to ensure adequate exposure to
various contexts and situations. However, the quality of the A-OKVQA, COCO
Caption, and OCR VQA datasets is considered inferior compared to LLaVA and
Mini-GPT4. To account for this disparity while still benefiting from the
additional data, we include a random sample of 5000 image-text pairs from the
A-OKVQA dataset and 512 image-text pairs each from the COCO Caption and OCR
VQA datasets in the training process.

To train the model, we utilize 8 A100 GPUs and complete the training process
within a single epoch. The batch size per GPU for both vision and language
instruction following data and language-only instruction following data is set
to 1. We use gradient accumulation, and update the parameters of the LoRA
every 16 iterations. Each iteration encompasses one vision-language pair and
one language-only instruction data. Consequently, the aggregate batch size
amounts to 256. We employ a learning rate of 1e-5, using a cosine learning
rate scheduler to adjust the learning rate during the training process.

###  4.2 The Quality of Data Matters

In order to enhance the diversity of vision and language data, we initially
incorporated a multitude of datasets in our preliminary experiments. However,
we observed that the quality of certain datasets, including VQA v2.0 [3],
OKVQA [9], GQA [5], CLEVR [6], and NLVR [15], was suboptimal. This is
primarily due to the fact that the {response} in these datasets is restricted
to one or two words (e.g., yes/no). Consequently, when these datasets are
incorporated into the training process, the model exhibits a tendency to
generate answers comprising merely one or two words. This brevity is not
conducive to user-friendliness. As a result, we have opted to exclude these
datasets from the final version of our study.

###  4.3 Demos

We present a variety of demos that exhibit the capabilities of MultiModal-GPT
in engaging in conversation with humans. As illustrated in Figure 2,
MultiModal-GPT can successfully identify lasagna and provide a comprehensive
recipe for preparing the dish when prompted by the user in the initial
dialogue. In the subsequent dialogue, MultiModal-GPT offers reasonable
suggestions on where to dine when the user inquires about potential eateries.
This exemplifies the model’s capacity for maintaining an ongoing dialogue.

Additionally, Figure 4 demonstrates MultiModal-GPT’s counting abilities, while
Figure 6 showcases its OCR capabilities. MultiModal-GPT is also adept at
addressing general inquiries regarding users’ travel plans, as depicted in
Figure 7. More demos can be found at https://github.com/open-mmlab/Multimodal-
GPT.

Figure 2: The MultiModal-GPT can give a recipe to bake lasagna, and tell users
where to eat it. Figure 3: The MultiModal-GPT can recognize the image of Elon
Musk, and answer questions about Elon Musk. Figure 4: The MultiModal-GPT knows
there are 4 women in the image, and knows what they are doing. Figure 5: The
MultiModal-GPT knows the film and knows which studio made the film. Figure 6:
The MultiModal-GPT can recognize the characters in the image and knows the
author of the book. Figure 7: The MultiModal-GPT can answer general questions
about the traveling of users. Figure 8: The MultiModal-GPT can generate a
detailed description for the image and has the ability to reason the season in
the image.

## References

  * [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  * [2] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [3] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * [4] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [5] Drew A Hudson and Christopher D Manning.  Gqa: A new dataset for real-world visual reasoning and compositional question answering.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019.
  * [6] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  * [7] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  * [8] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [9] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019.
  * [10] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.  Ocr-vqa: Visual question answering by reading text in images.  In ICDAR, 2019.
  * [11] OpenAI.  Gpt-4 technical report.  2023\.
  * [12] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4.  arXiv preprint arXiv:2304.03277, 2023.
  * [13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [14] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi.  A-okvqa: A benchmark for visual question answering using world knowledge.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VIII, pages 146–162. Springer, 2022.
  * [15] Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi.  A corpus of natural language for visual reasoning.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217–223, 2017.
  * [16] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [17] Zhiyang Xu, Ying Shen, and Lifu Huang.  Multiinstruct: Improving multi-modal zero-shot learning via instruction tuning.  arXiv preprint arXiv:2212.10773, 2022.
  * [18] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.


# A new experiment to detect dark matter in sub-MeV range using semiconductor
superlattice superstructures (SSS)

Kalpana Bora<EMAIL_ADDRESS>Department of Physics, Gauhati University,
Guwahati-781014, Assam, India

###### Abstract

About $26\%$ of the matter in our Universe is made up of Dark Matter (DM),
which interacts with Standard Model (SM) matter only through gravitational or
weak interactions. Many proposals have been made by scientists about the
possible candidates of DM - WIMPs, axions, ALPs, black holes etc. And range of
its mass could be extremely broad - from Planck scale to as light as
$10^{-22}$ eV. Experiments to detect DM are extremely challenging, as DM does
not exhibit appreciable interactions with ordinary matter. May be due to such
elusive nature, so far it has not been possible to detect DM, though many
experiments are going on worldwide to do so. With about 40 orders of magnitude
variation in their mass, it is possible that their gravitational interaction
too is very weak, and many creative proposals have been made to detect
possible DM candidates, with vast variation of techniques and target
materials. In this work, we propose a new experiment to detect sub-MeV range
DM particles, using the semiconductor superlattice superstructure (SSS) as the
target material. Such materials have band gap of the order of few hundreds of
milli eV, and are suitable for detecting sub-MeV range particles scattering
off electrons. The photons emitted as a result of excitation of SSS lie in the
micrometer range and may be detected via quantum cascade lasers (QCL).

keywords : Dark matter, Semiconductor Superlattice Supserstructures (SSS),
micrometer photons, intra-sub-band transitions, quantum cascade lasers (QCL).

In spite of the fact that about $26\%$ of the matter in our Universe is made
up of dark matter (DM), it has not been possible so far to detect it, may be
due to its elusive nature. It is believed to interact with ordinary matter
only through gravitational and weak interactions. Various proposals have been
made by researchers for the possible dark matter candidates, that range from
axions/axion like particles (ALPs) to black holes. Accordingly, mass of dark
matter candidates may range from $\sim 10^{-22}$ eV to the Planck scale. Many
experiments have been performed/planned by scientists worldwide, that include
both direct detection (DD) and indirect detection experiments and production
of DM candidates in accelerators. In DD experiments, the DM particle scatters
off the nucleus or electrons in the atom. For a recent review on DM searches,
please refer to Billard2021 -Heros2020  (and references therein).

However, there have been stringent exclusion limits and null results from most
of current DM detection experiments in the TeV range DM candidates Kahn2021 .
Thermal DM can have mass range (keV-100 TeV), with WIMPs (weakly interacting
massive particles, produced thermally via freeze-out) lying in range (1
GeV-100 TeV). A DD experiment looks for kinetic energy deposited in nucleus of
the target in DM scattering off the target nuclei. If $\rho_{\Phi}$ is the
density of DM, $m_{\Phi}$ is its mass and $n_{\Phi}$ its number density, then,
since the DM penetrates through our galaxy, then Kahn2021 ,

$\rho_{\Phi}=m_{\Phi}\times n_{\Phi}\sim 0.3-0.5\hskip
2.84544ptGeV/cm^{3},\hskip 2.84544ptv_{\Phi}\sim 10^{-3}\textrm{.}$ (1)

The DM-nucleus scattering cross-section $\sigma_{\Phi
N}=A^{2}\alpha_{W}^{2}/m_{\Phi}^{2}$, where $A$ is the atomic mass number of
the target material, $\alpha_{W}\sim 0.03$ is the coupling constant. Velocity
of bound DM in our galaxy is limited by the galactic escape velocity $\sim
10^{-3}c$. Hence, for a heavy nucleus of $A=100$, and for a DM mass
$m_{\Phi}=100$ GeV, scattering rate per nucleus $R_{\Phi}$ turns out to be,

$R_{\Phi}=n_{\Phi}\sigma_{\Phi N}v_{\Phi}\textrm{,}$ (2)

and can be computed to be $\sim 3\times 10^{-26}/s$. Therefore, huge detectors
are needed to have significant number of events per year. As the DM is non-
relativistic, for DM mass $m_{\Phi}=100$ GeV, its momentum $p_{\Phi}v_{\Phi}$
and kinetic energy $m_{\Phi}v_{\Phi}^{2}/2$ are such that this DM-nucleus
scattering is elastic. The WIMP DM below GeV is unaccessible to current
experimental searches, but presents opportunities for new experiments. The
detection of sub-GeV DM is very challenging because of insufficient signal due
to less recoil energy. For DM mass $\sim$ few MeV, the nuclear recoil energy
lies in the range $\sim$ eV, that is of the order of band gaps of
semiconductors. Also, as DM mass $m_{\Phi}$ decreases, its number density
increases, and hence smaller size detectors can also provide sufficient event
rate for DD experiments. WIMPs with very low mass, of the order of $\leq$ MeV
cannot produce sufficient nuclear recoil (NR) energy, which lies below the
threshold of most experiments, and hence coupling of DM to electrons (electron
recoil, ER) is used to detect such light DM particles. Condensed matter
systems are also used to detect light DM Kahn2021 , as bridging atoms together
lowers the excitation energy, and condensed matter have large densities.

Above discussion forms the motivation for this work, in which we have proposed
a new idea for detecting sub-MeV range DM, using ER in SSS materials. The band
gap of such materials is few hundreds of meV (milli eV), which matches with
electronic recoil energy for scattering by sub-MeV DM. When the SSS target
after excitation by DM comes to ground state, the photons emitted are in
micrometer range, which can be detected by quantum cascade lasers.

The nuclear recoil energy in DM-nucleus scattering (elastic) can be expressed
as

$E_{NR}=\frac{q^{2}}{2m_{N}}\leq\frac{2\mu_{\Phi
N}^{2}v_{\Phi}^{2}}{m_{N}}\textrm{,}$ (3)

where $\mu_{\Phi N}$ is the reduced mass of the DM-nucleus system, and
$v_{\Phi}$ is the DM velocity which is taken as sum of galactic escape
velocity and earth’s velocity to estimate the maximum nuclear recoil energy.
This recoil energy becomes insufficient and falls below threshold of most
current generation experiments to detect lower mass (sub-MeV) DM. For such DM,
energy is transferred more efficiently to an electron Essig2012 . The electron
recoil energy is

$E_{ER}\leq\frac{\mu_{\Phi}v_{\Phi}^{2}}{2}\leq 3.23\hskip
2.84544pteV(\frac{m_{\Phi}}{MeV})\textrm{,}$ (4)

and for a DM of mass 0.2 MeV, the electronic recoil energy is 255 meV (shown
in Table 1) if DM velocity is taken as $2.54\times 10^{-3}c$ (as sum of
galactic escape velocity and speed of earth, which turns out to be $\sim 764$
km/sec Battaglieri2017 ). This recoil energy can promote bound electrons from
valence bands to higher bands if the band gap is of the order of $\sim$ 255
meV, and when they return back to their ground state, micrometer (terra hertz)
photons are produced,

$\lambda=hc/E=12.3\hskip 2.84544pt\textrm{micrometer (25 THz), for E=100
meV.}$ (5)

Next, a mechanism is needed to detect such micrometer photons, because in the
scattering experiment, detection of such photons will act as the signal
(signature if a DM has reached the detector). From this discussion, we observe
that for DM of typical mass $\sim$ 0.2 MeV, the recoil energy for DM-electron
scattering is of the order of 250 meV, and hence we need a target with such
band gaps. In this work, we propose a novel experiment, where SSS can be used
for the purpose, as they have the desired band gap of hundreds of mill eV.

DM Mass Scattering by Xenon nucleus Scattering by electron 0.2 MeV 0.00196 meV
255 meV 0.5 MeV 0.00123 meV 806 meV

Table 1: Recoil energy for DM scattering by Xenon and electron

We give here an overview about SSS for the purpose of completeness of the work
Smith1990 ; Parker2005 . Superlattice is a periodic structure of layers of two
or more materials (say AlAs and GaAs layers, also GaInAs SSS), of similar
lattice constant and different band gaps. Typically, width of layers is orders
of magnitude larger than the lattice constant. There are in general two
methods of forming superlattice structure - periodic variation of donor or
acceptor impurities, or periodic variation of alloy composition introduced
during crystal growth. In the first method, due to larger thermal diffusion of
impurities, it is hard to maintain periodic nature of the potential, and hence
the second method is more popular. It should be noted that superlattice is not
same as multiple quantum well, because though they are very similar in their
structure, the barrier width of SSS is small enough (than quantum well), such
that quantum wells are coupled with each other. On the other hand, quantum
wells in multiple quantum well structures are highly localised. The solution
of Schroedinger’s equation using Kronig-Penny model gives discrete energy
bands separated by forbidden bands, just as in usual crystal structure energy
band. Difference is - here, the band gap can be engineered by controlling the
widh of alternating materials, and also by choosing the materials of
alternating layers. Because the thickness of alternating layers is orders of
magnitude larger than the lattice constant, the energy bands form in the first
Brillouin zone of the well, and first Brillouin zone is subdivided into many
mini-bands. These mini-bands are the main characteristic of superlattice
structures. The potential barrier of few nanometer can create a band gap of
few hundred milli eV in typical SSS (like GaInAs). This band gap corresponds
to micrometer region photons of the spectrum, which will be emitted if a sub-
MeV DM candidate is scattered off such SSS target.

Figure 1:  Spatial variation of conduction band edge E(z), with mini bands
$\nu$=a,b (shaded areas) for semiconductor superlattice Wacker2002  Figure 2:
Mini band conduction mechanism for superlattice transport Wacker2002  Figure
3:  Intra sub-band transitions and laser mechanism in QCL

Next challenge is to detect such micrometer photons, and QCL can be used for
the purpose. SSS are used in the making of QCL. In QCL, intra-sub-band
transitions of SSS are used, in place of inter-band transitions used in
traditional lasers. In the intra-sub-band transitions, carriers do not
recombine and disappear, rather they reside in the quantum well and can still
be transported. Electrons stream down a potential staircase and emit photons
at steps. These steps consist of quantum wells, in which control of tunnelling
yields population inversion between discrete conduction band excited states
Faist1994 . Direct evidence of laser action at micrometer wavelength, with
peak power in pulsed operation, is exhibited by a strong narrowing of emission
spectrum above threshold - and this acts as the signal that a micrometer
photon has been detected. Further details about how this experiment can be
realised, is beyond the scope of this work, and can be worked out by
experimentalists of the field.

Figure 4:  Schematic diagram of the proposed experiment

To summarise, we have proposed a new experiment to detect sub-MeV range DM,
using SSS, where the micrometer range photon as signal can be detected using
QCL. The band gap of SSS materials can be engineered by adjusting width of
alternating materials and with change of such materials, such that the DM in a
mass range can be detected (equivalent to detection of photons with a range of
wavelengths). We have just proposed the idea for a futuristic experiment, and
further details/challenges in designing etc. lie beyond the scope of this
paper and can be worked out by the expert experimentalists in the field. Hope
this proposal will open new avenues in the search and detection of DM.

## Acknowledgements

The author would like to thank the support by RUSA and FIST grants of Govt. of
India for upgrading the computer lab of the department where part of this work
was done.

## References

  * (1)

## References

* (2) J. Billard et al, APPEC Committee Report, [arXiv:hep-ex/2104. 07634[hep-ph]].
* (3) Maco Battaglieri et al, White paper on summary of Workshop - ‘US Cosmic Visions’, University of Maryland, March 2017, Report - FERMILAB-CONF-17-282-AE-PPD-T, [arXiv:hep-ph/1707. 04591[hep-ph]].
* (4) Tongyan Lin, TASI Lectures on DM models and direct detection, Published in PoS 333 (2019) 009, [arXiv:hep-ph/1904.07915[hep-ph]].
* (5) Carlos Perez de los Heros, published in Proceedings of Science, EPS-HEP2019 (2020) 694, [arXiv:2001.06193[astro-ph.HE]].
* (6) Yonathan Kahn and Tongyan Lin, [arXiv:2108.03239[hep-ph]].
* (7) R. Essig, J. Mardon, and T. Volansky Phys. Rev. D85, 076007 (2012), [arxiv:1108.5383].
* (8) D.L.Smith and C. Mailhiot, Rev. of Mod. Physics, Vol 62, 173, 1990.
* (9) A. Wacker, Phys. Rep. 357, p 1-111, 2002.
* (10) L. Esaki and L. L. Chang, Phys. Rev. Lett Vol 33, 495, 1975.
* (11) A. C. Tselis and J. J. Quinn, Phys. Rev B 29, 3318, 1984.
* (12) T. J. Parker and S.R.P. Smith, in Encyclopedia of Modern Optics, 2005.
* (13) Jerome Faist et. al., Science, Vol 264, p 553, 1994,


# Revisiting the Fragility of Influence Functions

Jacob R. Epifano, Ravi P. Ramachandran
Department of Electrical and Computer Engineering
Rowan Univsercity
Glassboro NJ, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
&Aaron J. Masino
Department of Biostatistics, Epidemiology, Informatics
University of Pennsylvania Perelman School of Medicine
Philadelphia PA, USA
<EMAIL_ADDRESS>
&Ghulam Rasool
Department of Machine Learning
Moffitt Cancer Center
Tampa FL, USA
<EMAIL_ADDRESS>

###### Abstract

In the last few years, many works have tried to explain the predictions of
deep learning models. Few methods, however, have been proposed to verify the
accuracy or faithfulness of these explanations. Recently, influence functions,
which is a method that approximates the effect that leave-one-out training has
on the loss function, has been shown to be fragile. The proposed reason for
their fragility remains unclear. Although previous work suggests the use of
regularization to increase robustness, this does not hold in all cases. In
this work, we seek to investigate the experiments performed in the prior work
in an effort to understand the underlying mechanisms of influence function
fragility. First, we verify influence functions using procedures from the
literature under conditions where the convexity assumptions of influence
functions are met. Then, we relax these assumptions and study the effects of
non-convexity by using deeper models and more complex datasets. Here, we
analyze the key metrics and procedures that are used to validate influence
functions. Our results indicate that the validation procedures may cause the
observed fragility.

_K_ eywords Machine Learning $\cdot$ Supervised Learning $\cdot$ Deep Learning
$\cdot$ Explainable AI $\cdot$ Influence Functions $\cdot$ Bayesian Neural
Networks

##  1 Introduction

Due to the black-box nature of Deep Neural Networks (DNNs), explaining the
predictions of these models remains a challenging problem. Several techniques
for addressing this challenge have been proposed such as saliency maps[1],
influence functions [2], concept activation vectors [3], and activation
atlases [4]. These techniques are not without problems. The fragility of these
methods have been well studied, but few works have tried to understand where
these methods break down [5, 6].111Code and Raw output files are available at
the following url: https://github.com/jrepifano/xai_is_fragile

Influence functions were originally proposed to diagnose and debug linear
models by predicting the parameter or loss change due to removing a training
instance [7]. Their extension to deep learning models, however, did not occur
until recently [2]. Influence functions and their applications have been well
studied since their reemergence and have since been adopted as a mainstream
tool for the interpretation of deep models in a variety of data modalities [8,
9, 10, 11], including high-risk areas such as mortality prediction for
patients in the Intensive Care Unit [12]. Due to the diversity of the use
cases for influence functions, understanding their limitations is imperative
if they are to be used to explain model behavior. Without key validation
procedures, we run the risk of providing misleading or incorrect information
to the model users.

To validate these methods, we must first agree on a metric to rate
explanations. Spearman correlation between the approximate and true loss
differences has been used as a metric to determine the accuracy of influence
estimates. The approximate loss differences are given by the influence
functions and the true loss differences are obtained by retraining an already
trained network after removing a specific training sample [2]. Recent works
have used this metric to study the effects that increases in model and dataset
size have on the influence functions. It has been found that influence
functions are extremely sensitive to these increases [6].

It is well known that increases in model and dataset size affect the curvature
of the loss function [13, 14, 15, 16]. Convexity of the loss function is a
critical assumption of influence functions as they heavily rely on the
approximation of the inverse Hessian-vector product. The stochastic estimation
algorithm used to compute the inverse Hessian-vector product assumes that the
Hessian is positive semidefinite [17, 18]. Preliminary work has been done to
try to remedy these problems via higher-order approximations [19] and group
influences [20], i.e., computing loss differences for more than one training
instance at a time.

(a)

(b)

Figure 1: Influence function performance evaluation on Iris dataset. Left:
constant depth experiment. Right: constant width experiment. Spearman
correlation between the true and approximate loss differences is on the y-axis
(higher is better). The error bars represent the 95% intervals obtained by
repeating the experiment 50 times. Blue: training without weight decay.
Orange: training without weight decay but with Stochastic Weight Averaging
(SWA). Green: training with weight decay. Red: training with weight decay and
SWA. Purple: training with BNN. We observe that the influence functions that
come from BNN are significantly better than the rest of the methods in almost
all cases. SWA has significant performance increases without the presence of
regularization but with regularization has little effect and was removed for
clarity. Statistical testing using one-way ANOVA revealed no significant
difference ($p>.05$) between correlation values for any of the model types.

When discussing fragility, we must look at the whole system, not just the
method in question. Deep neural networks have been shown to be sensitive to
small perturbations via the weight initialization or by the order in which the
data is given to the model [21, 22]. The problem lies in the noisy nature of
the gradients. This problem has been linked to poor model convergence as well
as explainability and attempts to address it include Gaussian averaging [21],
Stochastic Weight Averaging (SWA) [23, 22] and model averaging through
Bayesian Inference [24].

In this paper, we examine the cases where influence functions seemingly fail,
i.e. have low Spearman correlation between approximate and true loss
differences. We obtain the operands for the correlation using the retraining
procedure introduced in [2], where the approximate loss differences are
computed for the test point with the maximal loss using influence functions.
Each training point is removed one at a time and the neural network is
retrained from the optimal parameters until convergence in order to obtain the
true difference in the loss function values. We determined that this training
procedure is not valid for most applications of deep learning and present
evidence for these cases.

##  2 Background

###  2.1 Influence Functions

Consider a standard classification problem where a label $y$ is predicted for
each feature vector $x$. Let $z_{i}=(x_{i},y_{i})$, where $i=1,2,...,N$, for
$N$ instances in the dataset. It is assumed that we have a trained model where
$\theta$ represents the trained network parameters. Our loss function can be
written as $L(z,\theta)=\sum_{i=1}^{N}L(z_{i},\theta)$. Our optimal model
parameters are the set of parameters that minimize the loss:
$\hat{\theta}=\operatorname*{arg\,min}_{\theta\in\Theta}\sum_{i=1}^{N}L(z_{i},\theta)$
[2]. Koh _et al._ [2] offers insight on how to approximate the effect that
removing a training point $z$ has on the parameters $\theta$. We compute the
parameter change with $z$ upweighted by a small value, $\epsilon$. Using this
upweighting scheme we obtain a new set of parameters,
$\hat{\theta}_{\epsilon,z}=\textrm{arg}\min_{\theta\in\Theta}\frac{1}{n}\sum_{i=1}^{n}L(z_{i},\theta)+\epsilon
L(z,\theta)$ [2]. Cook _et al._ [7] has shown that as $\epsilon$ approaches
zero the influence of $z$ on the parameters is:

$\mathcal{I}_{\textrm{up,params}}(z)=\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}\bigg{\rvert}_{\epsilon=0}=-H^{-1}_{\hat{\theta}}\nabla_{\theta}L(z,\hat{\theta}),$
(1)

where
$H_{\hat{\theta}}=\frac{1}{n}\sum_{i=1}^{n}\nabla^{2}_{\theta}L(z_{i},\hat{\theta})$
is the Hessian. If we let $\epsilon=-\frac{1}{n}$, then we can approximate the
parameter change as
$\hat{\theta}_{-z}-\hat{\theta}\approx-\frac{1}{n}\mathcal{I}_{\textrm{up,params}}(z)$
[2]. To study the effect of removing a training point on a test point
$z_{\textrm{test}}$ on the loss function, we apply the chain rule: [2]:

$\mathcal{I}_{\textrm{up,loss}}(z,z_{\textrm{test}})=-\nabla_{\theta}L(z_{\textrm{test}},\hat{\theta})^{T}H_{\hat{\theta}}^{-1}\nabla_{\theta}L(z,\hat{\theta})$
(2)

Figure 2: Spearman correlation between the true and approximate loss
differences is on the y-axis (higher is better). The error bars represent the
95% intervals obtained by repeating the experiment 10 times. We observe that
the spearman correlation is only significant in the small fully connected
model on the MNIST dataset. As the number of parameters increases, the
influence function performance falls off sharply, which was expected. There
are no significant differences between VDP and the other large models.

###  2.2 Influence Function Guidance

Ideally, a model must be trained until the optimal parameters $\hat{\theta}$
are obtained in order to compute the influence functions. For a single test
instance, $z_{\textrm{test}}$, we would then compute the inverse Hessian-
vector product,
$\nabla_{\theta}L(z_{\textrm{test}},\hat{\theta})^{T}H_{\hat{\theta}}^{-1}$,
using stochastic estimation [17]. In reality, due to non-linearities in our
networks, our objective function may become non-convex and we obtain our
parameters $\tilde{\theta}$ via SGD, where $\tilde{\theta}\neq\hat{\theta}$.
In this case, the Hessian may have negative eigenvalues which would cause the
stochastic estimation algorithm to not converge. To address this, we adopt a
regularization scheme similar to L2 regularization discussed by Koh _et
al._[2]. We regularize the computation of the Hessian-vector product using a
damping term of $\lambda=0.01$. We can then compute the gradient of the loss
as $\nabla_{\theta}L(z,\hat{\theta})$. The inner product of the Hessian-vector
product and the gradient of the training instance results in a scalar value
that tells us the approximate change in loss to expect on $z_{\textrm{test}}$
if we were to remove the training instance $z$. Note that we compute the
gradient of the loss function with respect to only the parameters of the last
layer [2].

###  2.3 Non-convexity and Eigenvalues of the Hessian

Due to the importance of the Hessian in the computation of influence
functions, the convexity of the loss function and its effects on the Hessian
are important. Recall that influence functions assume the Hessian is positive
definite such that it is invertible. Koh _et al._ [2] have shown that even
with negative Hessian eigenvalues it is still possible to obtain good
influence estimates . It is understood that large overparameterized networks
affect the convexity of the loss function [14, 15], which we observe via the
eigenvalues of the Hessian. Basu _et al._ [6] have shown that larger
eigenvalues are correlated with decreases in the Spearman correlation metric
when network depth and width are increased. This contradicts the literature
where the long tail of the Hessian Eigen Spectral Density (ESD) has been well
studied for large DNNs and it has been shown that the largest eigenvalue does
not tend to increase as width of the network increases [13]. In this paper, we
utilize a method developed by Yao _et al._ [25] to compute the eigenvalues of
the Hessian in an effort to quantify the effect if any, of non-convexity and
non-convergence on Influence functions.

###  2.4 Bayesian Deep Neural Networks

The current state of the art for influence functions, suggests that by
applying L2 regularization to our networks during training, we can reduce the
negative effects that are associated with overparameterization [6].
Variational Bayesian Learning has been shown to result in superior
regularization, better model averaging and built-in uncertainty prediction
[24]. We select this method specifically for its regularization strength.

In this subsection, we present a modified version of the Extended Variational
Inference model proposed by Dera _et al._ [26]. We assume the covariance is
zero and only propagate variance for each parameter.

For a given classification problem, we want to estimate the posterior
distribution of the weights given the data, i.e., $p(\theta|D)$. This,
however, is intractable due to the high dimensionality of the parameter space.
We can approximate the true posterior by defining a variational distribution
$q(\theta)$, which is assumed to be Gaussian. Since we want the variational
distribution to be close to the true posterior, we minimize the Kullback -
Leibler (KL) divergence.

$\begin{split}\operatorname*{arg\,min}_{\theta}\textrm{KL}(q(\theta)||p(\theta|D))=&-E_{q(\theta)}\left[\log
p(D|\theta)\right]\\\ &+\textrm{KL}(q(\theta)||p(\theta))\end{split}$ (3)

To quantify the loss for the variational learning approach, we use the
Evidence Lower Bound (ELBO), $\mathcal{L}(\theta,D)$ which consists of two
parts, namely, the expected log-likelihood of the training data given the
weights and a regularization term,

$\mathcal{L}(\theta,D)=E_{q(\theta)}[\log
p(D|\theta)]-\textrm{KL}[q(\theta)|p(\theta)]$ (4)

where $\theta$ represents the weights of the network and $D$ represents the
data label pairs. Continuing the derivation gives

$\begin{split}\mathcal{L}(\theta,D)&=\frac{1}{N}\sum_{i=1}^{N}\log\left(\prod\sigma^{2}_{\hat{y}}\right)\\\
&+\frac{1}{2}\sum_{i=1}^{N}((\hat{y}-\mu_{\hat{y}})^{2}(\sigma^{2}_{\hat{y}})^{-1})\\\
&+\frac{1}{2}\sum_{n=1}^{l}(j\log\sigma^{2}_{l}-||\mu_{l}||^{2}_{F}-j\sigma^{2}_{l})\end{split}$
(5)

where: $\hat{y}$ is the label, $\mu_{\hat{y}}$ is the output mean,
$\sigma^{2}_{\hat{y}}$ is the output variance, $l$ is the number of hidden
layers and $j$ is the number of nodes in that layer.

(a)

(b)

Figure 3: Influence Function performance evaluation on Iris dataset. Left:
constant depth experiment. Right: constant width experiment. Largest
Eigenvalue is on the y-axis. The error bars represent the 95% intervals
obtained by repeating the experiment 50 times. Blue: training without weight
decay. Orange: training without weight decay but with Stochastic Weight
Averaging (SWA). Green: training with weight decay. Red: training with weight
decay and SWA. Purple: training with BNN. This figure shows little correlation
between curvature of the loss function. Statistical testing using one-way
ANOVA showed no difference in the top eigenvalue for any model in the width or
depth experiments.

For a neural network, we define the equations to propagate the first two
moments. First, we assume the input $x$ is deterministic. The weights and
biases of the first layer, $w$ are assumed to be Gaussian. If the incoming
input is deterministic (input layer), then the output, $z$ is:

$\displaystyle\mu_{z}$ $\displaystyle=\mu_{w}^{T}x+\mu_{w_{b}}$ (6)
$\displaystyle\sigma^{2}_{z}$
$\displaystyle=x^{2}[\sigma^{2}_{w}]^{T}+\sigma^{2}_{w_{b}}$ (7)

To propagate the moments through an arbitrary element-wise non-linear
function, $f$ (e.g., ReLU, SELU), we use a first order Taylor-series
approximation, to get an output, $a$:

$\displaystyle a$ $\displaystyle=f(z)$ (8) $\displaystyle\mu_{a}$
$\displaystyle\approx f(\mu_{z})$ (9) $\displaystyle\sigma^{2}_{a}$
$\displaystyle=\sigma^{2}_{z}\odot(f^{\prime}(\mu_{z}))^{2}$ (10)

If the incoming input, $a$ is a random variable (for the intermediate layers),
then the first two moments, $\tilde{y}$ are:

$\displaystyle\mu_{\tilde{y}}$ $\displaystyle=\mu_{v}^{T}\mu_{a}+\mu_{v_{b}}$
(11) $\displaystyle\sigma^{2}_{\hat{y}}$
$\displaystyle=\sigma^{2}_{v}[\sigma^{2}_{a}]^{T}+\mu^{2}_{v}[\sigma^{2}_{a}]^{T}+\mu^{2}_{a}[\sigma^{2}_{v}]^{T}+\sigma^{2}_{v_{b}}$
(12)

To propagate through a non-linear function that is not element-wise, e.g.
softmax, we use a first order Taylor-series approximation [27]. The output,
$\hat{y}$ of the non-linear function, $g$ is:

$\displaystyle\mu_{\hat{y}}$ $\displaystyle\approx g(\mu_{\tilde{y}})$ (13)
$\displaystyle\begin{split}\sigma^{2}_{\hat{y}}&\approx\mathbf{J}^{2}_{g}\odot\sigma^{2}_{\tilde{y}}\\\
&\approx(\mu_{\hat{y}}(1-\mu_{\hat{y}}))^{2}\odot\sigma^{2}_{\tilde{y}}\end{split}$
(14)

where $\mathbf{J}_{g}$ is the Jacobian of $g$.

##  3 Experiments

###  3.1 Iris Dataset

To study the effect of random initialization on influence function estimates,
we reproduced an experiment from Basu _et al._ [6] using the Iris dataset. The
Iris dataset consists of 150 instances with 4 features and 3 classes. The
decision to use this dataset as a benchmark is due to its simplicity. To make
our models more robust to random initialization, we considered weight decay as
well as Stochastic Weight Averaging (SWA) and Bayesian Neural Networks (BNNs)
as novel additions to this experiment [22, 23, 26].

This experiment was repeated for two types of DNNs: (1) DNNs with constant
width (number of nodes in a hidden layer) and variable depth (number of hidden
layers), and (2) DNNs with constant depth and variable width. In the
experiments with variable depth, the number of nodes per hidden layer was held
constant at 5 as in Basu _et. al._ [6]. In the variable width experiments, the
depth of the network was held constant at 1, i.e., one hidden layer only. We
used the Adam optimizer with an initial learning rate of 0.001 as in Basu _et.
al._ [6]. A learning rate scheduler was used to decrease the learning rate by
a factor of 10 if the loss did not decrease for 100 epochs. For the
experiments with weight decay, we used a constant value of 0.005 as in Basu
_et. al._ [6]. Each experiment was repeated 50 times.

Koh _et al._[2] showed that fine-tuning a trained DNN from the optimal
parameters is approximately equal to retraining the same network with a
training instance removed. Therefore, to obtain the true differences in loss
when removing a test point, we replicate the training procedure outlined by
Basu _et al._ [6]. The models are initially trained for 60k epochs of full-
batch gradient descent instead of SGD. The training instances are then sorted
by their loss and the 40 training instances with the maximal loss are
identified. We then fine-tune only the top layer for 7.5k epochs when
individually removing each of the training points with the highest loss.
Finally, we compute the influence function estimates for those training
instances with respect to the test instance with the highest loss. The
Spearman correlation between the true and approximate differences in loss are
then computed. The eigenvalues of the Hessian for each network were computed
via power iteration using the PyHessian Python package [25].

(a)

(b)

Figure 4: Example of miss-relation. Left: Depth 1 width 5 network with weight
decay on Iris dataset. Right: Small FC network with 128 nodes on MNIST
dataset. We show that when the loss function is convex, our estimates match
the true loss differences (Left, scale does not matter). When the loss
function is non-convex there is significant deviation from the true loss
differences. Both left and right receive an absolute spearman correlation of
0.85, which results in a miss-relation for the right graph.

###  3.2 MNIST and CIFAR10

We drastically increase the model and dataset size to study the performance of
influence functions in non-convex settings. Similar to the experiment
described in Basu _et al._ [6], we chose to look at a small fully connected
network, LeNet, and VGG13. Each model was trained in a similar manner as our
previous experiment. The Adam optimizer was used with an initial learning rate
of 0.001 and weight decay of 0.001. The learning rate was reduced by a factor
of 10 if the loss did not decrease after 2 epochs. The test instance with the
maximal loss was used to compute the influence functions and influence
functions were computed for all training instances. We deviate from our
previous experiment when choosing the training instances to remove and
retrain. The true loss difference was computed for the top 40 most influential
training points (highest absolute value) using the re-train from optimal
approximation. The Spearman correlation between the true and estimated
differences in loss was computed.

###  3.3 Statistical Analysis

We use one-way analysis of variance (ANOVA) to compare various dependent
variables and establish statistical significance in various experiments
described above.

##  4 Results and Discussion

###  4.1 Effect of Model Size on the Influence Function Estimates

In Figure 1, we present the Spearman’s rank correlation coefficient ($\rho$)
between the true and estimated loss differences for the Iris dataset for a
variety of model types and sizes. We present four different types of models,
including a model with L2 regularization, a model without L2 regularization, a
model with SWA, and a BNN. The figure presents models trained using an
increasing number of neurons in one layer (Figure 1-A) and increasing number
of layers with fixed number of neurons in each layer (Figure 1-B). The true
loss difference is found using the re-training strategy and the estimated loss
difference is found using equation 2. The error bars represent the 95%
confidence intervals obtained by repeating the experiment 50 times. It is
evident from both sub-figures that for any type of model (L2, No-L2, SWA, and
BNN), there is a minimal effect of increasing number of neurons or number of
layers on the quality of estimate (of the influence of a training point on the
selected test data point) provided by influence functions (using equation 2).
A statistical analysis performed using ANOVA did not reveal any significant
effect of number of neurons or layers on the Spearman correlation ($p>0.5$ for
all cases). Previously, Basu _et al._ [6] had reported increasing model size
(depth and width) degrades influence function estimates. We believe that the
discrepancy between the reported results is linked to statistical rigor as no
statistical tests or analyses were reported by Basu _et al._ [6] to establish
the effect of model size on the quality of estimates produced by influence
functions.

We also observe that the estimates provided by influence function are more
accurate for models with regularization, as shown in Figure 1, in particular,
the Bayesian models (BNNs) outperform all other methods. We consider that the
observed behavior is linked to (1) the “ensemble" or “average" effect
introduced by Bayesian approaches in the model training, and (2) the type of
regularization present in the ELBO loss function which has been shown to give
these models superior self-compression properties [28]. This performance
increase, however does not seem to carry over to our experiments with larger
datasets (Figure 2), where all models were trained with regularization. This
is congruent with the results obtained by Basu _et al._ [6] on the same
datasets.

####  4.1.1 The Largest Eigen Value

In Figure 3 we observe the same trend that Basu _et. al._ [6] found in the
Iris experiment, e.g., the eigenvalues of the Hessian increase with model
width and depth (ANOVA $p<0.05$). We do not however, relate the supposed
decrease in influence function estimates to the increasing top eigenvalue as a
proxy for curvature of the loss function given that our statistical results
from Figure 1 show that there are no significant differences between model
sizes and influence function performance. Given that Koh _et al._ [2] have
shown that even when most assumptions about convexity of the loss function
have been broken, i.e., the optimal parameters have not been obtained
($\tilde{\theta}\neq\hat{\theta}$) and the Hessian has negative eigenvalues
(Hessian not PD), we can still obtain “good" influence estimates. We postulate
that the problem lies with the methods that have been used to evaluate
influence functions.

(a)

(b)

Figure 5: The loss trajectories followed during re-raining loss. Left Depth 1
width 5 network with weight decay on Iris dataset. Right Small FC network with
128 nodes on MNIST dataset. Here we show the test loss as a function of re-
training in convex and non-convex settings. The sharp jumps of the right plot
indicate that the model leaves the minima that it settled in previously, which
breaks the assumptions of the influence functions.

###  4.2 The (In-)validity of Spearman’s Rank Correlation Coefficient

Spearman’s rank correlation coefficient is an established metric for
determining the accuracy of influence function estimates [2, 6, 20]. We note
that the output of Equation 2 is the difference in the loss function value for
the test instance if the training instance is removed. This loss difference
can be positive or negative. For a training instance to be influential, it
needs to have a large magnitude.

In Figure 4, we provide an example where Spearman’s correlation coefficient is
unable to capture the underlying relationship between the true loss difference
and estimated loss difference, where the estimate is being calculated using
equation 2. The horizontal axis in both sub-figures (Figure 4 left and right)
corresponds to the rank of the training point, where the rank is determined by
the approximate loss difference. Thus, we should expect to see the exact loss
differences (blue points) move from a large magnitude towards zero as we move
from left to right on the horizontal axis. In Figure 4 (a), the estimated and
true values (after ignoring the scale) are close to each other. In Figure 4
(b), the values of true and estimated loss differences are significantly
different from each other. However, the value of Spearman’s correlation
coefficient for both cases is approximately 0.85.

We consider that since the relationship between the estimated and true loss
function difference values may not always be monotonically decreasing or
increasing, the Spearman’s correlation can lead to misleading results.

###  4.3 Re-training for Optimal Parameters

To compute the Spearman’s correlation coefficient, we need to know the true
difference in the loss function value. This requires retraining models for
every training instance that we want to analyze. This is a very costly
operation in time. The re-training from optimal parameters has been shown to
be an approximately equivalent alternative to retraining from scratch [2].
Previous works have not proven that this approximation is valid for large
datasets [2, 6]. It has been well established that increasing model and
dataset complexity increases the largest eigenvalue of the Hessian of the loss
function [14, 15]. While we have demonstrated that the increasing curvature
does not affect estimates made by influence functions with small datasets and
models, with large datasets and models the extreme curvature of the loss
function makes us question the validity of the re-training approximation [29].
To study this, we looked at the loss of the test instance at each epoch during
re-training in both the Iris and MNIST experiments. In Figure 5, the test loss
difference is plotted against epochs on the horizontal axis. We note
significant differences differences in the trajectories followed by the
gradient descent algorithm for two cases (Iris - Figure 5 left and MNIST 5
right). The Iris model has a well damped convergence whereas the the MNIST
model is underdamped and does not seem to converge as smoothly as did Iris.

###  4.4 The Effect of Large Networks

We consider large neural networks as having more parameters, more non-linear
operations owing to their depth, and reluctantly requiring large datasets for
training. We note that originally Cook and Weisberg derived influence
functions for regression models, which can be considered as neural networks
with one layer and mean-square error loss function [7]. Recently, Koh _et al._
[2] extended the idea of using influence functions in deep neural networks by
treating all but the last layer of the deep neural network as a feature
extractor. The influence functions were computed with respect to only the last
layer. This practice seems to work in some cases and produce promising results
[2]. However, it does not account for the large dimensionality of the final
layer of modern neural networks. This proves to be a problem when these large
parameter matrices become ill-conditioned [30]. This problem was captured by
Basu _et al._ [6] in their analysis of large datasets like CIFAR-100 and
ImageNet where true differences in losses from removing training instances
resulted in very noisy results.

Large neural networks may have more layers (depth) and/or more operations per
layer (width). This results in increasing the number of non-linear operations
which are performed on the data for calculating the loss function. The most
popular implementation of influence functions, as defined by Koh _et al._ [2],
relies on only a first-order Taylor series approximation to efficiently
compute influence (Eq. 21). We argue that the increasing number of non-linear
operations strongly affects the convexity assumption of loss function
$R(\theta)$ (Eq. 16) as used in the mathematical relationships derived for
influence functions (Eq. 17). There is evidence suggesting that adding the
second term of the Taylor series in the influence function approximation
improves the estimates [19, 20].

Finally, large networks typically go hand-in-hand with large datasets. From
equation 1, it is evident that removing a training instance is equivalent to
up-weighting it by $\epsilon=-\frac{1}{n}$. In the Iris dataset,
$|\epsilon|\approx 6.6e-3$ compared to MNIST where $|\epsilon|\approx 1.6e-5$.
Any larger datasets will lead to smaller epsilon, that is:

$\displaystyle\hat{\bm{\theta}}_{-z}-\hat{\bm{\theta}}\approx 0\text{ or
}\hat{\bm{\theta}}_{-z}\approx\hat{\bm{\theta}}.$ (15)

In other words, owing to the large dataset, the influence of a single training
point on a test sample is asymptotically approaching zero. Perhaps the first-
order Taylor series approximation of the influence functions does not provide
enough resolution to predict loss differences when predicting on only one
training instance. If one wanted to use influence functions in large datasets
like CIFAR-100 and ImageNet, one would have to turn to higher-order
approximations of influence functions as well as group influences. In Basu _et
al._[20], promising results were obtained by examining the effect of a second-
order approximation as well as group influence. These solutions of course have
an associated cost. Adding a second-order term increases the cost and
complexity of the analysis. Optimal group selection is also a non-trivial and
expensive operation.

While there is theoretical evidence to suggest that the first-order
implementation of influence functions is fragile, due to the difficulty of
finding robust ways to empirically evaluate them in difficult settings, their
supposed fragility remains unclear.

##  5 Limitations

While we have established that the procedures used to measure the accuracy of
influence functions are flawed in multiple ways, we have not been able to
ascertain exactly where or why these procedures break down. It appears that
the answer lies with increasing model and dataset size. To precisely define
the boundaries on where violating the approximations that Koh _et al._ [2]
have established is valid, we would need to exhaustively search the space of
increasing complexity of the model and dataset.

##  6 Conclusion

Validating the performance of explanation methods is a key area of deep
learning that has not been well studied. In particular, the validation of
influence functions in deep learning has been an area of interest. In this
work, we analyzed several experiments from the recent literature in order to
understand the fragility of influence functions. We obtained results that
conflict with those of our peers, which we attribute to the repetition in our
experimental design as well as the misuse of the Spearman correlation metric
and retraining procedure.

While we have demonstrated that the methods we use to measure the accuracy of
influence functions are flawed, we must not conclude that influence functions
are uninformative. Due to the flaw in validation methodology, we do not have
any evidence to support the claim that the explanations provided by influence
functions are not accurate or faithful to the original model. Future efforts
should be focused on developing robust validation frameworks for explainable
methods in order to foster user-model trust.

##  Appendix A Influence Function Derivation

This derivation was taken directly from Koh _et al._ [2] and has been
reproduced below:

Influence functions are considered one of the classic technique from robust
statistics that can quantify the change in model parameters attributed to up-
weighting a training point by an infinitesimal amount. In the following, we
derive mathematical relationships for influence functions, examine their
underlying assumptions, and attempt to explain these in the context of large
neural networks. We start by defining an optimization problem where
$\hat{\theta}$ minimizes the empirical risk following Koh _et al._ [2]:

$R(\theta)\stackrel{{\scriptstyle\text{def}}}{{=}}\frac{1}{n}\sum_{i=1}^{n}L(z_{i},\theta).$
(16)

A fundamental assumption for influence functions is that $R$ is twice-
differentiable and strongly convex in $\theta$. That is, the Hessian, as
defined by:

$H_{\hat{\theta}}\stackrel{{\scriptstyle\text{def}}}{{=}}\nabla^{2}R(\hat{\theta})=\frac{1}{n}\nabla^{2}_{\theta}L(z_{i},\hat{\theta}),$
(17)

exists and is positive definite. The convexity assumption guarantees the
existence of $H_{\hat{\theta}}^{-1}$. Recall, that we approximate the removal
of a training point by up-weighting the parameters by a small quantity
$\epsilon\approx-\frac{1}{n}$, where $n$ is the total number of training data
points. The perturbed parameters, $\hat{\theta}_{\epsilon,z}$ can be written
as [2]:

$\hat{\theta}_{\epsilon,z}=\textrm{arg
min}_{\theta\in\Theta}\left[R(\theta)+\epsilon L(z,\theta)\right].$ (18)

The total parameter change by up-weighting a training example can be defined
as $\Delta_{\epsilon}=\hat{\theta}_{\epsilon,z}-\hat{\theta}$. Differentiating
with respect to $\epsilon$ and noting that $\hat{\theta}$ doesn’t depend on
$\epsilon$, we can write:

$\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}=\frac{d\Delta_{\epsilon}}{d\epsilon}.$
(19)

We note that for the optimal parameters $\hat{\theta}_{\epsilon,z}$, we can
rewrite eq. 18 as:

$0=\nabla R(\hat{\theta}_{\epsilon,z})+\epsilon\nabla
L(z,\hat{\theta}_{\epsilon,z}).$ (20)

Since $\hat{\theta}_{\epsilon,z}\rightarrow\hat{\theta}$ as
$\epsilon\rightarrow 0$, the first-order Taylor expansion of the right-hand
side produces:

$0\approx\left[\nabla R(\hat{\theta})+\epsilon\nabla
L(z,\hat{\theta})\right]+\left[\nabla^{2}R(\hat{\theta})+\epsilon\nabla^{2}L(z,\hat{\theta})\right]\Delta_{\epsilon}.$
(21)

Solving for $\Delta_{\epsilon}$,

$\Delta_{\epsilon}=-\left[\nabla^{2}R(\hat{\theta})+\epsilon\nabla^{2}L(z,\hat{\theta})\right]^{-1}\left[\nabla
R(\hat{\theta})+\epsilon\nabla L(z,\hat{\theta})\right]$ (22)

Since $\hat{\theta}$ minimizes $R$, then $\nabla R(\hat{\theta})=0$.
Neglecting higher order $\epsilon$ terms,

$\Delta_{\epsilon}\approx-\nabla^{2}R(\hat{\theta})^{-1}\nabla
L(z,\hat{\theta})\epsilon.$ (23)

When we substitute Equations 17 and 19, we have:

$\left.\frac{d\hat{\theta}_{\epsilon,z}}{d\epsilon}\right|_{\epsilon=0}=-H_{\hat{\theta}}^{-1}\nabla
L(z,\hat{\theta})\stackrel{{\scriptstyle\text{def}}}{{=}}\mathcal{I}_{\text{up,params}}(z).$
(24)

#### Acknowledgments

Jacob R. Epifano is supported by US Department of Education GAANN award
P200A180055. Ghulam Rasool was partly supported by NSF OAC-2008690.

## References

  * [1] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In In Workshop at International Conference on Learning Representations. Citeseer, 2014.
  * [2] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In International Conference on Machine Learning, pages 1885–1894. PMLR, 2017.
  * [3] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In International conference on machine learning, pages 2668–2677. PMLR, 2018.
  * [4] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah.  Activation atlas.  Distill, 4(3):e15, 2019.
  * [5] Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681–3688, 2019.
  * [6] Samyadeep Basu, Phil Pope, and Soheil Feizi.  Influence functions in deep learning are fragile.  In International Conference on Learning Representations, 2020.
  * [7] R Dennis Cook and Sanford Weisberg.  Residuals and influence in regression.  New York: Chapman and Hall, 1982.
  * [8] Gilad Cohen, Guillermo Sapiro, and Raja Giryes.  Detecting adversarial samples using influence functions and nearest neighbors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453–14462, 2020.
  * [9] Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong.  Fastif: Scalable influence functions for efficient model interpretation and debugging.  arXiv preprint arXiv:2012.15781, 2020.
  * [10] Donghoon Lee, Hyunsin Park, Trung Pham, and Chang D Yoo.  Learning augmentation network via influence functions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10961–10970, 2020.
  * [11] Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov.  Explaining black box predictions and unveiling data artifacts through influence functions.  arXiv preprint arXiv:2005.06676, 2020.
  * [12] Jacob R Epifano, Ravi P Ramachandran, Sharad Patel, and Ghulam Rasool.  Towards an explainable mortality prediction model.  In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2020.
  * [13] Levent Sagun, Leon Bottou, and Yann LeCun.  Eigenvalues of the hessian in deep learning: Singularity and beyond.  arXiv preprint arXiv:1611.07476, 2016.
  * [14] Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou.  Empirical analysis of the hessian of over-parametrized neural networks.  arXiv preprint arXiv:1706.04454, 2017.
  * [15] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.  An investigation into neural net optimization via hessian eigenvalue density.  In International Conference on Machine Learning, pages 2232–2241. PMLR, 2019.
  * [16] Guillaume Alain, Nicolas Le Roux, and Pierre-Antoine Manzagol.  Negative eigenvalues of the hessian in deep neural networks.  arXiv preprint arXiv:1902.02366, 2019.
  * [17] Barak A Pearlmutter.  Fast exact multiplication by the hessian.  Neural computation, 6(1):147–160, 1994.
  * [18] Naman Agarwal, Brian Bullins, and Elad Hazan.  Second-order stochastic optimization for machine learning in linear time.  The Journal of Machine Learning Research, 18(1):4148–4187, 2017\.
  * [19] Pang Wei Koh, Kai-Siang Ang, Hubert HK Teo, and Percy Liang.  On the accuracy of influence functions for measuring group effects.  arXiv preprint arXiv:1905.13289, 2019.
  * [20] Samyadeep Basu, Xuchen You, and Soheil Feizi.  On second-order group influence functions for black-box predictions.  In International Conference on Machine Learning, pages 715–724. PMLR, 2020.
  * [21] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [22] Pranava Madhyastha and Rishabh Jain.  On model stability as a function of random seed.  arXiv preprint arXiv:1909.10447, 2019.
  * [23] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Averaging weights leads to wider optima and better generalization.  arXiv preprint arXiv:1803.05407, 2018.
  * [24] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In International Conference on Machine Learning, pages 1613–1622. PMLR, 2015.
  * [25] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney.  Pyhessian: Neural networks through the lens of the hessian.  In 2020 IEEE International Conference on Big Data (Big Data), pages 581–590. IEEE, 2020.
  * [26] Dimah Dera, Ghulam Rasool, and Nidhal Bouaynaya.  Extended variational inference for propagating uncertainty in convolutional neural networks.  In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2019.
  * [27] Dan Simon.  Optimal state estimation: Kalman, H infinity, and nonlinear approaches.  John Wiley & Sons, 2006.
  * [28] Giuseppina Carannante, Dimah Dera, Ghulam Rasool, and Nidhal C Bouaynaya.  Self-compression in bayesian neural networks.  In 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2020.
  * [29] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse.  Three mechanisms of weight decay regularization.  arXiv preprint arXiv:1810.12281, 2018.
  * [30] David A Belsley, Edwin Kuh, and Roy E Welsch.  Regression diagnostics: Identifying influential data and sources of collinearity.  John Wiley & Sons, 2005.