# Designing an attack-defense game: how to increase robustness of financial
transaction models via a competition

Alexey Zaytsev1, Alex Natekin2, Evgeni Vorsin3, Valerii Smirnov3, Georgii
Smirnov3, Oleg Sidorshin4, Alexander Senin3, Alexander Dudin, Dmitry
Berestnev5

###### Abstract

Given the escalating risks of malicious attacks in the finance sector and the
consequential severe damage, a thorough understanding of adversarial
strategies and robust defense mechanisms for machine learning models is
critical. The threat becomes even more severe with the increased adoption in
banks more accurate, but potentially fragile neural networks. We aim to
investigate the current state and dynamics of adversarial attacks and defenses
for neural network models that use sequential financial data as the input.

To achieve this goal, we have designed a competition that allows realistic and
detailed investigation of problems in modern financial transaction data. The
participants compete directly against each other, so possible attacks and
defenses are examined in close-to-real-life conditions. Our main contributions
are the analysis of the competition dynamics that answers the questions on how
important it is to conceal a model from malicious users, how long does it take
to break it, and what techniques one should use to make it more robust, and
introduction additional way to attack models or increase their robustness.

Our analysis continues with a meta-study on the used approaches with their
power, numerical experiments, and accompanied ablations studies. We show that
the developed attacks and defenses outperform existing alternatives from the
literature while being practical in terms of execution, proving the validity
of the competition as a tool for uncovering vulnerabilities of machine
learning models and mitigating them in various domains.

## Introduction

The evolution of the modern financial sector has been marked by rapid
advancements in technology, enabling financial institutions to offer better
services with improved efficiency. One of the main contributors over the last
decade is machine learning (Dixon, Halperin, and Bilokon 2020; Babaev et al.
2019). Enhanced model quality built on timely and accurately collected data
leads to improvement of quality and decision making speed in banks (Tripathi
and Pavaskar 2012). However, these advancements have simultaneously opened up
new channels for malicious actors to exploit, one of which is the emergence of
adversarial attacks on machine learning models. The issue becomes even more
pressing in the context of financial transaction data, where the stakes have
explicit monetary value, and robust defense mechanisms are needed (Fursov et
al. 2021).

While close to natural language (Qiu et al. 2022) and event sequences data
(Khorshidi, Wang, and Mohler 2022; Shchur et al. 2021) in structure, financial
transaction data that consists of sequences of transactions produced by
customers have notable differences. They include in particular dependence on
macroeconomic situation, higher required attention ranges and higher diversity
of available features (Fursov et al. 2021).

Competitive evaluation has emerged as an effective way to measure and foster
advancements in machine learning (Goodfellow et al. 2013). However, current
competitions often overlook the two-side dynamics of adversarial attacks and
defenses and tend to ignore the unique challenges posed by financial
transaction data (Croce et al. 2021).

Figure 1: Competition scheme for attack and defense tracks

Through our investigation, we aim to contribute valuable insights into the
robust models landscape of the financial sector handling emerging risks
associated with artificial intelligence. In particular, the paper introduces a
solid approach to advance the development of robust models for the processing
of financial transaction data. Our primary contributions are:

  * •

A novel competition scheme: We propose a remarkable competition framework to
evaluate the robustness of machine learning models in the financial domain.
Our tournament phase encourages participants to actively probe and defend
against vulnerabilities, simulating real-world scenarios and enhancing the
reliability of the models. In addition, we release a new open dataset of
sequences of financial transactions with the credit default as the target.

  * •

Dynamic analysis: Through analyzing the dynamics of the competition, we
uncover insights into model sustainability and identify crucial factors
influencing attack and defense quality. Such analysis enables us to gauge the
models’ real-world efficacy.

  * •

Tailored strategies for financial transactions: We demonstrate that financial
transaction data require specialized algorithms for their attacks and
defenses, differentiating them from their counterparts in computer vision and
NLP. The experiments were conducted via open datasets: a dataset available
during the competition and additional widely used in the literature. We
hypothesize that similar techniques can work for the event sequence data.

The rest of this paper is structured as follows. Section Competition structure
presents the design and the structure of our competition. Section Results
delves into the analysis of the competition’s dynamics and the findings
related to the robustness of the models as well as the comparison of developed
attacks and defenses to existing baselines. Section Related work is devoted to
related work on the topic.

## Competition structure

The overall scheme for the competition is in Figure 1. Below we provide more
details on it.

### Data

The competition uses a year’s worth of anonymized financial transactions. The
data are complemented with a credit default target, often being most important
to a bank. The default rate in data is $0.04$. For each customer, we have a
sequence of transactions, with each transaction having an MCC, amount,
currency and time available. The MCC is the merchant category code that belong
to about $1000$ categories, like ATM cash or drug store visit. Together with
transaction amount, it is one of the most important indicators of customer’
behaviour. For competition purposes, we split all transactions into nine
cohorts, so parts of the data are independent of each other and revealed to
participants sequentially.

##### Cohorts available for participants

The cohort for adversarial attack design consists of 4200 customers without
labels. The cohort for the refinement and defense design has 7080 customers
with labels. For the sake of privacy, all transaction amounts within the
dataset have been anonymized to keep the privacy of the data. Each customer
within the dataset has at least 300 associated transactions in a sequence.

### Attack track

The goal of the attack track is to develop an attack that significantly
changes the output of a given model after a minor change of input. During
regular phase, an attacker has full access to the model. The details about the
GRU-based model to attack is provided in Appendix.

During the tournament phase, she or he attacks the model with no knowledge
about it. A participant alters a given sequence of financial transactions and
targets the ROC AUC degradation of the model quality. Below we provide details
on the given model, evaluation metrics, and restrictions that define an
adversarial attack.

##### Evaluation

We developed a system for automatic evaluation of solutions. It starts with a
sanity check that a solution satisfies change constraints. Then, the score is
calculated as the difference in ROC AUC value between the model’s predictions
for the initial set of transactions and the adversarially altered set of
transactions. To make conditions fairer, during the competition, participants
can access the results only on one half of the test set (a public part), while
the final ranking is identified via another half of the test set (a private
part).

##### Restriction for participants’ attacks

In designing the attack, we imposed constraints to preserve the authenticity
of a sequence, as banks often have models to detect and ignore fake
transactions. Specifically, an attack changes up to 10 transactions per
client. The permissible amounts are within minimum and maximum for the
considered MCC. To avoid issues with boundary substitutions, we slightly
reduced this interval with a coefficient $0.95$.

##### Baseline attack

As a baseline, we adopt a simple attack strategy. The model to attack produces
confidence scores for each example. We identified two representative customers
with top model scores for both classes. We take $10$ last transactions from
the representative customers and add them to sequences for customers from
opposite classes.

Despite these manipulations, the impact on the model was relatively modest,
with the model demonstrating robustness to the attack, maintaining a ROC AUC
score of approximately $0.69$. This suggests that the model was still capable
of fairly accurate predictions in the presence of the baseline attack.

### Defense track

Similar to the attack task, participants have access to a banking GRU binary
classification model that predicts client default. The competition also
provides access to a small labeled sample of client data with accompanying
materials. The exact format in which this model will be attacked is
alternation of a small number of transactions in a sequence fed into the
model.

The task is to construct a robust model for the same classification problem,
making it resilient to such vulnerabilities. While it may be beyond your
capabilities to prevent data breaches, it is entirely within your reach to
strive to protect the model from such adversities.

Figure 2: Solid lines are empirical cumulative density functions for different
stages of the attack track of the competition. Dashed lines are the top score
for each phase and a baseline score. Bigger ROC AUC differences are better
Figure 3: Solid lines are empirical cumulative density functions for different
stages of the attack track of the competition. Dashed lines are the top score
for each phase (1 or 2) and baseline scores. Bigger Harmonic means of ROC AUCs
are better

##### Evaluation

We calculate the ROC AUC values for model predictions for two samples: clean
data and attacked data. The harmonic mean of these two values is the final
quality metric. We compare this metric with the metric obtained via a baseline
model trained without any defense capabilities in mind.

##### Baseline for the defense track

For a baseline defense, we create a lightweight ensemble. We sample a share of
$0.9$ transactions from an initial sequence. We repeat this procedure $9$
times and average obtained predictions. Our investigation shows that for this
design of defense, we improve over defense’s main metric, reducing the effect
of the baseline attack Also, after the application of such a procedure, the
baseline attack has negligible adversarial properties.

##### Tournament phase

To facilitate the development of better approaches, we add separate tournament
phases after each track. Correspondingly, we have an attack track tournament
and a defense track tournament.

The design is the following: we select participants with top-10 scores for the
attack track and top-10 scores for the defense track. Each participant
presents two solutions, so we have 20 solutions for attack and 20 solutions
for defense in total.

As a solution for the attack track is a modified list of sequences of
transactions, and a solution for the defense track is an updated model
available as a Docker container, we can evaluate the quality for each pair of
the attack and the defense.

So, we have a matrix of scores. For the attack, it is the matrix of ROC AUC
differences between initial ROC AUC and ROC AUC for adversarial data. For the
defense, it is the matrix of harmonic means for ROC AUC values for a defended
model for clean and adversarially perturbed data.

The best attack is the attack with the best average score. The best defense is
the defense with the best average score. As we select the best attacks and
defenses, we can now rank their transferability and universality among top
solutions.

## Results

We split our analysis into two parts. The first part is devoted to evidence
obtained from competition results on the dynamic of it and presented
solutions. The second part is about wider experiments aimed at validating
conclusions from the competition in a more traditional setting.

### Competition results

##### Attack quality

The number of participants during the attack phase of the competition is $58$.
The participants included teams with strong experience in machine learning
competitions. They submitted $649$ solutions during the presented phases. As
we have two sub-phases for the attack phase, we can observe not only the
comparison of private and public scores but also the progress of scores during
the second part of the competition. There were two control points which we
call phases.

We present the empirical cumulative density function for top scores for each
participant for private and two public phases for an attack of a white box
model and a private score for an attack of a black box model in Figure 2. Only
the top ten participants with respect to private white box scores attacked
black box models. Finally, we expand our analysis with the dynamic of top
score in Figure 4

We can make several conclusions from these data. At first, as the initial ROC
AUC was about $0.6$, after the attack we see degradation to about $0.2$ score,
which makes the model unusable. This effect doesn’t hold for the black box
regime: even the strongest attacks only marginally harm the model. So, a
stinging attack should have access to a model. During the second stage of
attack against a black box model, we see that the improvement in the
robustness of models made attacks almost useless. Dynamic of the score suggest
that after obtaining a model it is enough to spent about two weeks to
completely break the model, so model owners should take actions after the
model leakage in a timely manner.

Figure 4: Dynamic of the top score for the attack competition. We report the
top score at each moment along with scores of submissions and Baseline score.
All reported scores are calculated at the private part of the data and was not
revealed until the end of the competition Figure 5: Dynamic of the top score
for the defense competition. We report the top score at each moment along with
scores of submissions and Baseline score. All reported scores are calculated
at the private part of the data and was not revealed until the end of the
competition

##### Defense quality

As the format of this part was less traditional, the number of participants
was $42$, while the strongest participants submitted to both tracks. Here we
also present the results for different stages: two stages for a given attack
and two tournament stages for unknown attacks developed by other participants.

We present the empirical CDF for the defense track in Figure 3. We expand our
analysis with the dynamic of top score in Figure 5.

The trends are in many ways similar to the attack track. One can improve over
the baselines if the model hasn’t been protected before. The part of the
improvement comes from the improvement of the model quality for clean data, as
we use the harmonic mean of the quality for clean and attacked data for
evaluation. We also note while weaker attacks lead to minor effects, the
defences significantly degrades when being put against stronger attacks. For
tournament phases, the protected models’ scores are close to baselines. We
also note a slight improvement if we compare the first and the second
checkpoints. The model would be significantly better defended after two weeks
of efforts making it an estimation of how much time do we need to break the
model. Finally, during the last month of the competition there were no
improvement in robustness, suggesting that we need more information and better
data to make further progress.

##### Defence versus attack quality

During the black box tournament stage of a competition, participants submit
their defended models and their attacked sequences of transactions. These
attacks and defended models stand against each other. The number of
participants at this stage was $10$ for both attacks and defenses. They were
selected as the ones that had the top $10$ scores during the previous non-
tournament stage. There were two tournament stages, but the results for them
are similar, so we provide detailed information only for the second one.

We present the ROC AUC decrease for each pair of attack and defense in Figure
6. We also provide scores for all attacks and defenses sorted by their median
values in Appendix.

Figure 6: ROC AUC decrease for pairs of attack and defense from the
competition stage. We removed unfair scores for pairs when the attack and
defense model authors were the same. We mark these pairs as white squares.

The solutions significantly differ in quality. That’s true for both attacks
and defense. Also, despite pure black-box nature of this phase, we still see
defenses and attacks with either almost zero decrease of ROC AUC and about
$0.05$ decrease of ROC AUC on average, suggesting that even in this regime one
can either defend or attack a model. We also see, that the attack transfers:
there is small number of attacks that perform well for some models, but have
bad quality on average.

### Considered attacks and defenses

#### Defended models

We consider five models with different defensive properties and raw data
quality produced by top teams. We start with a basic GRU-based neural network
NN base. Then we use the same model, but with augmentations: we run NN $10$
times, taking $0.9$ share of transactions uniformly random without return and
performing random permutation of transactions. Then we average the outputs to
get NN mix baseline. In addition to neural networks we consider three variants
of gradient boosting ensembles of decision trees. We start with a single
gradient boosting Boosting base with $400$ features in total aggregated from
sequences of transactions. The aggregates are mostly done via mean and sums
over a single MCC. We use CatBoost (Prokhorenkova et al. 2018) implementation
with training via distillation from an NN mix output. The second boosting
Boosting mix, 5 is a weighted average of two boosting models produced via
Boosting baseline and three boosting models that were construct without usage
of features that can be changed during an adversarial attack. The two first
models are produced via CatBoost and LightGBM (Ke et al. 2017) implementations
of a gradient boosting respectively. The last three models are based on
CatBoost. The weights of models were $0.5$, $0.5$, $1$, $1$, and $1$
correspondingly. Boosting mix, 2 uses only two first models from Boosting mix,
5. Finally, for Boosting mix filter we trained a Filter classifier that
identify transactions that are likely to be changed via an adversarial attack
and filter out such transactions keeping only reliable ones. A Filter
classifier is another gradient boosting that takes a single financial
transaction as an input that was trained using a subset of different attacks.
Boosting mix fiter and Boosting mix 5 are the approaches used by winner team
of the defence track.

Model | Public | Private
---|---|---
NN base | 0.7035 | 0.6876
NN mix | 0.7134 | 0.6960
Boosting base | 0.7415 | 0.7279
Boosting mix, 2 | 0.7403 | 0.7255
Boosting mix, 5 | 0.7519 | 0.7221
Boosting mix filter | 0.7529 | 0.7197
Boosting clean | 0.6883 | 0.6309
Table 1: ROC AUC values for considered models for clean data Model | No | Random | NN | Boost. | Boost. | NN base | NN mix | NN base + | NN base +
---|---|---|---|---|---|---|---|---|---
| attack |  | base | base | mix alt | gradient | gradient | Boost. base | Boost. mix alt
NN base | 0.7035 | 0.7007 | 0.3958 | 0.6924 | 0.6884 | 0.3343 | 0.6607 | 0.6720 | 0.4399
NN mix | 0.7134 | 0.7192 | 0.7048 | 0.7105 | 0.7127 | 0.7134 | 0.6976 | 0.7108 | 0.7076
Boosting base | 0.7415 | 0.7269 | 0.7188 | 0.3783 | 0.6038 | 0.7405 | 0.7017 | 0.3780 | 0.6670
Boosting mix, 2 | 0.7432 | 0.7383 | 0.7408 | 0.4250 | 0.7010 | 0.7458 | 0.7333 | 0.7251 | 0.4275
Boosting mix, 5 | 0.7519 | 0.7457 | 0.7473 | 0.5347 | 0.7247 | 0.7544 | 0.7426 | 0.5374 | 0.7361
Boosting mix filter | 0.7529 | 0.7461 | 0.7556 | 0.7157 | 0.7509 | 0.7519 | 0.7425 | 0.7148 | 0.7528
Table 2: ROC AUC values for various attacks and models for the initial Default dataset. We highlight with the bold font $\leq 0.6$ ROC AUC scores. Attack | No | NN | Boosting
---|---|---|---
| attack | base | base
Change # |  | 3 | 5 | 3 | 5
NN base | 0.719 | 0.388 | 0.361 | 0.694 | 0.685
NN mix | 0.721 | 0.652 | 0.594 | 0.708 | 0.698
Boost. base | 0.807 | 0.783 | 0.766 | 0.438 | 0.393
Boost. mix | 0.812 | 0.804 | 0.799 | 0.689 | 0.585
Table 3: ROC AUC values for various attacks and models for the Churn dataset.
We present results for different number of possible changes. We highlight with
the bold font $\leq 0.6$ ROC AUC scores.

#### Attacks

As the number of MCCs is limited, we adopt attacks close to a greedy brute
force approach. At each step, we generate a preselected large number of
possible substitutions of substitutions of transactions and select the ones
that most significantly decreases the model’s score. We generate $k_{0}=1000$
candidates at each iteration. We repeat this procedure for $k$ steps, where
$k$ is the number of allowed substitutions. In most cases $k=10$, if not
mentioned otherwise We consider attacks on NN base, NN mix, Boosting base,
Boosting mix alt and name the attacks correspondingly. Alternative Boosting
mix alt is an attack on an ensemble of boostings different from that used in
the model Boosting mix alt, which is the approach used by the winner of the
attack track, using a variety of different boosting models to generalize
better to different black-box defences. We also consider combinations of
models to try to produce stronger and varied attacks NN base + Boosting base
and NN base + Boosting mix. For them, we use an average score from two models
for the attack.

To enrich the space of attacks we also consider attack with a similar number
of random changes and two gradient-based attacks NN base grad, NN mix grad.
They are a variant of FGSM from the article (Fursov et al. 2021) that applies
gradient in the embedding space to generate the next substitution with
subsequent replacement via the closest by $l_{2}$ norm token.

### Numerical results

In this subsection, we present numerical results for a separate realization of
top solutions from the competition.

#### Model performance for clean data

Table 1 presents ROC AUC scores for introduced approaches for private and
public parts of the training dataset. For the the sake of comparison, we also
present scores for Boosting clean model that uses no information available for
the attack. Firstly, sequential nature of data can be safely ignored, as
permutation of input and aggregation after ensembling lead to the performance
improvement. Secondly, even if distilled from a neural network model, a
gradient boosting shows improved performance. The last two boosting models
should stand out only in the attack scenario, as they are designed with
improved robustness in mind. We also observe that the overall ranking of
models for public and private parts of the dataset are close, signifying
relative stability of used evaluation.

#### Attack and defense performance

Table 2 presents ROC AUC scores for pairs of considered attacks and defenses
for the default dataset released with the competition. Table 3 presents
results for an additional dataset with the churn (client leaving) target
frequently used in the literature (Fursov et al. 2021). The first conclusion
to make is that competition performance highly correlates with deeper
investigation conducted here, suggesting that a competition format is a viable
way to develop and validate new approaches for adversarial robustness
performance. The conclusion also replicates for another considered dataset. We
also observe that all attacks decrease performance of models. However, for
most of the cases while the decrease harms the model predictive power,
outperforming random changes. Notable exception is when we attack the model in
a white-box scenario. In this case, some attacks can actually try to reverse
the ranking produced by the model, leading to $<0.5$ ROC AUC. Last but no the
least is that gradient boosting models demonstrate better attack robustness,
especially combined with filtering and ensembling.

#### Sensitivity to the number of changed transactions

To make comparison deeper, we varied various constraints for attack and found
the number of possible changes the most significant. We present the comparison
for different number of possible changes in Figure 7 and appendix. If models
are similar, adding more possible steps to attack helps a lot. If they are
different, the difference in little.

Figure 7: Comparison of ROC AUC values after attacks for $3$ and $10$
admissible changes. We separately present cases with models of similar
architecture, of different architecture, and with random changes.

## Related work

The finance sector remains a prime target for malevolent actors. Malicious
actions of various types lead to significant losses to banks and their
customers (Tripathi and Pavaskar 2012). Broad adoption of complex machine
learning models in banking led to new risk management opportunities. So, the
emergence of adversarial attacks on such models poses challenges that require
urgent attention. In light of this, there has been a growing interest in
studying the mechanisms behind adversarial attacks and developing defense
systems. However, only a tiny part of this research involves understanding and
analyzing financial transaction data modeling.

##### Models of Financial Transactions Data

Sequences of financial transaction data offer a comprehensive understanding of
a client’s behaviors. Neural networks (NNs) have been widely adopted in this
area due to their ability to process large-scale, complex sequences with high
performance without intermediate steps of feature generation. Research
indicates the utilization of convolutional neural networks (CNN) (Fursov et
al. 2021) and recurrent neural networks (RNN) (Babaev et al. 2019) for
prediction tasks, including credit scoring and churn detection. The connection
between clients can also be explored via the adoption of graph neural networks
(Sukharev et al. 2020), as the connections between clients often provide a
rich source of information about their default behaviour (Muñoz-Cancino et al.
2023). The presented neural networks can also serve as encoders, providing
representations suitable for solving numerous problems (Babaev et al. 2022).
We also note a connection between event sequences often described as temporal
point processes and sequences of financial transactions, as papers in this
area routinely use such datasets as a part of their methods’ evaluation
protocol (Zhuzhel et al. 2023). Moreover, financial transactions from major
bank clients are a good indicator of macroeconomic trends, making possible
predictions of diverse macroeconomic indexes (Begicheva and Zaytsev 2021).
Nowadays, decisions including central for bank credit scoring and overall
strategy rely on neural network-based processing of financial transaction
data.

##### Adversarial Attacks and Defenses

As such, the usage of neural network models expands despite known
vulnerabilities such as adversarial attacks. Small adversarial perturbations
of an input to NN can lead to significant change of output (Szegedy et al.
2014), often completely disrupting model predictive power. A taxonomy of
attacks (Tabassi et al. 2019) and recent reviews (Xu et al. 2020) mention
several types of attacks, such as evasion, poisoning, and reverse engineering
attacks. Meanwhile, the defense strategies range from adversarial training and
defensive distillation to feature squeezing and ensembling. These defenses
fortify the models against adversarial interference by enhancing their ability
to resist perturbations or by detecting and nullifying the effects of attacks.
The horizons of applications of machine-based attacks continue to expand in
recent years (Jueguen 2023). One direction is to explore the vulnerability of
different types of models, including decision trees (Vos and Verwer 2022) and
logistic regression (Dan, Wei, and Ravikumar 2020). Another option is to
consider different application areas. In particular, adversarial attacks have
gained significant attention in the financial sector due to the critical
implications of successful breaches (Kumar et al. 2021) even for tabular
imbalanced data (Cartella et al. 2021). The paper (Fursov et al. 2021)
considered adversarial attacks and defenses for financial transactions data
modality. They proposed to use a gradient-enhanced generative model to create
an attack and considered discriminator-based and adversarial training-based
defense strategies. Other approaches for attacking event sequence models are
presented in (Khorshidi, Wang, and Mohler 2022). In sum, these attacker-
defender opposition continues leading to advancement of both robustness and
attack strength.

##### Competitions in machine learning

A natural way to define a two-sided game is a contest. Competitions have
emerged as a powerful tool to drive innovation and accelerate progress in
machine learning and AI research. One of the pioneering works examining the
importance of contests in machine learning is presented in the work
(Goodfellow et al. 2013). It discusses how one should design competitions and
how the results move forward research in machine learning. The competitions
foster a collaborative and competitive environment that prompts participants
to devise novel methods and algorithms. The current state-of-the-art in
machine learning competitions is presented in (Carlens 2023). Most
competitions focus on improving the performance of the models, paying little
attention to the robustness of proposed solutions. Moreover, in all
competitions, participants don’t directly compete against each other, and all
interactivity at available platforms is constrained to RL-related contests.
Moreover, in all competitions, participants don’t compete against each other,
and all interactivity at available platforms is constrained to RL-related
contests.

An alternative to competitions are benchmarks that collect metrics from
various papers, the most prominent being _paperswithcode_
111https://paperswithcode.com/. Specifically in the context of adversarial
attacks and defenses, the article (Dong et al. 2020) proposed adversarial
robustness benchmarks for computer vision problems that was utilized in a
competition (Dong et al. 2023). Another essential benchmark of adversarial
defenses was presented in (Croce et al. 2021). The authors proposed to
evaluate several models against the AutoAttack approach for computer vision
problems. However, due to apparent constraints, competitions are often limited
to a static environment without confrontation of participants.

##### Research Gap

Despite the significant advancements in understanding adversarial attacks and
defenses, a gap persists in financial transaction data. Available research
focuses on other data modalities and types of models. Furthermore, although
machine learning competitions appear for various machine learning problems,
they don’t concentrate on financial transaction data and work in a static
environment. Our study aims to bridge this gap by developing a competition to
explore adversarial strategies in financial systems and evaluate their
countermeasures to enhance their robustness. Due to the described competition
design, we can identify the best attacks and defenses, improving our
understanding of constructing robust models based on financial transaction
data.

## Conclusions

We see that the best options for developing the most powerful attacks and
defense strategies rely on principles common to most data science competitions
and adversarial attacks literature. However, the competition format allows
more flexibility, leading to new insights into how adversarial attacks work in
close to real-life black and white-box regimes. We identified a set of
hypotheses to check and run an in-detail comparison afterward to create the
best attack and improve existing ones. Moreover, from the competition, we
identify how long it takes to break and protect a model.

For machine learning models that use sequences of financial transaction data
as the input, it is possible to run a brute-force greedy search of an attack.
We usually obtain a model breakage with only about $3\%$ of transactions
altered. While results are more impressive for images, the attacks are also
possible for financial transactions, potentially leading to significant
changes in model output.

Regarding the defense, the strongest countermeasure is model concealing, as
the transfer of attacks shows moderate results in all considered scenarios.
Another option is to increase the number of models in an ensemble that make a
decision and include a model with input features that that are difficult to
manipulate in an attack. Also, an option is to filter the input elements of a
sequence: we can identify suspicious transactions and don’t consider them
during model inference. As we work at a token level, we still can infer the
output of a model.

Finally, our investigation suggests that with limited amounts of training data
and possibilities of using effective feature engineering approaches, gradient
boosting models can be efficiently distilled from neural networks with similar
or even better quality on unstructured data, while being significantly more
robust to adversarial attacks.

## References

  * Babaev et al. (2022) Babaev, D.; Ovsov, N.; Kireev, I.; Ivanova, M.; Gusev, G.; Nazarov, I.; and Tuzhilin, A. 2022.  CoLES: contrastive learning for event sequences with self-supervision.  In _Proceedings of the 2022 International Conference on Management of Data_ , 1190–1199.
  * Babaev et al. (2019) Babaev, D.; Savchenko, M.; Tuzhilin, A.; and Umerenkov, D. 2019.  ET-RNN: Applying deep learning to credit loan applications.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2183–2190.
  * Begicheva and Zaytsev (2021) Begicheva, M.; and Zaytsev, A. 2021.  Bank transactions embeddings help to uncover current macroeconomics.  In _2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 1742–1748. IEEE.
  * Carlens (2023) Carlens, H. 2023.  State of Competitive Machine Learning in 2022.  _ML Contests Research_.  Https://mlcontests.com/state-of-competitive-data-science-2022.
  * Cartella et al. (2021) Cartella, F.; Anunciacao, O.; Funabiki, Y.; Yamaguchi, D.; Akishita, T.; and Elshocht, O. 2021.  Adversarial attacks for tabular data: Application to fraud detection and imbalanced data.  _arXiv preprint arXiv:2101.08030_.
  * Croce et al. (2021) Croce, F.; Andriushchenko, M.; Sehwag, V.; Debenedetti, E.; Flammarion, N.; Chiang, M.; Mittal, P.; and Hein, M. 2021.  RobustBench: a standardized adversarial robustness benchmark.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Dan, Wei, and Ravikumar (2020) Dan, C.; Wei, Y.; and Ravikumar, P. 2020.  Sharp statistical guaratees for adversarially robust gaussian classification.  In _International Conference on Machine Learning_ , 2345–2355. PMLR.
  * Dixon, Halperin, and Bilokon (2020) Dixon, M. F.; Halperin, I.; and Bilokon, P. 2020.  _Machine learning in finance_ , volume 1170.  Springer.
  * Dong et al. (2020) Dong, Y.; Fu, Q.-A.; Yang, X.; Pang, T.; Su, H.; Xiao, Z.; and Zhu, J. 2020.  Benchmarking adversarial robustness on image classification.  In _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 321–331.
  * Dong et al. (2023) Dong, Y.; Liu, C.; Xiang, W.; Su, H.; and Zhu, J. 2023.  Competition on robust deep learning.  _National Science Review_ , 10(6): nwad087.
  * Fursov et al. (2021) Fursov, I.; Morozov, M.; Kaploukhaya, N.; Kovtun, E.; Rivera-Castro, R.; Gusev, G.; Babaev, D.; Kireev, I.; Zaytsev, A.; and Burnaev, E. 2021.  Adversarial attacks on deep models for financial transaction records.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2868–2878.
  * Goodfellow et al. (2013) Goodfellow, I. J.; Erhan, D.; Carrier, P. L.; Courville, A.; Mirza, M.; Hamner, B.; Cukierski, W.; Tang, Y.; Thaler, D.; Lee, D.-H.; et al. 2013.  Challenges in representation learning: A report on three machine learning contests.  In _Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part III 20_ , 117–124. Springer.
  * Jueguen (2023) Jueguen, J. V. 2023.  _Broadening the Horizon of Adversarial Attacks in Deep Learning_.  Ph.D. thesis, Universidad del País Vasco-Euskal Herriko Unibertsitatea.
  * Ke et al. (2017) Ke, G.; Meng, Q.; Finley, T.; Wang, T.; Chen, W.; Ma, W.; Ye, Q.; and Liu, T.-Y. 2017.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ , 30.
  * Khorshidi, Wang, and Mohler (2022) Khorshidi, S.; Wang, B.; and Mohler, G. 2022.  Adversarial Attacks on Deep Temporal Point Process.  In _2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 1–8. IEEE.
  * Kumar et al. (2021) Kumar, N.; Vimal, S.; Kayathwal, K.; and Dhama, G. 2021.  Evolutionary adversarial attacks on payment systems.  In _2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA)_ , 813–818. IEEE.
  * Loshchilov and Hutter (2018) Loshchilov, I.; and Hutter, F. 2018.  Decoupled Weight Decay Regularization.  In _International Conference on Learning Representations_.
  * Mei and Eisner (2017) Mei, H.; and Eisner, J. M. 2017.  The neural hawkes process: A neurally self-modulating multivariate point process.  _Advances in neural information processing systems_ , 30.
  * Muñoz-Cancino et al. (2023) Muñoz-Cancino, R.; Bravo, C.; Ríos, S. A.; and Graña, M. 2023.  On the combination of graph data for assessing thin-file borrowers’ creditworthiness.  _Expert Systems with Applications_ , 213: 118809.
  * Prokhorenkova et al. (2018) Prokhorenkova, L.; Gusev, G.; Vorobev, A.; Dorogush, A. V.; and Gulin, A. 2018.  CatBoost: unbiased boosting with categorical features.  _Advances in neural information processing systems_ , 31.
  * Qiu et al. (2022) Qiu, S.; Liu, Q.; Zhou, S.; and Huang, W. 2022.  Adversarial attack and defense technologies in natural language processing: A survey.  _Neurocomputing_ , 492: 278–307.
  * Shchur et al. (2021) Shchur, O.; Türkmen, A. C.; Januschowski, T.; and Günnemann, S. 2021.  Neural temporal point processes: A review.  _arXiv preprint arXiv:2104.03528_.
  * Shen et al. (2018) Shen, G.; Tan, Q.; Zhang, H.; Zeng, P.; and Xu, J. 2018.  Deep learning with gated recurrent unit networks for financial sequence predictions.  _Procedia computer science_ , 131: 895–903.
  * Sukharev et al. (2020) Sukharev, I.; Shumovskaia, V.; Fedyanin, K.; Panov, M.; and Berestnev, D. 2020.  EWS-GCN: Edge weight-shared graph convolutional network for transactional banking data.  In _2020 IEEE International Conference on Data Mining (ICDM)_ , 1268–1273. IEEE.
  * Szegedy et al. (2014) Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I.; and Fergus, R. 2014.  Intriguing properties of neural networks.  In _2nd International Conference on Learning Representations, ICLR 2014_.
  * Tabassi et al. (2019) Tabassi, E.; Burns, K. J.; Hadjimichael, M.; Molina-Markham, A. D.; and Sexton, J. T. 2019.  A taxonomy and terminology of adversarial machine learning.  _NIST IR_ , 2019: 1–29.
  * Tripathi and Pavaskar (2012) Tripathi, K. K.; and Pavaskar, M. A. 2012.  Survey on credit card fraud detection methods.  _International Journal of Emerging Technology and Advanced Engineering_ , 2(11): 721–726.
  * Vos and Verwer (2022) Vos, D.; and Verwer, S. 2022.  Robust optimal classification trees against adversarial examples.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 8520–8528.
  * Xu et al. (2020) Xu, H.; Ma, Y.; Liu, H.-C.; Deb, D.; Liu, H.; Tang, J.-L.; and Jain, A. K. 2020\.  Adversarial attacks and defenses in images, graphs and text: A review.  _International Journal of Automation and Computing_ , 17: 151–178.
  * Zhuzhel et al. (2023) Zhuzhel, V.; Grabar, V.; Boeva, G.; Zabolotnyi, A.; Stepikin, A.; Zholobov, V.; Ivanova, M.; Orlov, M.; Kireev, I.; Burnaev, E.; et al. 2023.  Continuous-time convolutions model of event sequences.  _arXiv preprint arXiv:2302.06247_.

aaa

Figure 8: Scheme of the attacked GRU model Figure 9: Difference between
private and public scores for the attack track of the competition Figure 10:
Difference between private and public scores for the defense track of the
competition

##  Appendix A Appendix

In appendix, we provide additional technical details on used methods, deeper
analysis of the competition results, and additional experiments for considered
attacks and defenses.

##  Appendix B Technical details of the best performing methods

##### Training of a model for the attack

In order to train the model for the attack, we utilize transactions from
$50000$ customers. On top of preprocessed data, we train a GRU neural network
suitable for financial problems and event sequences in general (Shen et al.
2018; Mei and Eisner 2017) and for processing financial transactions (Babaev
et al. 2019). The scheme of the model that includes preprocessing is available
in Figure 8. After the training, we obtained a model with a $0.7$ ROC AUC
value, which is typical for the considered target. Below we provide additional
details on the training process.

For each transaction, we obtain embeddings, where each feature has a separate
embedding vector. These embeddings are concatenated, thereby representing each
transaction as a single vector. A sample for each customer consists of $300$
transactions, with the output is the predicted default class probability.

Preprocessing includes feature generation and transformation. Each transaction
was enriched by adding categorical temporal features such as the hour, day,
day of the week, and month. The transaction amount was binned into 100
quantiles, transforming all features into categorical variables.

We employ the AdamW optimizer (Loshchilov and Hutter 2018) with a learning
rate set at $3e-4$, using the binary cross entropy as the loss function. The
training adopts two regularization techniques: a spatial dropout with a
probability of $0.5$ before the GRU layer and a dropout layer with the same
probability before the linear layer.

#### The best attack

The best attack approach is a variation of SamplingFool (Fursov et al. 2021)
that showed the best results for financial transaction data. In particular,
the attack imitates a random search over a discrete space of sequences of
transactions:

  1. 1.

At each iteration of an attack we generate $k$ candidates and rank them
according to a model $\hat{f}(\mathbf{x})$.

  2. 2.

We select top $k_{0}$ changes and proceed to the next iteration.

As the number of iterations is equal the admissible number of changes, in the
end we have a set of changes that is admissible.

##### Hyperparameters

The attack uses $k=10000$ candidates at each step and $k_{0}=100$. As a model
$\hat{f}(\mathbf{x})$ we use a given GRU-based model or a set of gradient
boosting models. Usage of a given model or an ensemble model were selected at
random with probability $0.5$ to make possible attacks on a diverse models.

The ensemble for the imitation of the score of true model is $100$ of gradient
boosting models that use MCC codes and amount (so, they are features an
attacker can affect). We use a CatBoost gradient boosting implementation. The
derivative features are common set of aggregates for financial transactions.
To diversify the models and improve their quality, we learn each model using a
subset of generated aggregates. Each separate model was significantly worse,
than the baseline model, but in total the quality of models doesn’t affect the
quality of attacks.

#### The best defence

We note, that permutation of transactions doesn’t affect the model score. So,
to make the model more robust we can perform a fixed number of permutations
and get predictions of the initial model. Averaging of these predictions gives
the most robust option. So, the defense has two components: a permutation
algorithm and a model used to evaluated permuted sequences of transactions.

As the model the defense use a distilled model from the base one. It is also
an ensemble that was learned to distill the big model using only a subset of
given features. It was also a gradient boosting model.

##  Appendix C Additional results

### Analysis of competitions results

Figure 9 presents a comparison of public and private scores for participants
during the attack stage of the competition. Also, due to overfitting, the
private scores are almost always lower than the public score, making the
attack less powerful. We suspect that a single point with the reverse effect
is a result of purposeful masking of the public score.

Figure 10 presents a comparison of public and private scores for participants
during the defence stage of the competition. The results suggest significant
differences between public and private scores with later not being available
to participant during the competition suggesting another type of overfitting.

### Analysis of competitions tournament

Figures 11 and 12 present performance of all attacks against all defenses. One
can see in more details difference in performance between top-10 attacks and
defenses.

Figure 11: Performance of attacks during the black box tournament stage. Each
row represents a single attack. Each dot represents a single attacked model.
Figure 12: Performance of the defended models during the black box tournament
stage. Each row represents a single defended model. Each dot represents a
single performed attack.

### Sensitivity and ablation studies

We present, how quality of the attack is affected by the difference between
model architectures and number of admissible substitutions of transactions. In
addition, we vary the parameter $k$ — the number of admissible changes. Also,
we consider an alternative that generates more realistic and concealed
substitutions. It allows changes only of less prominent MCCs and uses more
realistic amounts for MCCs. We use the prefix gen to mark such approaches. The
results are in Figure 13.

We see continuous trend on decreasing of model performance, as we allow an
attack to change more tokens in a sequence. If an attacked surrogate model and
a true model are close to each other, the difference in performance in drastic
reaching $0.4$ ROC AUC for $10$ possible substitutions. If models are close to
each other e.g. a single Boosting model and a Boosting mix ensemble, the
attack also works, but the results are not so impressive. Finally, if we use
different architectures for different models, we observe performance drop
close to that related to random changes. With respect to the parameter $k$, we
observe, that generation of more possible options to select from lead to
stronger attacks even in the case of significant differences between true and
attacked models.

Figure 13: ROC AUC versus the number of changes for different types of True
and Surrogate (used for attack) models. X-axis is the number of possible
substitutions during an attack, Y-axis is the ROC AUC metric for an attacked
model after an attack.


# Spach Transformer: Spatial and Channel-wise Transformer Based on Local and
Global Self-attentions for PET Image Denoising

Se-In Jang, Tinsu Pan, Ye Li, Pedram Heidari, Junyu Chen, Quanzheng Li, and
Kuang Gong This work was supported by the National Institutes of Health under
grants R03EB030280, R21AG067422, R01AG078250 and P41EB022544. Corresponding
author: Kuang Gong (email: kgong@mgh.harvard.edu).S. Jang, Q. Li and K. Gong
are with Gordon Center for Medical Imaging and Center for Advanced Medical
Computing and Analysis , Massachusetts General Hospital and Harvard Medical
School, Boston, MA, USA T. Pan is with Department of Imaging Physics,
University of Texas MD Anderson Cancer Center, Houston, TX, USAY. Li is with
Center for Advanced Medical Computing and Analysis, Massachusetts General
Hospital and Harvard Medical School, Boston, MA, USAP. Heidari is with
Division of Nuclear Medicine and Molecular Imaging, Massachusetts General
Hospital and Harvard Medical School, Boston, MA, USAJ. Chen is with Department
of Electrical and Computer Engineering and Department of Radiology, Johns
Hopkins University, Baltimore, MD, USA

###### Abstract

Position emission tomography (PET) is widely used in clinics and research due
to its quantitative merits and high sensitivity, but suffers from low signal-
to-noise ratio (SNR). Recently convolutional neural networks (CNNs) have been
widely used to improve PET image quality. Though successful and efficient in
local feature extraction, CNN cannot capture long-range dependencies well due
to its limited receptive field. Global multi-head self-attention (MSA) is a
popular approach to capture long-range information. However, the calculation
of global MSA for 3D images has high computational costs. In this work, we
proposed an efficient spatial and channel-wise encoder-decoder transformer,
Spach Transformer, that can leverage spatial and channel information based on
local and global MSAs. Experiments based on datasets of different PET tracers,
i.e., 18F-FDG, 18F-ACBC, 18F-DCFPyL, and 68Ga-DOTATATE, were conducted to
evaluate the proposed framework. Quantitative results show that the proposed
Spach Transformer can achieve better performance than other reference methods.

###### Index Terms:

Positron Emission Tomography, Image Denoising, Transformer, Self-attention

##  I Introduction

Positron emission tomography (PET) is an imaging modality widely used in
oncology, neurology, and cardiology studies [1, 2, 3, 4, 5, 6]. It can observe
molecular-level activities in vivo through the injection of specifically
designed radioactive tracers. Due to various physical degradation factors and
limited counts received, the signal-to-noise ratio (SNR) of PET is inferior to
other imaging modalities, which comprises its clinical values in diagnosis,
prognosis, staging and treatment monitoring. Further enhancing PET image
quality is essential to improve its diagnosis and treatment-monitoring
accuracy.

For the past decades, various post-processing methods have been investigated
to further improve PET image quality. Gaussian filtering is widely used in
clinical scanners to reduce the noise but it can also smooth out image
structures. Non-local mean [7], wavelet [8], highly constrained back-
projection processing [9, 10], and guided filtering [11] have been proposed to
further preserve image details based on pre-defined filters or similarity
calculation. Recently, the convolutional neural network (CNN)-based approaches
have become a dominant trend in various PET research topics such as image
denoising [12, 13, 14, 15, 16, 17, 18, 19, 20, 21] and image reconstruction
[22, 23, 24, 25, 26, 27], due to their better performance than other state-of-
the-art methods. The CNN-based Unet architecture [28] is the most widely used
network architecture currently. Performance of CNN for PET image denoising can
be further improved by different loss-function designs, e.g., a perceptual
loss [13, 29, 30] formed by a pre-trained network or a learned discriminative
loss from the additional discriminator network [31, 29, 32, 33, 34, 30, 35,
36].

One pitfall of CNN is that it specifically focuses on local spatial
information and thus has a limited receptive field. The multi-head self-
attention (MSA)-based structure, i.e., transformer, has the ability to capture
long-range information [37, 38, 39]. Based on the aggregation of tokens,
global MSAs were first developed for natural language processing [37]. Vision
Transformer (ViT) was then proposed where global MSAs were successfully
applied to sequences of spatial image patches for image classification [38].
Due to global MSA’s quadratically growing computational complexity along the
spatial dimension, Swin Transformer [40] was proposed to efficiently calculate
local MSAs using shifted windows that offered linear computational complexity.
Swin Transformer calculated local MSAs on multiple stages and achieved better
classification performance than global MSAs. However, local MSAs of Swin
Transformer is still limited regarding the receptive field of MSA. Restormer
[41] was recently proposed to efficiently compute global MSAs along the
channel dimension with linear computational complexity for image restoration
tasks. Regarding medical imaging, various transformer-based architectures were
investigated for medical image segmentation [42, 43, 44, 45, 46, 47]. For PET
image denoising, transformer-based architectures have not yet been fully
investigated.

Figure 1: A brief overview of Spach Transformer for PET image denoising.

Figure 2: Diagram of the proposed Spach Transformer for PET image denoising.
Details of the channel-wise transformer block and the spatial-wise transformer
block are given in Fig. 3 and Fig. 4, respectively.

In this work, we proposed a spatial and channel-wise encoder-decoder
transformer, denoted as Spach Transformer, that can leverage spatial and
channel information from oncological PET images. The proposed Spach
Transformer includes spatial-wise and channel-wise transformer blocks, which
can efficiently perform image denoising while also preserving image details,
e.g., small lesion uptakes. The spatial-wise block is based on local attention
and the channel-wise block is based on global attention. As the channel-wise
transformer block may ignore the spatial information, a gated-conv feed-
forward network (GCFN) block was further designed to incorporate spatial
attributes between layers in the channel-wise transformer block. A brief
overview of the proposed Spach Transformer is presented in Fig. 1. It encoded
the input and built two deep latent features in spatial and channel
directions. The decoder further took the two latent features concatenated as
input to generate the final denoised PET image based on skip connections. To
evaluate the effectiveness of the proposed Spach Transformer for PET image
denoising, clinical 18F-FDG, 18F-ACBC, 18F-DCFPyL, and 68Ga-DOTATATE datasets
were utilized to quantitatively compare the performance of different CNN and
transformer-based methods.

The main contributions of this work include : (1) proposing a novel spatial
and channel-wise encoder-decoder transformer integrating local and global
attentions for PET image denoising and tumor-uptake preserving; (2) designing
a GCFN block to incorporate spatial attributes in the channel-wise transformer
block to boost the performance; (3) evaluations using clinical datasets based
on different PET tracers. This paper is organized as follows. Section 2
presents the related works and the proposed Spach Transformer in detail.
Experiments, results and discussions are shown in Sections 3, 4 and 5,
respectively. Finally, conclusions are drawn in Section 5.

##  II Method

Fig. 2 shows the detailed diagram of the proposed Spach Transformer. The
encoder part of Spach Transformer consists of two levels of spatial-wise
transformer blocks and four levels of channel-wise transformer blocks. The
decoder part of Spach Transformer includes four levels of channel-wise
transformer blocks. Below we will describe major elements of the Spach
Transformer (Sec. II-A) as well as details of the spatial-wise transformer
block (Sec. II-B) and the channel-wise transformer block. (Sec. II-C).

###  II-A Spach Transformer: Spatial and Channel-wise Transformer

Given a low-dose PET image ${\bf X}\in\mathbb{R}^{H\times W\times D\times 1}$,
Spach Transformer first obtained low-level feature embeddings ${\bf
F}_{0}\in\mathbb{R}^{H\times W\times D\times C}$ based on a $3\times 3\times
3$ convolution, where $H$, $W$, and $D$ indicate the voxel dimension and $C$
is the number of channels. The four layers of the channel-wise encoder
transformer blocks further processed the embedding ${\bf F}_{0}$ to extract
deep latent features ${\bf F}_{l_{c}}$.

Addtionally, following [38, 40], a patch partitioning module produced non-
overlapping patches $\mathbf{x}_{i}\in\mathbb{R}^{P^{3}}$ from the low-dose
PET image ${\bf X}$, where $P$ is the patch size and was set to 4 in our
implementation. A linear embedding layer was added to project these non-
overlapping voxels to an arbitrary dimension as follows:

$\mathbf{z}_{0}=[\mathbf{x}_{0}\mathbf{E};\mathbf{x}_{1}\mathbf{E};\dotsc;\mathbf{x}_{N_{p}}\mathbf{E}]\in\mathbb{R}^{N_{P}\times
d},$ (1)

where $\mathbf{E}\in\mathbb{R}^{P^{3}\times d}$ is a projection matrix, and
$\displaystyle N_{P}=\tfrac{H}{P}\times\tfrac{W}{P}\times\tfrac{D}{P}$ is the
number of patches. The spatial-wise encoder transformer blocks and the patch-
merging layer used the tokens $\mathbf{z}_{0}$ to extract deep latent features
${\bf F}_{l_{s}}$. Role of the patch-merging layer is to reduce the spatial
size by half along each dimension.

The encoder part described above gradually reduced the spatial size while
expanding the channel capacity. The final latent features were generated by
concatenating the latent features from the two transformer paths as
$\mathbf{F}_{l}=\left[\mathbf{F}_{l_{c}},\mathbf{F}_{l_{s}}\right]$. The final
latent features $\mathbf{F}_{l}$ were further supplied to the decoder path to
generate the high-quality PET image $\hat{\mathbf{X}}$. Following [41], a
$1\times 1\times 1$ convolution and a concatenation operation was utilized to
reduce the channel size by half at each decoder layer, followed by a channel-
wise transformer block to recover the spatial information. The deep features
$\mathbf{F}_{d}$ were further refined to obtain robust features
$\mathbf{F}_{r}$ at the PET image resolution scale. Finally, a $3\times
3\times 3$ convolution layer was applied on $\mathbf{F}_{r}$ to produce the
residual image $\mathbf{R}$, and the final denoised PET image was generated as
$\hat{\mathbf{X}}=\mathbf{R}+\mathbf{X}$.

###  II-B Spatial-wise Transformer Block

Swin Transformer [40] proposed to use window-based and shift window-based MSA
modules to replace the standard MSA modules. Fig. 3(a) shows the diagram of
the transformer block, which is based on the following operations:

$\displaystyle\hat{\mathbf{z}}_{l}$
$\displaystyle=MSA_{W}\left(LN(\mathbf{z}_{l-1})\right)+\mathbf{z}_{l-1},$ (2)
$\displaystyle\mathbf{z}_{l}$
$\displaystyle=MLP\left(LN(\hat{\mathbf{z}}_{l})\right)+\hat{\mathbf{z}}_{l},$
$\displaystyle\hat{\mathbf{z}}_{l+1}$
$\displaystyle=MSA_{SW}\left(LN(\mathbf{z}_{l})\right)+\mathbf{z}_{l},$
$\displaystyle\mathbf{z}_{l+1}$
$\displaystyle=MLP\left(LN(\hat{\mathbf{z}}_{l+1})\right)+\hat{\mathbf{z}}_{l+1},$

where $MSA_{W}$ and $MSA_{SW}$ are regular window-based MSA (W-MSA) and
shifted window-based MSA (SW-MSA), respectively, $MLP$ is the multi-layer
perceptron (MLP) module with the Gaussian error linear unit (GELU) [48] as the
activation layer, and $LN$ is the layer normalization (LN) module [49] which
was inserted before the W-MAS, SW-MSA and MLP modules. Residual connection was
used after each module. Fig. 3(b) shows the window-based self-attention
diagram. The self-attention was calculated as [40] :

$\text{Attention}(\mathbf{Q}_{s},\mathbf{K}_{s},\mathbf{V}_{s})=\text{Softmax}\left(\mathbf{Q}_{s}\mathbf{K}_{s}^{T}/\sqrt{d}+\mathbf{B}\right)\mathbf{V}_{s},$
(3)

where $\mathbf{B}\in\mathbb{R}^{M^{3}\times M^{3}}$ is the relative position
of tokens in each window, $d$ is the query/key dimension. and $M^{3}$ is the
number of patches in the 3D image window. The query $\mathbf{Q}_{s}$, key
$\mathbf{K}_{s}$, and value $\mathbf{V}_{s}$ matrices were computed as:

$\mathbf{Q}_{s}=W_{l}^{Q}\mathbf{Y},\mathbf{K}_{s}=W_{l}^{K}\mathbf{Y},\mathbf{V}_{s}=W_{l}^{V}\mathbf{Y},$
(4)

where $W_{l}^{Q}\in\mathbb{R}^{M^{3}\times M^{3}}$,
$W_{l}^{K}\in\mathbb{R}^{M^{3}\times M^{3}}$ , and
$W_{l}^{V}\in\mathbb{R}^{M^{3}\times M^{3}}$ are the linear projection
matrices, and $\mathbf{Y}\in\mathbb{R}^{M^{3}\times d}$ is the input after the
LN layer.

###  II-C Channel-wise Transformer Block

The Restormer work [41] designed the multi-Dconv head transposed attention
(MDTA) modules to replace the standard MSA modules, which computed self-
attention along the channel dimension instead of the spatial dimension. This
can allow a global attention calculation with linearly computational
complexity. In Restormer, each transformer block included a MDTA and a gated-
Dconv feed-forward network (GDFN) module. Though Restormer showed competing
performance in natural image enhancement, based on our experiments, we found
that this configuration has the pitfall of losing local spatial information.
Apart from transferring local spatial information in parallel using the
spatial-wise transformer blocks as described in Sec. II-B, in the proposed
channel-wise transformer block (see Fig. 4(a)), we further designed a gated-
conv feed-forward network (GCFN) to better capture local spatial information
and thus improve the lesion-uptake recovery. Below we will introduce the
details of MDTA, GDFN and GCFN.

####  II-C1 MDTA

Fig. 4(b) shows the diagram of the MDTA module. Given an input feature map
${\bf F}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{D}\times\hat{C}}$, a
layer normalization module was first applied to obtain
$\mathbf{Y}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{D}\times\hat{C}}$.
The query
$\mathbf{Q}_{c}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{D}\times\hat{C}}$,
key
$\mathbf{K}_{c}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{D}\times\hat{C}}$,
and value
$\mathbf{V}_{c}\in\mathbb{R}^{\hat{H}\times\hat{W}\times\hat{D}\times\hat{C}}$
matrices were then obtained after a $1\times 1\times 1$ pixel-wise convolution
operation (encoding channel-wise context) and a $3\times 3\times 3$ channel-
wise convolution operation (aggregating pixel-wise cross-channel context). By
reshaping operations,
$\hat{\mathbf{Q}}_{c}\in\mathbb{R}^{\hat{H}\hat{W}\hat{D}\times\hat{C}}$,
$\hat{\mathbf{K}}_{c}\in\mathbb{R}^{\hat{C}\times\hat{H}\hat{W}\hat{D}}$, and
$\hat{\mathbf{V}}_{c}\in\mathbb{R}^{\hat{H}\hat{W}\hat{D}\times\hat{C}}$ were
obtained from $\mathbf{Q}_{c}$, $\mathbf{K}_{c}$, and $\mathbf{V}_{c}$,
respectively, and were used to generate a transposed-attention map
$\mathbf{A}\in\mathbb{R}^{\hat{C}\times\hat{C}}$ instead of the massive
spatial attention map of size
$\hat{H}\hat{W}\hat{D}\times\hat{H}\hat{W}\hat{D}$ [37, 38]. The output
feature of the MDTA module $\hat{\mathbf{F}}$ can be expressed as

$\displaystyle\hat{\mathbf{F}}=W_{p}\text{Attention}(\hat{\mathbf{Q}}_{c},\hat{\mathbf{K}}_{c},\hat{\mathbf{V}}_{c})+\mathbf{F},$
(5)
$\displaystyle\text{Attention}(\hat{\mathbf{Q}}_{c},\hat{\mathbf{K}}_{c},\hat{\mathbf{V}}_{c})=\hat{\mathbf{V}}_{c}\cdot\text{Softmax}({\hat{\mathbf{Q}}_{c}\cdot}\hat{\mathbf{K}}_{c}/\alpha),$

where $\alpha$ is a learnable scaling parameter to control the magnitude of
the dot product output ${\hat{\mathbf{Q}}_{c}\cdot}\hat{\mathbf{K}}_{c}$, and
$W_{p}$ is a linear projection matrix. Note that the output of the self-
attention operation was reshaped to have the same size as the input ${\bf F}$.

(a) The spatial-wise transformer block.

(b) The window-based multi-head self-attention.

Figure 3: Diagrams of (a) the spatial-wise transformer block and (b) the
window-based attention calculation.

(a) The proposed channel-wise transformer block.

(b) The multi-dconv head transposed attention (MDTA).

(c) The gated-dconv feed-forward network (GDFN).

(d) The gated-conv feed-forward network (GCFN).

Figure 4: Diagrams of (a) the proposed channel-wise transformer block, (b) the MDTA module, (c) the GDFN module and (d) the GCFN module. TABLE I: According to channel number change $C$, an ablation study of different model parameters of Spach Transformer (GCFN). | C |  Model Parameters | PSNR | SSIM
---|---|---|---|---
| Avg | Std | Avg | Std
Spach Transformer | 6 | 7,034,951 | 54.9753 | 3.6359 | 0.9367 | 0.0147
| 12 | 19,218,323 | 55.0840 | 3.6092 | 0.9403 | 0.0128
| 24 | 59,350,043 | 55.0911 | 3.6103 | 0.9358 | 0.0147
| 36 | 120,501,731 | 55.1003 | 3.6193 | 0.9379 | 0.0137
| 48 | 202,673,387 | 55.1443 | 3.6036 | 0.9368 | 0.0138

(a) The predicted images and the distance images of the whole body.

(b) The enlarged images where the blue rectangles are located in (a).

Figure 5: Coronal and zoomed-in views of the denoised PET images of Spach
Transformer according to different numbers of channels $C$ based on an
18F-DCFPyL dataset

(a) The predicted images and the distance images of the whole body.

(b) The enlarged images where the blue rectangles are located in (a).

Figure 6: Coronal and zoomed-in views of the denoised PET results of different
methods based on an 18F-DCFPyL dataset

(a) The predicted images and the distance images of the whole body.

(b) The enlarged images where the blue rectangles are located in (a).

Figure 7: Coronal and zoomed-in views of the denoised PET results of the
state-of-the-art methods based on an 18F-FDG dataset

(a) The predicted images and the distance images of the whole body.

(b) The enlarged images where the blue rectangles are located in (a).

Figure 8: Coronal and zoomed-in views of the denoised PET results of different
methods based on an 18F-ACBC dataset

(a) The predicted images and the distance images of the whole body.

(b) The enlarged images where the blue rectangles are located in (a).

Figure 9: Coronal and zoomed-in views of the denoised PET results of different
methods based on a 68Ga DOTATATE dataset

####  II-C2 GDFN

A gating mechanism [50] can help selectively pass useful information to the
next layer in the network architecture. Due to this, the gating network could
suppress less informative features. In [41], the gating mechanism was utilized
in the GDFN module (see Fig. 4(c)), which was calculated as

$\displaystyle\hat{\mathbf{F}}=W_{p}^{0}\text{Gating}(\mathbf{F})+\mathbf{F},$
(6)
$\displaystyle\text{Gating}(\mathbf{F})=\phi\left(W_{c}^{1}W_{p}^{1}{LN}(\mathbf{F})\right)\odot
W_{c}^{2}W_{p}^{2}{LN}(\mathbf{F}).$

Here ${\mathbf{F}}$ and $\hat{\mathbf{F}}$ are the input and output features,
$\odot$ is an element-wise multiplication, $\phi$ is the GELU non-linearity
activation layer, and $LN$ indicates the LN layer. $W_{p}^{0}$, $W_{p}^{1}$,
and $W_{p}^{2}$ indicate the linear projection matrices. $W_{c}^{1}$ and
$W_{c}^{1}$ represent $3\times 3\times 3$ channel-wise convolutions.

####  II-C3 GCFN

Since the above MDTA and GDFN modules entirely focus on utilizing channel
information for image denoising, it may lose spatial information. To preserve
spatial attributes between layers, a gated-conv feed-forward network (GCFN)
module (see Fig. 4(d)) was added to the channel-wise transformer block. The
only difference between GCFN and GDFN are that the $3\times 3\times 3$
channel-wise convolutions used in GDFN ($W_{c}^{1}$ and $W_{c}^{1}$) were
replaced by $3\times 3\times 3$ spatial-wise convolutions ($W_{s}^{1}$ and
$W_{s}^{1}$). We hypothesized that the spatial-wise convolutions could better
utilize spatial information and thus further help recovery lesion uptakes.

##  III Experiments

###  III-A Datasets

To evaluate performance of the proposed Spach Transformer, clinical whole-body
18F-FDG, 18F-ACBC, 18F-DCFPyL, and 68Ga-DOTATATE datasets acquired from the GE
DMI PET/CT scanner were utilized. These tracers have unique distributions in
the organs and are currently the most widely used tracers in oncology PET
studies. For each dataset, 1/4 of the events were extracted from the listmode
data to generate the 1/4 low-dose data, which was reconstructed and supplied
as the network input. The corresponding normal-dose image was adopted as the
training label. Both the low-dose and normal-dose PET images were
reconstructed based on the ordered subset expectation maximization (OSEM)
algorithm (3 iterations and 17 subsets) with point spread function (PSF) and
time of flight (TOF) modeling. For the training set, 30 18F-FDG and 30
18F-ACBC datasets were included. For the validation set, 4 18F-FDG datasets
were used. For the testing set, 10 18F-FDG, 10 18F-ACBC, 10 18F-DCFPyL, and 18
68Ga-DOTATATE datasets were employed. Note that 18F-DCFPyL and 68Ga-DOTATATE
datasets were not included in the training set, but in the testing set. This
is to evaluate the robustness of the deep learning structures to datasets of
new tracers unseen during the training phase. The whole-body PET images had a
dimension of $256\times 256\times 345$ and a voxel size of $2.73\times
2.73\times 2.8$ mm3. During network training, the input image was divided into
$96\times 96\times 96$ cubes due to the GPU memory limit.

###  III-B Implementation and quantification

The Unet [28], Swin Transformer-based structure [47], and Restormer [41] were
adopted as the reference methods. All the architectures utilized the
Charbonnier loss [51] as the training loss function. For Spach Transformer,
the channel size of the transformer blocks can influence the network
parameters a lot. We have investigated the performance of the Spach
Transformer with different channel sizes. During network training, the AdamW
optimizer was used and 300 epochs were run with the initial learning rate of
1e-5 gradually reduced to 1e-8 using cosine annealing [52]. The proposed
network was implemented on the Nvidia Tesla V100 GPU.

As for quantification, the peak signal-to-noise ratio (PSNR) and structural
similarity index (SSIM) were utilized to quantify the global performance of
different methods. For oncological PET images, preserving tumor uptake is
essential for disease diagnosis and monitoring. To evaluate the performance of
different methods regarding tumor-uptake recovery, the contrast-to-noise (CNR)
was calculated at the tumor region as

$\text{CNR}=(V_{\text{tumor}}-V_{\text{ref}})/{\sigma}_{\text{ref}}.$ (7)

Here $V_{\text{tumor}}$ is the mean value of the tumor region of interest
(ROI). $V_{\text{ref}}$ and $\sigma_{\text{ref}}$ are the reference ROIs’ mean
value and standard deviation, respectively. The reference ROIs were selected
from uniform regions inside the liver region.

##  IV Results

###  IV-A Effect of model parameters

We want to first investigate the performance of Spach Transformer with
different network parameters. The channel number used in the transformer
architecture has a big impact on the model size and was varied to investigate
the network performance with different network parameters. Table I shows the
averaged PSNR and SSIM values for different model parameters of the Spach
Transformer based on varying the channel size $C$ ($C=\\{6,12,24,36,48\\}$).
It can be observed that the best PSNR was achieved when $C=48$, while the best
SSIM was obtained when $C=12$. Fig. 5 shows the denoised PET images from one
18F-DCFPyL dataset with different channel numbers. To evaluate how close the
predicted images are to the ground truth (GT), the images of $L2$-norm
distance between the predicted and GT images were also generated. It can be
observed that when $C=6$, the denoised image is blurred and the uptake in the
zoomed-in tumor region cannot be fully recovered. When the channel number $C$
was increased to 12, the network can better recover the tumor uptake and also
has a better denoising performance. After further increasing the network
parameter, performance of the network on tumor-uptake recovery increased a
little and became stable. Based on the PSNR and SSIM results as well as the
image appearance, the channel size $C$ was set to 12 in all our studies. The
comparison methods were all set to have a similar number of network parameters
as the Spach Transformer with $C=12$.

TABLE II: Quantitative comparison of PSNR with the state-of-the-art methods. Tracer | Unet | Swin Transformer | Restormer |  |  Spach Transformer
---
w/o GCFN
Spach Transformer
| Mean | Std | Mean | Std | Mean | Std | Mean | Std | Mean | Std
18F-DCFPyL | 53.9726 | 5.7870 | 53.5862 | 5.7802 | 53.8871 | 5.7728 | 53.9818 | 5.7754 | 54.0927 | 5.7879
18F-FDG | 52.6028 | 3.1194 | 52.1079 | 3.2607 | 52.5813 | 3.0841 | 52.6135 | 3.0513 | 52.8678 | 2.9585
18F-ACBC | 58.3996 | 7.0715 | 57.6852 | 7.2052 | 58.4845 | 7.0278 | 58.6009 | 7.0026 | 58.7500 | 6.9883
68Ga DOTATATE | 54.0784 | 3.7722 | 53.3199 | 3.8628 | 54.4369 | 3.7322 | 54.4289 | 3.7438 | 54.8294 | 3.6679
Avg | 54.6492 | 5.3646 | 54.0323 | 5.4194 | 54.7790 | 5.3398 | 54.8267 | 5.3439 | 55.0840 | 5.2985
Model Parameters | 21,781,520 | 20,631,864 | 21,156,805 | 21,022,921 | 19,218,323
TABLE III: The SSIM values of different methods on 18F-DCFPyL, 18F-FDG, 18F-ACBC and 68Ga DOTATATE datasets. Tracer | Unet | Swin Transformer | Restormer |  |  Spach Transformer
---
w/o GCFN
Spach Transformer
| Mean | Std | Mean | Std | Mean | Std | Mean | Std | Mean | Std
18F-DCFPyL | 0.9234 | 0.0076 | 0.9125 | 0.0080 | 0.9370 | 0.0078 | 0.9369 | 0.0079 | 0.9439 | 0.0069
18F-FDG | 0.9364 | 0.0063 | 0.9282 | 0.0079 | 0.9472 | 0.0075 | 0.9477 | 0.0076 | 0.9539 | 0.0074
18F-ACBC | 0.8894 | 0.0059 | 0.8759 | 0.0052 | 0.8955 | 0.0061 | 0.8975 | 0.0059 | 0.9138 | 0.0048
68Ga-DOTATATE | 0.9121 | 0.0168 | 0.9166 | 0.0122 | 0.9432 | 0.0119 | 0.9417 | 0.0129 | 0.9458 | 0.0127
Avg | 0.9148 | 0.0196 | 0.9097 | 0.0204 | 0.9328 | 0.0215 | 0.9327 | 0.0208 | 0.9404 | 0.0169

(a) 18F-DCFPyL

(b) 18F-FDG

(c) 18F-ACBC

(d) 68Ga DOTATATE

Figure 10: The CNR values of different methods based on (a) 18F-DCFPyL, (b)
18F-FDG, (c) 18F-ACBC and (d) 68Ga DOTATATE datasets.

###  IV-B Comparisons with reference methods

Fig. 6, 7, 8, and 9 present the coronal view of the results generated by the
Spach Transformer and the reference methods based on one 18F-DCFPyL, one
18F-FDG, one 18F-ACBC, and one 68Ga DOTATATE dataset. From Fig. 6(a), 7(a),
8(a), and 9(a), it can be observed that Restormer has better denoising
performance than that of the Unet and Swin Transformer. It can also be
observed that the Unet and Swin Transformer can achieve relatively higher
uptake values than that of the Restormer in the tumor regions. This might be
because the Unet and Swin Transformer focus more on spatial information,
whereas the Restormer is highly dependent on the channel information. As the
Spach Transformer can utilize both spatial and channel information together,
it can better preserve the tumor details while also achieving competitive
denoising performance than other reference methods.

Table II and III show that the Spach Transformer achieved average PSNR of
55.0840 and aveerage SSIM of 0.9403, better than the Unet, Swin Transformer,
Restormer, and Spach Transformer without GCFN. Compared to Restormer, Spach
Transformer achieved a PSNR gain of 0.305 and an SSIM gain of 0.008 because of
the extra spatial information utilized. Compared to Swin Transformer, Spach
Transformer achieved a PSNR gain of 1.052 and an SSIM gain of 0.031 because of
the global channel information leveraged.

To examine the local performance of different methods at the tumor regions, we
have further quantified the tumor CNR for datasets with confirmed tumors. Fig.
10 shows the CNR results of all the methods for datasets with different
tracers. As observed from Fig. 6, 7, 8, and 9, the Restormer has worse
performance regarding tumor-uptake preserving, which is confirmed by the CNR
plot in Fig. 10. The Spach Transformer obtained better CNR results in the
tumor regions than other reference methods because of the combination of
spatial and channel information.

##  V Discussion

For PET image denoising, one challenge is to reduce the noise while also
preserving the tumor uptake. The latest deep learning architectures (e.g., the
Swin Transformer and the Restormer) have focused on utilizing spatial or
channel information in an efficient way to improve the specific image-
processing task. In this work, we proposed a novel spatial and channel-wise
transformer architecture, Spach Transformer, aiming to suppress the noise and
at the same time preserving the image details. Oncology PET datasets from
different tracers were utilized to evaluate the performance of different
network structures.

From the evaluations, we observed that the spatial information-oriented
network architectures (e.g., the Unet and Swin Transformer) can highlight the
tumor region well but the results are still noisy. In contrast, the channel
information-oriented network architecture (e.g., the Restormer) can achieve
better denoising performance but failed to preserve the tumor uptakes, which
can be observed through the CNR plot and the zoomed-in tumor regions of the
coronal views. The SSIM, PSNR and CNR quantification show that the proposed
Spach Transformer can achieve the best performance, demonstrating the benefits
of combining both spatial and channel information.

Another thing to note is that all the network structures were trained using
18F-FDG and 18F-ACBC datasets, while the testing datasets include two
additional tracers, i.e., 18F-DCFPyL and 68Ga DOTATATE, which were unseen
during the training phase. 18F-DCFPyL is a newly FDA approved prostate-
specific membrane antigen (PSMA) tracer for prostate cancer diagnosis, which
does not have many cases available when we began this work. 18Ga-DOTATATE is
for neuroendocrine tumor diagnosis and has worse image quality as compared to
the 18F-based tracer. Evaluations on datasets from these two tracers can help
evaluate the robustness of the network to unseen data distributions and
explore the possibility of deep learning-based cross-tracer applications.
Results show that all the networks have good performance on those unseen data
distributions and the proposed Spach Transformer has the best performance.
Apart from cross-tracer testing, further evaluating the performance of the
networks on datasets from different scanners is one of our future tasks.

##  VI Conclusion

In this work, we proposed a spatial and channel-wise transformer that utilized
the spatial and channel information for PET image denoising. The proposed
Spach Transformer achieved better performances in terms of CNR, PSNR, and SSIM
on 18F-FDG, 18F-ACBC, 18F-DCFPyL, and 68Ga-DOTATATE datasets compared to other
reference methods. Our future work will focus on further quantitative
evaluations based more clinical datasets.

## References

  * [1] W. H. Sweet, “The uses of nuclear disintegration in the diagnosis and treatment of brain tumor,” _New England Journal of Medicine_ , vol. 245, no. 23, pp. 875–878, 1951.
  * [2] S. R. Cherry, T. Jones, J. S. Karp _et al._ , “Total-body pet: maximizing sensitivity to create new opportunities for clinical research and patient care,” _Journal of Nuclear Medicine_ , vol. 59, no. 1, pp. 3–12, 2018.
  * [3] J. W. Fletcher, B. Djulbegovic, H. P. Soares _et al._ , “Recommendations on the use of 18f-fdg pet in oncology,” _Journal of Nuclear Medicine_ , vol. 49, no. 3, pp. 480–508, 2008.
  * [4] T. Beyer, D. W. Townsend, T. Brun _et al._ , “A combined pet/ct scanner for clinical oncology,” _Journal of nuclear medicine_ , vol. 41, no. 8, pp. 1369–1379, 2000.
  * [5] M. F. Di Carli, S. Dorbala, J. Meserve _et al._ , “Clinical myocardial perfusion pet/ct,” _Journal of Nuclear Medicine_ , vol. 48, no. 5, pp. 783–793, 2007.
  * [6] G. El Fakhri, S. Surti, C. M. Trott _et al._ , “Improvement in lesion detection with whole-body oncologic time-of-flight pet,” _Journal of Nuclear Medicine_ , vol. 52, no. 3, pp. 347–353, 2011.
  * [7] J. Dutta, R. M. Leahy, and Q. Li, “Non-local means denoising of dynamic pet images,” _PloS one_ , vol. 8, no. 12, p. e81390, 2013.
  * [8] N. Boussion, C. Cheze Le Rest, M. Hatt _et al._ , “Incorporation of wavelet-based denoising in iterative deconvolution for partial volume correction in whole-body pet imaging,” _European journal of nuclear medicine and molecular imaging_ , vol. 36, no. 7, pp. 1064–1075, 2009.
  * [9] B. T. Christian, N. T. Vandehey, J. M. Floberg _et al._ , “Dynamic pet denoising with hypr processing,” _Journal of Nuclear Medicine_ , vol. 51, no. 7, pp. 1147–1154, 2010.
  * [10] C. W. Bevington, J.-C. Cheng, and V. Sossi, “A 4-D iterative HYPR denoising operator improves PET image quality,” _IEEE Transactions on Radiation and Plasma Medical Sciences_ , vol. 6, no. 6, pp. 641–655, 2021.
  * [11] J. Yan, J. C.-S. Lim, and D. W. Townsend, “Mri-guided brain pet image filtering and partial volume correction,” _Physics in medicine & biology_, vol. 60, no. 3, p. 961, 2015.
  * [12] L. Xiang, Y. Qiao, D. Nie _et al._ , “Deep auto-context convolutional neural networks for standard-dose pet image estimation from low-dose pet/mri,” _Neurocomputing_ , vol. 267, pp. 406–416, 2017.
  * [13] K. Gong, J. Guan, C.-C. Liu _et al._ , “Pet image denoising using a deep neural network through fine tuning,” _IEEE Transactions on Radiation and Plasma Medical Sciences_ , vol. 3, no. 2, pp. 153–161, 2018.
  * [14] S. Kaplan and Y.-M. Zhu, “Full-dose pet image estimation from low-dose pet image using deep learning: a pilot study,” _Journal of digital imaging_ , vol. 32, no. 5, pp. 773–778, 2019.
  * [15] J. Cui, K. Gong, N. Guo _et al._ , “Pet image denoising using unsupervised deep learning,” _European journal of nuclear medicine and molecular imaging_ , vol. 46, no. 13, pp. 2780–2789, 2019.
  * [16] K. T. Chen, E. Gong, F. B. de Carvalho Macruz _et al._ , “Ultra–low-dose 18f-florbetaben amyloid pet imaging using deep learning with multi-contrast mri inputs,” _Radiology_ , vol. 290, no. 3, pp. 649–656, 2019.
  * [17] F. Hashimoto, H. Ohba, K. Ote _et al._ , “Dynamic pet image denoising using deep convolutional neural networks without prior training datasets,” _IEEE access_ , vol. 7, pp. 96 594–96 603, 2019.
  * [18] C. O. da Costa-Luis and A. J. Reader, “Micro-networks for robust mr-guided low count pet imaging,” _IEEE transactions on radiation and plasma medical sciences_ , vol. 5, no. 2, pp. 202–212, 2020.
  * [19] G. Schramm, D. Rigie, T. Vahle _et al._ , “Approximating anatomically-guided pet reconstruction in image space using a convolutional neural network,” _Neuroimage_ , vol. 224, p. 117399, 2021.
  * [20] A. Mehranian, S. D. Wollenweber, M. D. Walker _et al._ , “Image enhancement of whole-body oncology [18f]-fdg pet scans using deep neural networks to reduce noise,” _European journal of nuclear medicine and molecular imaging_ , vol. 49, no. 2, pp. 539–549, 2022.
  * [21] R. S. Daveau, I. Law, O. M. Henriksen _et al._ , “Deep learning based low-activity pet reconstruction of [11c] pib and [18f] fe-pe2i in neurodegenerative disorders,” _NeuroImage_ , vol. 259, p. 119412, 2022.
  * [22] K. Kim, D. Wu, K. Gong _et al._ , “Penalized pet reconstruction using deep learning prior and local linear fitting,” _IEEE transactions on medical imaging_ , vol. 37, no. 6, pp. 1478–1487, 2018.
  * [23] K. Gong, J. Guan, K. Kim _et al._ , “Iterative pet image reconstruction using convolutional neural network representation,” _IEEE transactions on medical imaging_ , vol. 38, no. 3, pp. 675–685, 2018.
  * [24] K. Gong, C. Catana, J. Qi _et al._ , “Pet image reconstruction using deep image prior,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 7, pp. 1655–1665, 2018.
  * [25] I. Häggström, C. R. Schmidtlein, G. Campanella _et al._ , “Deeppet: A deep encoder–decoder network for directly solving the pet image reconstruction inverse problem,” _Medical image analysis_ , vol. 54, pp. 253–262, 2019.
  * [26] H. Lim, I. Y. Chun, Y. K. Dewaraja _et al._ , “Improved low-count quantitative pet reconstruction with an iterative neural network,” _IEEE transactions on medical imaging_ , vol. 39, no. 11, pp. 3512–3522, 2020\.
  * [27] A. Mehranian and A. J. Reader, “Model-based deep learning pet image reconstruction using forward–backward splitting expectation–maximization,” _IEEE transactions on radiation and plasma medical sciences_ , vol. 5, no. 1, pp. 54–64, 2020.
  * [28] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [29] J. Ouyang, K. T. Chen, E. Gong _et al._ , “Ultra-low-dose pet reconstruction using generative adversarial network with feature matching and task-specific perceptual loss,” _Medical physics_ , vol. 46, no. 8, pp. 3555–3564, 2019.
  * [30] Z. Hu, H. Xue, Q. Zhang _et al._ , “Dpir-net: Direct pet image reconstruction based on the wasserstein generative adversarial network,” _IEEE Transactions on Radiation and Plasma Medical Sciences_ , vol. 5, no. 1, pp. 35–43, 2020.
  * [31] I. Goodfellow, J. Pouget-Abadie, M. Mirza _et al._ , “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [32] Y. Lei, X. Dong, T. Wang _et al._ , “Whole-body pet estimation from low count statistics using cycle-consistent generative adversarial networks,” _Physics in Medicine & Biology_, vol. 64, no. 21, p. 215017, 2019.
  * [33] L. Zhou, J. D. Schaefferkoetter, I. W. Tham _et al._ , “Supervised learning with cyclegan for low-dose fdg pet image denoising,” _Medical image analysis_ , vol. 65, p. 101770, 2020.
  * [34] T.-A. Song, S. R. Chowdhury, F. Yang _et al._ , “Pet image super-resolution using generative adversarial networks,” _Neural Networks_ , vol. 125, pp. 83–91, 2020.
  * [35] A. Sanaat, I. Shiri, H. Arabi _et al._ , “Deep learning-assisted ultra-fast/low-dose whole-body pet/ct imaging,” _European journal of nuclear medicine and molecular imaging_ , vol. 48, no. 8, pp. 2405–2415, 2021\.
  * [36] S. Xue, R. Guo, K. P. Bohn _et al._ , “A cross-scanner and cross-tracer deep learning method for the recovery of standard-dose imaging quality from low-dose pet,” _European journal of nuclear medicine and molecular imaging_ , vol. 49, no. 6, pp. 1843–1856, 2022.
  * [37] A. Vaswani, N. Shazeer, N. Parmar _et al._ , “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [38] A. Dosovitskiy, L. Beyer, A. Kolesnikov _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [39] K. Han, Y. Wang, H. Chen _et al._ , “A survey on vision transformer,” _IEEE transactions on pattern analysis and machine intelligence_ , 2022.
  * [40] Z. Liu, Y. Lin, Y. Cao _et al._ , “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [41] S. W. Zamir, A. Arora, S. Khan _et al._ , “Restormer: Efficient transformer for high-resolution image restoration,” _arXiv preprint arXiv:2111.09881_ , 2021.
  * [42] J. Chen, Y. Lu, Q. Yu _et al._ , “Transunet: Transformers make strong encoders for medical image segmentation,” _arXiv preprint arXiv:2102.04306_ , 2021.
  * [43] Y. Xie, J. Zhang, C. Shen _et al._ , “Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2021, pp. 171–180.
  * [44] W. Wang, C. Chen, M. Ding _et al._ , “Transbts: Multimodal brain tumor segmentation using transformer,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 109–119.
  * [45] Y. Zhang, H. Liu, and Q. Hu, “Transfuse: Fusing transformers and cnns for medical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2021, pp. 14–24.
  * [46] A. Hatamizadeh, Y. Tang, V. Nath _et al._ , “Unetr: Transformers for 3d medical image segmentation,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 574–584.
  * [47] J. Chen, Y. Du, Y. He _et al._ , “Transmorph: Transformer for unsupervised medical image registration,” _arXiv preprint arXiv:2111.10480_ , 2021.
  * [48] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” _arXiv preprint arXiv:1606.08415_ , 2016.
  * [49] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [50] Y. N. Dauphin, A. Fan, M. Auli _et al._ , “Language modeling with gated convolutional networks,” in _International conference on machine learning_. PMLR, 2017, pp. 933–941.
  * [51] J. T. Barron, “A general and adaptive robust loss function,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4331–4339.
  * [52] I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” _arXiv preprint arXiv:1608.03983_ , 2016.