af

# Robust Model Compression Using Deep Hypotheses

Omri Armstrong, Ran Gilad-Bachrach

###### Abstract

Machine Learning models should ideally be compact and robust. Compactness
provides efficiency and comprehensibility whereas robustness provides
resilience. Both topics have been studied in recent years but in isolation.
Here we present a robust model compression scheme which is independent of
model types: it can compress ensembles, neural networks and other types of
models into diverse types of small models. The main building block is the
notion of depth derived from robust statistics. Originally, depth was
introduced as a measure of the centrality of a point in a sample such that the
median is the deepest point. This concept was extended to classification
functions which makes it possible to define the depth of a hypothesis and the
median hypothesis. Algorithms have been suggested to approximate the median
but they have been limited to binary classification. In this study, we present
a new algorithm, the _Multiclass Empirical Median Optimization_ (MEMO)
algorithm that finds a deep hypothesis in multi-class tasks, and prove its
correctness. This leads to our _Compact Robust Estimated Median Belief
Optimization_ (CREMBO) algorithm for robust model compression. We demonstrate
the success of this algorithm empirically by compressing neural networks and
random forests into small decision trees, which are interpretable models, and
show that they are more accurate and robust than other comparable methods. In
addition, our empirical study shows that our method outperforms Knowledge
Distillation on DNN to DNN compression.

##  1 Introduction

Large models, such as Deep Neural Networks (DNNs) and ensembles achieve high
accuracy on diverse problems [6, 37]. However, their size presents a challenge
in many cases because of their resource requirements and their
incomprehensibility [25]. The lack of interpretability of large machine
learning models is a limitation especially for applications that require
critical decision making such as medical diagnostics [23, 7] and hiring
decisions [18]. Small models are efficient in terms of computational cost and
memory footprint while also being more interpretable. When small models are
required, it has been shown that compressing a large model often outperform
models that were trained small from the get-go [4].

In model compression [4], a large model $M$ is first trained and then
compressed into a smaller model $m$. Most compression schemes are designed for
specific classes of functions [9]. DNN compression schemes include parameter
pruning and quantization [15, 35], low rank factorization and sparsity [32,
11], and Knowledge Distillation (KD) [21] where temperature is used on $M$’s
predictions to create ’soft’ predictions on which $m$ is trained on. There are
also schemes that convert one class of functions to another class, for
example, DNNs to Soft Decision Trees (SDT) [13] or to gradient boosted trees
[8], and trees to DNNs [2].

In this work, we present a new compression scheme that can compress almost any
type of Machine Learning (ML) model to almost any type of smaller model. To be
able to work with diverse learning models, we use the large model $M$ as an
oracle to train a small model $m$. However, in the compression step we avoid
using common training techniques that minimize a loss function over the
training data generated by the oracle [4] since such processes are sensitive
to perturbations [14] and thus are not robust. Instead, we use $M$ to generate
a belief, which is the conditional probability $p(Y=y|X=x)$ and use methods
based on maximizing predicate depth [14].

In our context a belief is a distribution $p(Y=y|X=x)$ where $y$ is one of the
possible classes and $x$ is a record.111We sometimes use the shorthand
notation $p(y|x)$ to denote $p(Y=y|X=x)$. Intuitively, a model $m$ has a
predicate depth $d$ if for every (or most) points $x$ it holds that
$p\left(m(x)|x\right)\geq d$. When $d$ is large, the model is robust to slight
changes in the prior belief $p$ [14]. Therefore, in our method, we extract a
belief from large model $M$ and train a small model $m$ by finding a model
with a large predicate depth from a class of small models. Following [14], we
call the model with the largest predicate depth, the median.

The median has robustness properties by design and compactness is achieved by
restricting the search to classes of small models. An algorithm for
approximating the median hypothesis was introduced in [14] but it is limited
to binary classification. To implement our procedure, we first present the
Multiclass Empirical Median Optimization (MEMO) algorithm for finding the
deepest model out of a function class $\mathcal{F}$ of multi-class
classifiers. Second, we present the Compact Robust Estimated Median Belief
Optimization (CREMBO) algorithm for model compression by finding a deep model
when $\mathcal{F}$ is a class of compact models.

After deriving the algorithms and proving their properties we present an
empirical evaluation of these methods and compare them to existing methods. We
first demonstrate the ability of our method to compress both Random Forests
(RF) [22] and DNNs to small decision trees [31] since they are compact and
interpretable [25, 3]. Then we show that for DNN to DNN compression CREMBO
outperforms the commonly used KD compression scheme on several model
architectures. Our empirical study shows that CREMBO generates models that are
more accurate and more robust than any comparable method.

To the best of our knowledge, this is the first work to study robustness of
model compression methods. Our main contributions presented in this paper are:
The novel CREMBO algorithm for robust model compression. The CREMBO algorithm
is the first to use the concept of deep hypotheses for compression, it is a
flexible algorithm, allowing for compression of diverse model types. Our
empirical evaluation shows that it succeeds in creating compact models that
are more robust and more accurate than any other comparable method. In
addition, we present the MEMO algorithm which extends the ability to find the
median hypothesis to multi-class classifiers. Our code is available at
_https://github.com/TAU-MLwell/Rubust-Model-Compression_.

The rest of the paper is organized as follows: We present the predicate depth
and preliminaries in Section 2. In Section 3 we detail the MEMO algorithm with
proofs. In Section 4 we describe the CREMBO algorithm and in Section 5 we
report our experiments and results. We conclude the paper with a discussion of
the results.

##  2 Background and Notations

Tukey [36] presented depth as a centrality measure of a point in a sample. The
Predicate depth is an extension of the Tukey depth that operates on the space
of classification functions. The predicate depth, as defined for binary
classification tasks [14], measures the agreement of a function $f$ with the
majority vote on $x$. A deep function will always have a large agreement with
its prediction among the class $\mathcal{F}$. The median hypothesis is defined
as the deepest possible function.

######  Definition 1.

[14] Let $\mathcal{F}$ be a function class and let $Q$ be a probability
measure over $\mathcal{F}$. The predicate depth of $f$ on the instance $x\in
X$ with respect to $Q$ is defined as

$D_{Q}(f\mid x)=P_{g\sim Q}[g(x)=f(x)]$

The predicate depth of $f$ with respect to $Q$ is defined as

$D_{Q}(f)=\inf_{x\in X}D_{Q}(f\mid x)$

A common measure of stability is the breakdown point [19]. The breakdown point
measures how much $Q$ must change in order to produce an arbitrary value of
the statistic. The rationale of using deep hypotheses to achieve robustness
derives from a result presented in [14], showing that the breakdown point of
the median hypothesis is proportional to its depth while the breakdown point
of hypotheses acquired in the standard procedure of minimizing some loss
functions (MAP hypothesis) is in fact zero. Another advantage of using deep
hypotheses is that deeper hypotheses have better bounds on their
generalization error.

In practice it might be infeasible to calculate the depth function and find
the median hypothesis. However, it can be approximated using the empirical
depth function:

######  Definition 2.

[14] Given a sample $S=\\{x_{1},...,x_{m}\\}$ s.t. $x_{i}\in X$, a sample
$T=\\{f_{1},...,f_{n}\\}$ s.t. $f_{j}\in\mathcal{F}$ and a function $f$. The
empirical depth on instance $x_{i}\in X$ with respect to $T$ is defined as

$\hat{D}_{T}(f|x_{i})=\frac{1}{n}\sum_{j}1_{f_{j}(x_{i})=f(x_{i})}$

The empirical depth with respect to $T$ is defined as

$\hat{D}_{T}^{S}(f)=\min_{i}\hat{D}_{T}(f|x_{i})$

The empirical depth, as introduced in Definition 2, uses a sample of records
$S$ and a sample of hypotheses $T$ to get an empirical estimate of the
agreement between members of the hypothesis class. However, generating the
sample $T$ is a challenging task; for example, if the hypothesis class is the
class of DNNs, many of them need to be trained to be able to estimate the
empirical depth. Note, however, that $T$ is only used for estimating the
probability $p(y|x)$ for a given record $x$ and a class $y$ and therefore it
is sufficient to assume that there is an oracle $\mathcal{O}(x,y)$ that given
a point $x$ and a class $y$ returns the fraction of the hypotheses that
predict label $y$ for point $x$. This allows us to redefine the empirical
depth as follows:

$\hat{D}_{\mathcal{O}}(f|x_{i})=\mathcal{O}(x_{i},f(x_{i}))$ (1)
$\hat{D}_{\mathcal{O}}^{S}(f)=\min_{i}\hat{D}_{\mathcal{O}}(f|x_{i})$ (2)

The previous definitions of $\hat{D}_{T}(f|x_{i}),\hat{D}_{T}^{S}(f)$
presented in Definition 2 are a special case of these definitions in which the
oracle is $\mathcal{O}(x,y)=\frac{1}{n}\sum_{j}1_{f_{j}(x)=y}$ where $f_{j}\in
T$.

##  3 Multi Class Empirical Median Optimization

We now present the Multiclass Empirical Median Optimization (MEMO) algorithm.
MEMO finds a function $f$ that maximizes the empirical depth, that is
$f=\operatorname*{arg\,max}_{f\in\mathcal{F}}\hat{D}_{\mathcal{O}}^{S}(f)$ for
multi-class classifiers. As mentioned in Section 2, a deep function will have
an agreement with a large fraction of the hypotheses (or posterior belief) on
its predictions. Another way to look at it is to say that whenever it makes a
prediction, it avoids predictions which are in small minorities according to
the belief $p(y|x)$.

Note first that for a given point $x$, the depth $\hat{D}_{\mathcal{O}}(f|x)$
takes its values in the set $\\{\mathcal{O}(x,y):y\in Y\\}$ where $Y$ is the
set of classes (possible labels). Hence, for any given record $x$ the depth
takes values in a set of size at most $|Y|$. Therefore, given a sample $S$ the
set of depth values is:

$\left\\{\hat{D}_{\mathcal{O}}^{S}(f):f\in\mathcal{F}\right\\}\subseteq\left\\{\mathcal{O}(x,y):x\in
S,~{}~{}y\in Y\right\\}~{}~{}~{}.$

Hence, its size is at most $|S||Y|$. Therefore, finding the deepest function
$f\in\mathcal{F}$ can be completed by searching for the largest value
$d\in\left\\{\mathcal{O}(x,y):x\in S,~{}~{}y\in Y\right\\}$ for which there
exists $f\in\mathcal{F}$ such that $\hat{D}_{\mathcal{O}}^{S}(f)\geq d$.
Assuming that we know how to verify whether there exists a function $f$ with a
depth of at least $d$, the deepest function can be found by using binary
search, in $\log|S|+\log|Y|$ steps.

The remaining challenge for finding a deep hypothesis in the multi-class case
is designing the procedure where given a sample $S$ and a desired depth $d$
returns $f\in\mathcal{F}$ such that $\hat{D}_{\mathcal{O}}^{S}(f)\geq d$ if
one exists and returns _”fail”_ otherwise. Let $Y_{i}\subseteq Y$ be the set
of classes for which $\mathcal{O}(x_{i},y)\geq d,~{}~{}x_{i}\in S$. In Theorem
1 we show that $\hat{D}_{\mathcal{O}}^{S}(f)\geq d$ if, and only if,
$f(x_{i})\in Y_{i}$ for every $x_{i}\in S$. Therefore, the procedure we are
looking for is a learning algorithm that receives a sample
$S=\\{x_{i},Y_{i}\\}_{i=1}^{m}$ and learns a function $f$ such that $\forall
i~{}f(x_{i})\in Y_{i}$. This learning problem is different from the standard
classification problem, it can be implemented as a multi-label learning
problem at the training phase whereas on inference, only the class with the
highest probability is selected. Modifying the learning algorithms for
decision trees or DNNs to support this is relatively easy.

These observations allow us to introduce the _Multiclass Empirical Median
Optimization_ (MEMO) algorithm (Algorithm 1) and to prove its correctness in
Theorem 1. In terms of performance, the number of iterations required by the
MEMO algorithm for the binary search is

$\log\left(\left|\left\\{\mathcal{O}(x_{i},y):x_{i}\in S,y\in
Y\right\\}\right|\right)$ (3)

The size of the set in (3) is bounded by the number of unique values that the
oracle can return, which can be very small. Consider, for example, the case of
compressing a Random Forest. One natural way to implement the oracle
$\mathcal{O}$ is to say that $\mathcal{O}(x,y)$ is the fraction of the trees
in the forest which predict that the class is $y$ for some $x\in S$. In this
case, the number of unique values that $\mathcal{O}$ can return is at most
$|M|+1$ where $|M|$ is the number of trees in the forest. Therefore, the
number of iterations required when using MEMO to compress a Random Forest is
$\log(|M|+1)$. When compressing models such as DNNs, the large model $M$
returns a score for each class that can be converted into probabilities using
softmax which the oracle can use as its return value. In this case, the number
of unique values is bounded by the fidelity in which the values are encoded.
If $b$ bits are used to describe the scores, there would be at most $2^{b}$
unique values that the oracle can return and the number of iteration would be
bounded by $b$. Therefore, even if a DNN is used as large model, and 32 bits
numbers are used to represent its outputs, the number of iterations required
by the MEMO algorithm will be $\leq 32$.

Input:

  * •

A sample $S\in X^{m}$

  * •

An oracle $\mathcal{O}(x,y)$

  * •

A learning algorithm $\mathcal{L}$ which given a sample of the form
$\hat{S}=\\{(x_{i},Y_{i})\\}_{i=1}^{m}$ where $Y_{i}\subseteq Y$ returns a
function $f\in\mathcal{F}$ consistent with it if such a function exists and
”fail” otherwise.

Output: A function $f\in\mathcal{F}$ and the depth
$\hat{D}_{\mathcal{O}}^{S}(f)$

1

2begin

3 Let
$\Theta\leftarrow\\{d_{1}<d_{2}<\ldots<d_{m}\\}=\mbox{sort}\left(\left\\{\mathcal{O}(x_{i},y):x_{i}\in
S,y\in Y\right\\}\right)$

4 Run binary search over $\Theta$ unique values to find the largest threshold
$d$ for which the following procedure does not fail:

5 begin

6 for  _i=1,…,m_ do

7 $Y_{i}=\\{y\in Y\;s.t.\;\mathcal{O}(x_{i},y)\geq d\\}$

8 Let $\hat{S}\leftarrow\\{(x_{i},Y_{i})\\}_{i=1}^{m}$

9 Let $f\leftarrow\mathcal{L}(\hat{S})$

10 Return $f$

11

12 Return $f,d$

13

Algorithm 1 Multiclass Empirical Median Optimization (MEMO) Algorithm

######  Theorem 1.

If $\mathcal{F}\neq\emptyset$ then the MEMO algorithm will return a function
$f^{*}$ and a depth $d^{*}$ such that

$d^{*}=\hat{D}_{\mathcal{O}}^{S}(f^{*})=\max_{f\in\mathcal{F}}\hat{D}_{\mathcal{O}}^{S}(f)$

###### Proof.

Recall that $\hat{D}_{\mathcal{O}}^{S}(f)=\min_{x\in
S}\left(\mathcal{O}(x,f(x))\right)$ and therefore, for every
$f\in\mathcal{F}$, $\hat{D}_{\mathcal{O}}^{S}(f)$ is in the set of thresholds
$\Theta$ defined in Algorithm 1. Furthermore, since $d_{1}$ is the minimal
possible threshold then $\forall
f\in\mathcal{F},\hat{D}_{\mathcal{O}}^{S}(f)\geq d_{1}$. Therefore, the binary
search will always return some function $f$ for some threshold $d$.

Assume that $\hat{S}$ was generated with threshold $d$. If there exists
$f\in\mathcal{F}$ such that $\hat{D}_{\mathcal{O}}^{S}(f)\geq d$ then for
every $x\in S$, $\mathcal{O}(x,f(x))\geq d$ and therefore $\mathcal{L}$ will
not fail. However, if $d>\max_{f}\hat{D}_{\mathcal{O}}^{S}(f)$, there is no
$f\in\mathcal{F}$ s.t. $\forall x\in S,f(x)\in\\{y\in
Y~{}s.t.~{}\mathcal{O}(x,y)\geq d\\}$ and $\mathcal{L}$ will fail. Therefore,
the binary search will always terminate when finding the maximal $d$ for which
there exists $f\in\mathcal{F}$ with $\hat{D}_{\mathcal{O}}^{S}(f)\geq d$ and
since this is the maximal value with this property, it has to be that
$d^{*}=\max_{f}\hat{D}_{\mathcal{O}}^{S}(f)$.

To see that $d^{*}=\hat{D}_{\mathcal{O}}^{S}(f^{*})$ recall that from the
definition of $\mathcal{L}$ it follows that $\forall x\in S,f^{*}(x)\in\\{y\in
Y~{}s.t.~{}\mathcal{O}(x,y)\geq d^{*}\\}$ and therefore
$d^{*}\leq\hat{D}_{\mathcal{O}}^{S}(f^{*})$ but from the maximal property of
$d^{*}$ we also know that $d^{*}\geq\hat{D}_{\mathcal{O}}^{S}(f^{*})$ which
completes the proof. ∎

To prove the robustness of MEMO we use the breakdown point as a measure of
robustness [19]. We adjust the definition to our setting in the following way:

######  Definition 3.

The breakdown point of a compression algorithm $\mathcal{C}$ with the oracle
$\mathcal{O}$ and a sample $S$ is

$breakdown(\mathcal{C},\mathcal{O},S)=\max_{f\in\mathcal{F}}\min_{\mathcal{O}^{\prime}s.t.\mathcal{C}(\mathcal{O}^{\prime},S)=f}\left\|\mathcal{O}-\mathcal{O}^{\prime}\right\|_{\infty}$

Definition 3 implies that the breakdown point is the amount of change to the
oracle that is required to allow the compression algorithm to generate an
arbitrary model where the change is measured in total variation distance. The
following theorem proves the robustness of MEMO:

######  Theorem 2.

Let $\mathcal{O}$ be an oracle and $S$ be a sample. Let $\hat{d}$ be the depth
returned by the MEMO algorithm. If $p^{*}=\min_{x\in S,y\in
Y}\mathcal{O}(x,y)$, then the breakdown point of MEMO with the oracle
$\mathcal{O}$ and the sample $S$ is at least
$\nicefrac{{(\hat{d}-p^{*})}}{{2}}$.

###### Proof.

The proof follows from Theorem 1 and uses the same technique as Theorem 9 in
[14]: if $x^{*}\in S$ and $y^{*}\in Y$ are such that
$p^{*}=\mathcal{O}(x^{*},y^{*})$ and $m=\mathcal{C}(\mathcal{O},S)$ and
$m^{\prime}=\mathcal{C}(\mathcal{O}^{\prime},S)$ then if
$m^{\prime}(x^{*})=y^{*}$ then there exists $x\in S$ such that
$\mathcal{O}^{\prime}(x,m(x))\leq\mathcal{O}^{\prime}(x^{*},y^{*})$. Hence,
$\hat{d}-p^{*}\leq\left(\mathcal{O}(x,m(x))-\mathcal{O}^{\prime}(x,m(x))\right)+\left(\mathcal{O}^{\prime}(x^{*},y^{*})-\mathcal{O}(x^{*},y^{*})\right)$
Therefore, at least one of the r.h.s. terms in the last inequality must be
greater than $\nicefrac{{(\hat{d}-p^{*})}}{{2}}$ ∎

Note that the robustness here adds to robustness induced by the soft-max used
to generate the oracle from the model $M$ which is the only source of
robustness for KD.

##  4 Compact Robust Estimated Median Belief Optimization

One nice property of the MEMO algorithm is that it does not require the oracle
$\mathcal{O}$ to return the true probabilities $p(y|x)$. It is sufficient that
the oracle will return $\mathcal{O}(x,y)=g\left(p\left(y|x\right)\right)$
where $g$ is some monotone increasing function. In this case, the algorithm
will return the deepest function $f^{*}$ regardless of the choice of the
function $g$. However the returned depth will be modified by $g$; it would be
$g(d^{*})$ where $d^{*}$ here refers to the true depth; i.e., the one that
would have been computed if $g$ was the identify function.

In model compression, we compress some large model $M$ into a smaller model
$m$. Since we use $M$ to construct the oracle $\mathcal{O}$ it is essential
that $M$ returns probabilities or, as discussed above, some monotone
increasing function of these probabilities. This can be achieved, for example,
by using a softmax layer at the end of a DNN or by taking the agreement
probabilities of an ensemble (see a discussion in Section 6 about additional
methods). These conversions allow the use of model $M$ as the oracle
$\mathcal{O}$. To achieve compression, the search for a deep function is made
in a class of small models $\mathcal{F}$. In this setup, running the MEMO
algorithm will find a compact function with the largest depth. However, the
limited capacity of the function class $\mathcal{F}$ combined with some
possible outliers in the data may make the constraints too stringent. One way
to see that is to note that when the dataset $S$ increases in size, more and
more constraints are added to the MEMO algorithm, which decreases the maximal
possible depth and therefore the depth function, as a method to distinguish
between good and bad models, loses its dynamic range. This may make it hard to
distinguish between functions that will generalize well and other functions
that will not. To overcome this issue, we relax the constraint such that
instead of requiring that $\forall x\in S,~{}~{}~{}\mathcal{O}(x,f(x))\geq d$
we require that the condition holds for _most_ $x\in S$. To this end, we
define the $\delta$-insensitive empirical depth:

######  Definition 4.

Let $\mathcal{O}$ be an oracle, let $S=\\{x_{1},...,x_{m}\\}\in X^{m}$ be a
sample and let $\delta\in[0,1]$. The $\delta$-insensitive empirical depth of
$f$ with respect to $\mathcal{O}$ is

$\hat{D}_{\mathcal{O}}^{S,\delta}(f)=\max_{T\subseteq
S,\left|T\right|\geq(1-\delta)\left|S\right|}\hat{D}_{\mathcal{O}}^{S}(f)$

The $\delta$-insensitive empirical depth requires that function $f$ will have
a large agreement with $\mathcal{O}$ on all but a set of instances at a
proportion smaller or equal to $\delta$. In the language of robust statistics,
the $\delta$-insensitive empirical depth can be considered as a trimmed
estimator [10].

The _Compact Robust Estimated Median Belief Optimization_ (CREMBO) algorithm
(Algorithm 2) handles the trade off between robustness and accuracy by
optimizing with respect to the $\delta$-insensitive empirical depth.
Unfortunately, optimizing with respect to this measure is harder and therefore
the CREMBO algorithm is not guaranteed to find the deepest hypothesis with
respect to the $\delta$-insensitive empirical depth function and instead finds
an approximation.

The CREMBO algorithm finds a function with large $\delta$-insensitive
empirical depth that performs well on a validation set. It starts with the
solution provided by the MEMO algorithm. This provides an initial depth
$d^{*}$ that can be achieved with $\delta=0$. The algorithm increases the
required depth and for each threshold $d$ of the depth it generates the set of
allowed labels $Y_{i}=\\{y\in Y~{}s.t.~{}\mathcal{O}(x_{i},y)\geq
d\\},~{}\forall x_{i}\in S$ much like in the MEMO algorithm. However, since
the depth is greater than the depth of the empirical median returned by the
MEMO algorithm, there is no hypothesis in $\mathcal{F}$ that is consistent
with this sample. Therefore, it allows the learning algorithm to return a
hypothesis that is consistent with most of the training points but not all of
them.

Since we do not know what a good value would be for $\delta$ up-front, a
validation set is used by CREMBO to compare the hypotheses returned for
different depth thresholds and select the best one. The selection criteria may
be, accuracy, F1 or the AUC for example. The CREMBO algorithm uses linear
search on the thresholds to find $m$. In cases where there are many threshold
values it is possible to perform the search with steps of size $\Delta$. This
way, the number of iterations needed for the algorithm is
$\frac{\left|\left\\{\mathcal{O}(x_{i},y):~{}x_{i}\in S,~{}y\in
Y\right\\}\right|}{\Delta}$. Using different search methods such as line
search can reduce the number of iterations exponentially [16].

Input:

  * •

A sample $S\in X^{m}$

  * •

An oracle $\mathcal{O}(x,y)$

  * •

A validation set $\mathcal{Z}\in(X\times Y)^{u}$

  * •

The median hypothesis $f^{*}$, a set of sorted thresholds $\Theta$ and depth
$d^{*}$ computed by the MEMO algorithm ($f^{*},\Theta,d^{*}\leftarrow MEMO()$)

  * •

A learning algorithm $\mathcal{A}$ that given a sample of the form
$\hat{S}=\\{(x_{i},Y_{i})\\}_{i=1}^{m}$ where $Y_{i}\subseteq Y$ trains a
function $f\in\mathcal{F}$ and returns it

  * •

Evaluation metric $V$, that given a function $f$ and a validation set
$\mathcal{Z}$ returns the set score

  * •

A step size $\Delta$

Output: A deep compact function $f\in\mathcal{F}$

1 begin

2 best $\leftarrow V(f^{*},\mathcal{Z}$)

3 $f\leftarrow f^{*}$

$D\leftarrow\Theta[\Theta\geq d^{*}][::\Delta]$ // Get all values in $\Theta$
larger than $d^{*}$ with $\Delta$ interval

4

5 for  _d in $D$_ do

6

7 for  _i=1,…,m_ do

8 $Y_{i}=\\{y\in Y\;s.t.\;\mathcal{O}(x_{i},y)\geq d\\}$

9 $\hat{S}\leftarrow\\{(x_{i},Y_{i})\\}_{i=1}^{m}$

10 $h\leftarrow\mathcal{A}(\hat{S})$

11 score $\leftarrow$ V($h$, $\mathcal{Z}$)

12 if  _score $>$ best_ then

13 $f\leftarrow h$

14 best $\leftarrow$ score

15

16 Return $f$

17

Algorithm 2 Compact Robust Estimated Median Belief Optimization (CREMBO)

##  5 Experiments

We evaluate the CREMBO algorithm using two sets of experiments. On the first
set of experiments we evaluate the generalization and robustness of the CREMBO
algorithm (Section 5.1). On the second set, we test CREMBOs ability to create
accurate compact models on the DNN to DNN compression task and compare it to
KD (Section 5.2).

###  5.1 Compressing to Interpretable Models

To evaluate the CREMBO algorithm as a robust model compression scheme, we
conducted two experiments, a generalization experiment in which the compressed
models accuracy and win rate were evaluated using 10-fold cross-validation and
a robustness experiment where the compressed models were evaluated on the
level of their agreement. In each experiment two types of models were
compressed, a Random Forest model (RF) [22] which is an ensemble model and a
Deep Neural Network (DNN). Both models were compressed with the CREMBO
algorithm to a small, fixed depth decision tree, the median tree (MED). These
trees were compared to two other same depth trees: benchmark tree (BM), which
is trained on the original training data
$S_{train}=\\{(x_{i},y_{i})\\}_{i=1}^{m}$ and a student tree (ST), trained on
labels generated from the large model (teacher) predictions
$S_{teacher}=\\{(x_{i},M(x_{i}))\\}_{i=1}^{m}$.

We evaluated the CREMBO algorithm on five classification tasks (Table 1) from
the UCI repository [12]. To implement the DNNs we used PyTorch [29]. The DNNs
are all fully connected with two hidden layers of 128 units with ReLu
activation functions. They were trained with an ADAM optimizer with default
parameters and batch size of 32 for 10 epochs. For the Random Forest and
decision tree models we used scikit-learn [30] package. The Random Forest
model was trained with 100 trees with a maximal depth of 12 and balanced
weights. All the decision tree models were trained with a maximal depth of 4,
so they are small and interpretable, and balanced weights.

Dataset | Instances | Attributes | Classes
---|---|---|---
Dermatology | 366 | 33 | 6
Heart | 304 | 13 | 5
Arrhythmia | 452 | 279 | 16
Breast cancer | 569 | 30 | 2
Iris | 150 | 4 | 3
Table 1: Dataset statistics | Accuracy | Win rate
---|---|---
Dataset | RF | BM | ST | MED | BM | ST | MED
Dermatology | 98.05 | 82.26 | 82.18 | 90.62 | 2 | 22.5 | 75.5
Heart | 56.24 | 38.93 | 38.93 | 52.6 | 0 | 0.5 | 99.5
Arrhythmia | 70.89 | 9.36 | 7.7 | 54.79 | 0 | 0 | 100
Breast cancer | 96 | 93.25 | 93.13 | 92.47 | 13.5 | 32 | 54.5
Iris | 94.53 | 92.53 | 92.53 | 94.66 | 0 | 12 | 88
Table 2: Accuracy and win rate results (in percentage) over 10-fold CV of benchmark tree (BM), student tree (ST) and median tree (MED) averaged over 20 experiments where the compressed model is a Random Forest (RF) | Accuracy | Win rate
---|---|---
Dataset | DNN | BM | ST | MED | BM | ST | MED
Dermatology | 97.56 | 82.2 | 82.22 | 89.55 | 6 | 25.5 | 68.5
Heart | 64.27 | 38.87 | 49.6 | 52.4 | 0.5 | 33 | 66.5
Arrhythmia | 70.41 | 9.46 | 15.33 | 55.6 | 0 | 6.5 | 93.5
Breast cancer | 93.87 | 93.21 | 94.03 | 92.68 | 25.5 | 45 | 29.5
Iris | 36.92 | 92.33 | 92.06 | 91.19 | 13.5 | 15 | 71.5
Table 3: Accuracy and win rate results (in percentage) over 10-fold CV of,
benchmark tree (BM), student tree (ST) and median tree (MED) averaged over 20
experiments where the compressed model is a Deep Neural Network (DNN)

#### Generalization

To evaluate the generalization ability of the compressed models we used
10-fold cross-validation (CV). In each round 9 folds are used as the training
set $S_{train}$ and the remaining fold is used as a test set. We first train
the large model $M$ and a benchmark tree on $S_{train}$, then using $M$
predictions we create $S_{teacher}$ and train the student tree. To find the
median tree, we split $S_{train}$ into a train and validation sets,
$S^{\prime}_{train}$, $S_{val}$, with a random $15\%$ split and run the CREMBO
algorithm. The accuracy on the test set is calculated for all models and later
averaged on all rounds. In addition, we measure the win rate for each model.
The win rate is the percentage of rounds in which a model outperformed the
other models. We repeated the experiment 20 times and the average results are
provided in Table 2, and Table 3. The results show that for Random Forest
compression, the median tree had the best accuracy and win rates by a
considerable margin on all datasets except for the Breast cancer dataset on
which the benchmark tree had better accuracy by a relatively small margin. For
DNN compression, there were similar results for the Dermatology, Heart,
Arrhythmia and Breast cancer datasets. On the first three, the median tree
outperformed the other trees and on the last it was less accurate. An
interesting result emerged for the Iris dataset. The DNN is clearly
overfitted, since it has an average accuracy score of only $36.92\%$ on the
test sets. The student tree was hardly affected since the DNN predictions on
the training set were very accurate. On the other hand, there was a negative
impact on the median tree since the belief probabilities $p(y|x)$ provided by
the DNN were not accurate enough. Nevertheless, the median tree still had the
highest win rate and much higher accuracy on the test sets than the larger DNN
model.

#### Robustness

In Theorem 2 we were able to prove the robustness of MEMO to changes in the
oracle. Here, we evaluate robustness empirically by training the big model $M$
with different training sets and measuring the impact on the compressed
models. To measure similarity between compressed models we say that models
_agree_ on $x$ if they make the same prediction on this point, regardless of
the correctness of this prediction. In the experiment we divided the dataset
into a train and test sets with a random $15\%$ split. To simulate data
perturbations, we used 10-fold CV on the training set. On each round, we took
9 of the 10 folds to be $S_{train}$ while the remaining fold was omitted. The
training process of large model $M$ and the trees was done in the same manner
as in the generalization experiment. We measured the agreement of same type
trees across rounds on the test set and averaged the score. This experiment
was repeated 20 times and the average scores are presented in Table 4. The
median tree was more robust on 7 out of 10 test settings (4 out of 5 when
compressing to trees and 3 out of 5 when compressing to neural nets), in some
cases with very large margins. On the other 3 cases it was close to the other
techniques in terms of robustness.

The results from the generalization and the robustness experiments show that
the CREMBO algorithm is able to compress Random Forests and DNNs to small
decision trees that are more accurate and robust than same sized trees trained
with comparable methods on a variety of datasets. The average accuracy
improvement over datasets (in absolute percentage) was $13.76\%$ for RF
compression and $9.57\%$ for DNN compression and the average robustness
improvements were $12.7\%$ and $7.8\%$ for RF and DNN compression
respectively. We note that our results are statistically significant.

| Random Forest | DNN
---|---|---
Dataset | BM | ST | MED | BM | ST | MED
Dermatology | 91.89 | 91.9 | 92.18 | 91.88 | 90.01 | 92.67
Heart | 63.04 | 63.08 | 70.29 | 63.17 | 60.76 | 68.51
Arrhythmia | 32.76 | 32.17 | 89.91 | 32.63 | 21.03 | 72.72
Breast cancer | 94.7 | 94.85 | 95.91 | 94.71 | 93.3 | 92.63
Iris | 99.52 | 99.65 | 97.85 | 99.47 | 96.67 | 94.44
Table 4: Agreement results (in percentage) of benchmark tree (BM), student tree (ST) and median tree (MED) averaged over 20 experiments where the large models compressed are Random Forest (left) and Deep Neural Network (right) $M$ | $m$ | T | Baseline($m$) | KD | CREMBO
---|---|---|---|---|---
ResNet18 | LeNet-5 | 20 | 70.76 | 71.53 | 72.27
VGG16 | LeNet-5 | 5 | 70.76 | 70.49 | 72.43
ResNet18 | MobileNetV2 | 5 | 91.97 | 92.17 | 92.19
VGG16 | MobileNetV2 | 5 | 91.97 | 92.15 | 92.35
Table 5: Accuracy results ($\%$) for large model $M$ compression to small model $m$ with Knowledge Distillation (KD) and CREMBO on CIFAR-10. Baseline accuracy for the small model $m$ and Temperature (T) used are provided Model |  $\\#$ Parameters | Accuracy ($\%$)
---|---|---
resnet18 | 11173962 | 93.15
VGG16 | 14728266 | 92.22
MobileNetV2 | 2296922 | 91.97
LeNet-5 | 62006 | 70.76
Table 6: Number of model parameters and baseline accuracy on CIFAR-10

###  5.2 DNN to DNN Compression

DNN to DNN compression is a highly studied filed [4]. To test our method’s
ability to compress large DNNs to compact DNNs, we used CREMBO to compress
large DNNs to compact DNNs and compared them to baseline models, i.e., models
trained small from the get-go, and to compact models generated with Knowledge
Distillation (KD). We compressed two types of large DNNs, ResNet18 [20] and
VGG16 [34] to two compact DNNs, LeNet-5 [26] and MobileNetV2 [33]. Where
LeNet-5 is a very small DNN and MobileNetV2 is a compact DNN designed to run
on mobile devices. The models were trained on the CIFAR-10 dataset [24]. The
number of parameters and baseline results of all models on CIFAR-10 are
presented on Table 6.

The training process was the same for all DNNs. We used ADAM optimizer, batch
size of 128, learning rate of 0.01 for 60 epochs and then learning rate of
0.001 for another 30 epochs. We first trained the large DNNs ($M$) on the
training set. Then we divided the training set to a train and validation set
with a random $10\%$ split. We used the validation set for the CREMBO
algorithm and to find the best KD temperature value out of $[3,5,8,20]$ for
each model. After finding the best temperature values, CREMBO and KD were used
to compress $M$ to compact DNNs ($m$). We implemented KD as in [21] using both
soft and regular targets as recomended in [21]. The results of our experiments
are presented in Table 5.

The results show that CREMBO improves the baseline and outperforms KD on all
tested models. This is another testimony for CREMBO’s flexibility and ability
to compress large models to compact models that generalize well.

##  6 Conclusions

In this study we presented a novel robust model compression scheme for multi-
class classifiers that can compress variety of large models, such as DNNs and
ensembles, into compact models. To ensure robustness it uses tools from robust
statistics; namely, the statistical depth and trimmed estimators. We presented
the MEMO algorithm, a new algorithm for finding the empirical median
hypothesis in the multi-class setting. For model compression we introduced the
CREMBO algorithm. CREMBO uses a trimmed version of the depth function to
search for deep hypotheses in a class of compact classifiers and therefore
achieve both robustness and compression. We demonstrated the ability of CREMBO
to compress both DNNs and ensembles into small decision trees which are more
accurate and robust than trees trained with comparable methods. This is useful
for different explainability purposes since small trees are comprehensible
while ensembles and DNNs are much harder to interpret. The robustness and
accuracy of the compressed model ensure that it represents the large model it
captures well.

Compressing models is also advantageous in other scenarios, such as when a
model is to be used on a device with limited resources or when latency is
critical. This is especially true for large DNNs which are known for their
large size and computing demands. To this end, we evaluated CREMBO’s ability
to compress large DNNs to compact DNNs and showed that CREMBO outperforms KD.

Although CREMBO works for a variety of model types, it is possible to add
model specific variations to further improve results. For example, adding
temperature to CREMBO in DNN to DNN compression. DNNs predictions tend to be
overconfident and in general not well-calibrated [17]. This means that the
posterior probabilities we get from $M$ can be overconfident as well. Adding
temperature has been said to improve calibration [17]. This variation can be
seen as a combination of KD and CREMBO in which the allowed labels get
weighted according to the soft predictions. We leave this and other possible
variations for future work.

##  7 Broader Impact

Machine learning models are affecting our daily lives in numerous ways; hiring
decisions [18], parole decisions [1], self-driven cars and even the news we
see on social media [27] are all controlled in some way by machine learning
models. Given the high stakes of decisions made by these models, it is clear
that the ethical aspects of machine learning cannot be neglected. However, the
complexity of deep learning models and large ensembles makes them
incomprehensible. This means, for example, that it is hard to verify that
these models are not biased against protected groups [28] or that they do not
memorize personal information [5]. Therefore, the ability to create a small
interpretable model that preserves the main logic acquired by a larger model,
which is the topic of this study, is an important tool in understanding a
machine learning model and identifying potential risks encoded in it [7]. The
robustness of the process of creating a small model is critical since it
provides guarantees that the small model represents its larger counterpart
well.

Robust compression is also useful when models are applied on resource limited
devices or in cases where latency is critical. When models are applied on
wearable devices or other mobile devices, power consumption, memory
consumption, and bandwidth are key factors. Compression in these settings
makes it possible to create accurate models that can work within the available
resources.

Therefore, we conclude that robust compression is likely to have an impact on
several important facets of machine learning.

## References

  * Angwin et al. [2016] Angwin, J.; Larson, J.; Mattu, S.; and Kirchner, L. 2016.  Machine bias.  _ProPublica, May_ 23: 2016.
  * Banerjee [1997] Banerjee, A. 1997.  Initializing neural networks using decision trees.  _Computational learning theory and natural learning systems_ 4: 3\.
  * Bénard et al. [2020] Bénard, C.; Biau, G.; Da Veiga, S.; and Scornet, E. 2020.  Interpretable Random Forests via Rule Extraction.  _arXiv preprint arXiv:2004.14841_ .
  * Buciluǎ, Caruana, and Niculescu-Mizil [2006] Buciluǎ, C.; Caruana, R.; and Niculescu-Mizil, A. 2006.  Model compression.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 535–541.
  * Carlini et al. [2019] Carlini, N.; Liu, C.; Erlingsson, Ú.; Kos, J.; and Song, D. 2019.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _28th Security Symposium (Security 19)_ , 267–284.
  * Caruana, Karampatziakis, and Yessenalina [2008] Caruana, R.; Karampatziakis, N.; and Yessenalina, A. 2008.  An empirical evaluation of supervised learning in high dimensions.  In _Proceedings of the 25th international conference on Machine learning_ , 96–103.
  * Caruana et al. [2015] Caruana, R.; Lou, Y.; Gehrke, J.; Koch, P.; Sturm, M.; and Elhadad, N. 2015.  Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission.  In _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , 1721–1730.
  * Che et al. [2016] Che, Z.; Purushotham, S.; Khemani, R.; and Liu, Y. 2016.  Interpretable deep models for ICU outcome prediction.  In _AMIA Annual Symposium Proceedings_ , volume 2016, 371. American Medical Informatics Association.
  * Cheng et al. [2017] Cheng, Y.; Wang, D.; Zhou, P.; and Zhang, T. 2017.  A survey of model compression and acceleration for deep neural networks.  _arXiv preprint arXiv:1710.09282_ .
  * Daszykowski et al. [2007] Daszykowski, M.; Kaczmarek, K.; Vander Heyden, Y.; and Walczak, B. 2007.  Robust statistics in data analysis—A review: Basic concepts.  _Chemometrics and intelligent laboratory systems_ 85(2): 203–219.
  * Denil et al. [2013] Denil, M.; Shakibi, B.; Dinh, L.; Ranzato, M.; and De Freitas, N. 2013.  Predicting parameters in deep learning.  In _Advances in neural information processing systems_ , 2148–2156.
  * Dua and Graff [2017] Dua, D.; and Graff, C. 2017.  UCI Machine Learning Repository.  URL http://archive.ics.uci.edu/ml.
  * Frosst and Hinton [2017] Frosst, N.; and Hinton, G. 2017.  Distilling a neural network into a soft decision tree.  _arXiv preprint arXiv:1711.09784_ .
  * Gilad-Bachrach and Burges [2013] Gilad-Bachrach, R.; and Burges, C. J. 2013.  Classifier selection using the predicate depth.  _The Journal of Machine Learning Research_ 14(1): 3591–3618.
  * Gong et al. [2014] Gong, Y.; Liu, L.; Yang, M.; and Bourdev, L. 2014.  Compressing deep convolutional networks using vector quantization.  _arXiv preprint arXiv:1412.6115_ .
  * Grippo, Lampariello, and Lucidi [1986] Grippo, L.; Lampariello, F.; and Lucidi, S. 1986.  A nonmonotone line search technique for Newton’s method.  _SIAM Journal on Numerical Analysis_ 23(4): 707–716.
  * Guo et al. [2017] Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , 1321–1330. JMLR. org.
  * Hamilton [2018] Hamilton, I. A. 2018.  Why It’s Totally Unsurprising That Amazon’s Recruitment AI Was Biased against Women.  _Business Insider, October_ 13\.
  * Hampel [1971] Hampel, F. R. 1971.  A general qualitative definition of robustness.  _The Annals of Mathematical Statistics_ 1887–1896.
  * He et al. [2016] He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Hinton, Vinyals, and Dean [2015] Hinton, G.; Vinyals, O.; and Dean, J. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ .
  * Ho [1995] Ho, T. K. 1995.  Random decision forests.  In _Proceedings of 3rd international conference on document analysis and recognition_ , volume 1, 278–282. IEEE.
  * Kononenko [2001] Kononenko, I. 2001.  Machine learning for medical diagnosis: history, state of the art and perspective.  _Artificial Intelligence in medicine_ 23(1): 89–109.
  * Krizhevsky, Hinton et al. [2009] Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images .
  * Lage et al. [2019] Lage, I.; Chen, E.; He, J.; Narayanan, M.; Kim, B.; Gershman, S.; and Doshi-Velez, F. 2019.  An evaluation of the human-interpretability of explanation.  _arXiv preprint arXiv:1902.00006_ .
  * LeCun et al. [1998] LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ 86(11): 2278–2324.
  * Liu, Dolan, and Pedersen [2010] Liu, J.; Dolan, P.; and Pedersen, E. R. 2010.  Personalized news recommendation based on click behavior.  In _Proceedings of the 15th international conference on Intelligent user interfaces_ , 31–40.
  * Mehrabi et al. [2019] Mehrabi, N.; Morstatter, F.; Saxena, N.; Lerman, K.; and Galstyan, A. 2019.  A survey on bias and fairness in machine learning.  _arXiv preprint arXiv:1908.09635_ .
  * Paszke et al. [2017] Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; and Lerer, A. 2017.  Automatic differentiation in PyTorch.  In _NIPS-W_.
  * Pedregosa et al. [2011] Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; and Duchesnay, E. 2011.  Scikit-learn: Machine Learning in Python.  _Journal of Machine Learning Research_ 12: 2825–2830.
  * Quinlan [1986] Quinlan, J. R. 1986.  Induction of decision trees.  _Machine learning_ 1(1): 81–106.
  * Rigamonti et al. [2013] Rigamonti, R.; Sironi, A.; Lepetit, V.; and Fua, P. 2013.  Learning separable filters.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2754–2761.
  * Sandler et al. [2018] Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2018.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 4510–4520.
  * Simonyan and Zisserman [2014] Simonyan, K.; and Zisserman, A. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ .
  * Srinivas and Babu [2015] Srinivas, S.; and Babu, R. V. 2015.  Data-free parameter pruning for deep neural networks.  _arXiv preprint arXiv:1507.06149_ .
  * Tukey [1975] Tukey, J. W. 1975.  Mathematics and the picturing of data.  In _Proceedings of the International Congress of Mathematicians, Vancouver, 1975_ , volume 2, 523–531.
  * Usmanim [2018] Usmanim, Z. u. H. 2018.  How to Win Kaggle Competitions.  https://www.kaggle.com/getting-started/44997.  (accessed May 22, 2020).


# Unruh effect in curved space-time and hydrodynamics

R. V. Khakimov  Joint Institute for Nuclear Research, Joliot-Curie str. 6,
Dubna 141980, Russia  NRC Kurchatov Institute, Moscow, Russia  Physics
Department, Lomonosov Moscow State University, 1-2 Leninskie Gory, Moscow
119991, Russia  G. Yu. Prokhorov  Joint Institute for Nuclear Research,
Joliot-Curie str. 6, Dubna 141980, Russia  NRC Kurchatov Institute, Moscow,
Russia  O. V. Teryaev  Joint Institute for Nuclear Research, Joliot-Curie str.
6, Dubna 141980, Russia  NRC Kurchatov Institute, Moscow, Russia  V. I.
Zakharov  NRC Kurchatov Institute, Moscow, Russia  Joint Institute for Nuclear
Research, Joliot-Curie str. 6, Dubna 141980, Russia  Pacific Quantum Center,
Far Eastern Federal University, 10 Ajax Bay, Russky Island, Vladivostok
690950, Russia

## Abstract

We consider an accelerated relativistic fluid in four-dimensional (anti-)de
Sitter space-time. Analyzing only hydrodynamic equations, we construct the
equilibrium stress-energy tensor. We confirm that (A)dS vacuum corresponds to
a thermal bath in the accelerated frame with a temperature, depending on the
acceleration in a flat higher-dimensional (namely, five-dimensional) space, in
which curved space-times are embedded. We develop the duality between
hydrodynamics and gravity finding a direct relationship between the transport
coefficients in flat and curved space-times.

##  1 Introduction

Hydrodynamics allows us to describe many physical systems and phenomena, not
only classical, but also quantum. A particularly striking example is the
extreme state of strongly coupled matter in the deconfinement phase, i.e. the
quark-gluon plasma (QGP) found in heavy-ion collisions, which can be described
as relativistic fluid system [1]. The main tools of the hydrodynamic approach
are the conservation laws (e.g., of the stress-energy tensor (SET) and
currents), as well as the gradient expansion [2, 3]. The idea of the gradient
expansion is that hydrodynamic quantities are given by series in terms of
space-time derivatives acting on hydrodynamic degrees of freedom, such as
temperature, chemical potential, fluid four-velocity ($\displaystyle T$,
$\displaystyle\mu$, $\displaystyle u_{\mu}$). Considering hydrodynamics in
curved space-time, we additionally take into account the gradients of the
space-time metric $\displaystyle g_{\mu\nu}$.

A well-known example of the application of the first-order current gradient
expansion is the work [4] in which the relationship is established between
seemingly very distant phenomena: hydrodynamics and gauge chiral anomaly (see
also another original derivation [5]). This successful application of the
methods of hydrodynamics to the analysis of quantum anomalies led to a series
of subsequent works, such as [6, 7], but mostly also for a gauge chiral
anomaly. Therefore, of undoubted interest are the works on the study of
another famous anomaly, the gravitational chiral anomaly [8, 9, 10, 11].

This list includes recent work in the context of the hydrodynamic approach to
anomalies [12], in which the relationship was established between the
transport coefficients in flat space-time and the gravitational axial anomaly,
which was called kinematical vortical effect (KVE). This effect is a
manifestation of the duality between flat space-time and curved space-time
(gravity), because it was shown that the axial current in a vortical and
accelerated fluid in a flat space-time is determined by the gravitational
chiral anomaly. In [12] derivation of the KVE was made in Ricci-flat space-
time approximation $\displaystyle R_{\mu\nu}=0$ . The next logical step is to
generalize our derivation and consider space-time with non-zero Ricci tensor,
in the simplest case proportional to space-time metric $\displaystyle
R_{\mu\nu}=\Lambda g_{\mu\nu}$ , so-called Einstein manifolds. This means that
expressions for hydrodynamic quantities will contain terms with scalar
curvature $\displaystyle R$. Finding the corresponding expansions in the case
of the stress-energy tensor and the search for new elements of duality between
hydrodynamics and gravity is the main goal of this paper.

Besides duality, we will give a new perspective on the interesting question of
the Unruh effect in curved space-time. It is known that the Hawking effect
extends to other space-times with a horizon, in particular, there is a well-
known analog of Hawking radiation in an accelerated frame, the Unruh effect
[13, 14]. The radiation temperature (the Unruh temperature) depends on the
acceleration $\displaystyle T_{U}=\frac{|a|}{2\pi}$, where $\displaystyle
a_{\mu}=u^{\nu}\partial_{\nu}u_{\mu}$ is the proper acceleration,
$\displaystyle|a|=\sqrt{-a^{\mu}a_{\mu}}$, and $\displaystyle u_{\mu}$ is the
four-velocity. There is also another well-known example of the de Sitter
space-time, whose temperature is determined by the scalar curvature
$\displaystyle T_{R}=\frac{\sqrt{R/12}}{2\pi}$ [15]. Later, the combined case
with constant curvature and acceleration was considered, and the temperature
of the radiation was shown to be a combination of $\displaystyle T_{U}$ and
$\displaystyle T_{R}$

$\displaystyle\displaystyle
T_{UR}=\frac{a_{5}}{2\pi}=\sqrt{T^{2}_{U}+T^{2}_{R}}=\frac{\sqrt{|a|^{2}+R/12}}{2\pi}\>.$
(1)

It is noteworthy that (1) is valid both for accelerated dS [16, 17]
($\displaystyle R>0$) and anti-de Sitter (AdS) [17] ($\displaystyle R<0$)
spaces.

This combined case became indicative, demonstrating the role of the flat
higher-dimensional space-times in which curved space-times can be embedded. As
in the Unruh effect, the temperature (1) is determined by acceleration, not
four, but five-dimensional, the square of which is
$\displaystyle|a_{5}|^{2}=|a|^{2}+R/12$. In this letter, we analyze the
thermal radiation in accelerated (A)dS space from the point of view of
relativistic hydrodynamics. We argue that (1) can be obtained from the basic
hydrodynamic equations and the general relativistic covariance, if we know the
hydrodynamic expansions in a curved space-time in the case with acceleration
only. This approach develops the mentioned similarity between thermodynamics
and gravity.

For clarity, let us start the consideration with the case of hydrodynamics in
flat space-time and ordinary Unruh effect.

##  2 Unruh effect from hydrodynamics: acceleration in flat space-time

Let us consider a relativistic non-dissipative fluid with four-velocity
$\displaystyle u_{\mu}$, constant acceleration $\displaystyle a_{\mu}$ and
zero chemical potential $\displaystyle\mu=0$ in flat space-time (we choose the
signature $\displaystyle(+,-,-,-)$). For simplicity, the particles that form
the fluid will be assumed to be massless (or nearly massless, when the mass is
much less than the other dimensional parameters). The stress-energy tensor can
be constructed in terms of gradient expansion that terminates at the fourth
order [18, 19, 20]. The corresponding transport coefficients can be found
within the quantum statistical approach from the correlators with boost
operators. In particular, for the spins $\displaystyle s=0$ and $\displaystyle
s=1/2$ we obtain [21, 22, 23, 24]111Due to the covariance of (2) and (3), the
metric can be both the Minkowski or, for example, the Rindler metric.

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{s=0}$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\Big{(}\frac{4\pi^{2}T^{4}}{90}-\frac{|a|^{4}}{360\pi^{2}}\Big{)}u^{\mu}u^{\nu}-\Big{(}\frac{\pi^{2}T^{4}}{90}-\frac{|a|^{4}}{1440\pi^{2}}\Big{)}g^{\mu\nu},$
(2) $\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{s=1/2}$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\Big{(}\frac{28\pi^{2}T^{4}}{180}+\frac{T^{2}|a|^{2}}{18}-\frac{17|a|^{4}}{720\pi^{2}}\Big{)}u^{\mu}u^{\nu}-\Big{(}\frac{7\pi^{2}T^{4}}{180}+\frac{T^{2}|a|^{2}}{72}-\frac{17|a|^{4}}{2880\pi^{2}}\Big{)}g^{\mu\nu},$
(3)

It is essential that both SETs are equal to zero at the Unruh temperature
[13], which is a direct indication of the Unruh effect [25]

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle(T=T_{U})=0.$ (4)

The explanation is the following. The matrix element of the SET is divergent
and should be renormalized. A renormalization is used for which the matrix
element for the Minkowski vacuum state (when there are no particles in the
inertial frame) is zero. Relativistic covariance tells us that a tensor equal
to zero in one reference frame is equal to zero in any other, including non-
inertial ones [26]. Thus, it follows from the equality (4) that the finite
temperature in the accelerated system corresponds to the Minkowski vacuum
state. And this is, actually, the Unruh effect.

However, it is easy to show that a naive generalization of (2) and (3) to the
case of curved space-time, with just a change of derivatives to covariant
ones, e.g., $\displaystyle a_{\mu}=u^{\nu}\nabla_{\nu}u_{\mu}$, leads to non-
conserved SETs. In particular (the details are given in the next Section), we
would obtain from (2) a non-zero contribution to the divergence in the case of
the curved space-time, namely
$\displaystyle\nabla_{\mu}\langle\hat{T}^{\mu\nu}\rangle_{(s=0)}=-\frac{a^{2}R}{4320\pi^{2}}a^{\nu}\neq
0$. This indicates that there are additional effects of curvature, that are
not taken into account, which we consider in the next Section.

##  3 Hydrodynamic gradient expansion: acceleration and constant curvature

Consider now the same fluid, but in (A)dS space-time with $\displaystyle
R_{\mu\nu}=\Lambda g_{\mu\nu}$, where $\displaystyle R=4\Lambda$ is a constant
curvature. In this case, the gravitational field will be considered as
external. Gradient expansion should now also take into account the
contribution of terms with scalar curvature $\displaystyle R=4\Lambda$ of the
space-time (which is the second order in the gradients of the metric). Let us
write down the expansion for the SET in the general case of massless particles
with an arbitrary spin. Also, to facilitate the finding of covariant
derivatives and further calculations, it will be convenient to switch to a
dimensionless (or ‘‘thermal’’) acceleration
$\displaystyle\alpha_{\mu}=\frac{a_{\mu}}{T}$. Therefore the SET in the fourth
order in terms of gradients will have the form

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\Big{(}\rho_{0}+A_{1}\alpha^{2}+A_{2}R+B_{1}\alpha^{4}+B_{2}\alpha^{2}R+B_{3}R^{2}\Big{)}u^{\mu}u^{\nu}-$
(5)
$\displaystyle\displaystyle-\Big{(}p_{0}+A_{3}\alpha^{2}+A_{4}R+B_{4}\alpha^{4}+B_{5}\alpha^{2}R+B_{6}R^{2}\Big{)}g^{\mu\nu}+$
$\displaystyle\displaystyle+\Big{(}A_{5}+B_{7}\alpha^{2}+B_{8}R\Big{)}\alpha^{\mu}\alpha^{\nu}+\mathcal{O}(\nabla^{6}),$

where $\displaystyle\rho_{0}(T)$, $\displaystyle p_{0}(T)$, $\displaystyle
A_{i}(T)$ and $\displaystyle B_{i}(T)$ are coefficients that depend on the
only dimensional parameter, temperature $\displaystyle T$. We distinguished
the coefficients corresponding to the terms of different orders of gradients:
$\displaystyle A_{i}(T)$ correspond to the second order, and $\displaystyle
B_{i}(T)$ to the fourth order. Some terms of expansion, e.g., $\displaystyle
a^{\mu}u^{\nu}$, are dropped for reasons of PT-symmetries. Although, the
expansion may include higher-order terms, which we denote as
$\displaystyle\mathcal{O}(\nabla^{6})$, but in most known similar cases (for
example, systems with only acceleration (2), (3)), the expansion of the SET of
massless fields terminates at the fourth order [18, 22, 19], i.e.
$\displaystyle\mathcal{O}(\nabla^{6})=0$. It is reasonable to assume that in
the case under consideration there are also no higher-order terms.

The SETs (2) and (3) are now special cases of (5) in flat space-time limit. On
the other hand, there is the well-known SET for the (A)dS vacuum state [27,
28]222The corresponding vacuum contribution is additional to the
$\displaystyle\Lambda$-term, which is responsible for the geometric
background.

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{\text{vac}}=\frac{k}{4}R^{2}g^{\mu\nu}\,.$
(6)

The conservation equations form the basis of hydrodynamics, in particular, for
the SET, we have

$\displaystyle\displaystyle\nabla_{\mu}\langle\hat{T}^{\mu\nu}\rangle=0,\quad\langle\hat{T}^{\mu}_{\mu}\rangle=kR^{2}.$
(7)

The second equation contains the famous gravitational conformal anomaly, and
the numerical coefficient $\displaystyle k$ depends on the spin of the
microscopic constituents [27, 29, 30]

$\displaystyle\displaystyle k^{(s=0)}=\frac{1}{34560\pi^{2}}\,,\quad
k^{(s=1/2)}=\frac{11}{34560\pi^{2}}\,.$ (8)

We assume that system is in a global thermodynamic equilibrium, which implies
that the inverse temperature vector
$\displaystyle\beta_{\mu}=\frac{u_{\mu}}{T}$ satisfies Killing equation [31]

$\displaystyle\displaystyle\nabla_{\mu}\beta_{\nu}+\nabla_{\nu}\beta_{\mu}=0.$
(9)

Chosen Killing vector is to be time-like and also future-oriented, therefore
equilibrium condition possible for stationary space-times, as well as for
static, as a special case. Note that for the static AdS space-time coordinates
are global, while for the dS they cover only part of the manifold forming the
static patch. The use of the condition (9) is dictated both by the physical
formulation of the problem and by a significant simplification of the
calculations, which actually allows us to study the higher order hydrodynamic
effects in a gravitational field.

To find the covariant derivative of the SET, we need to determine the
covariant derivatives of temperature and acceleration. Using (9) and the
definition of the Riemann tensor as a commutator of derivatives we obtain

$\displaystyle\displaystyle\nabla_{\mu}\nabla_{\nu}\beta_{\alpha}=-R^{\rho}_{\>\>\mu\nu\alpha}\beta_{\rho}\,,$
(10)

as for any Killing vector. Finally, using (9),(10) and the condition
$\displaystyle u_{\mu}u^{\mu}=1$ we obtain

$\begin{cases}\nabla_{\mu}T=T^{2}\alpha_{\mu},\\\
\nabla_{\mu}\alpha_{\nu}=-T\alpha^{2}u_{\mu}u_{\nu}-\frac{R}{12\,T}(g_{\mu\nu}-u_{\mu}u_{\nu}).\end{cases}$
(11)

According to the first of the equations, the temperature gradient is the
source of acceleration, which is consistent with the well-known Luttinger
relation [32]333Thus, there are no external forces other than gravity and the
SET is covariantly conserved in (7).. To avoid unnecessarily cumbersome
expressions, we will divide the tensor into two parts with different orders of
gradients and consider them separately, which is possible, since each of the
orders forms an independent system of equations.

###  3.1 Second-order gradients

The SET in the second order in gradients has the form

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{(2)}=\left(A_{1}\alpha^{2}+A_{2}R\right)u^{\mu}u^{\nu}-\left(A_{3}\alpha^{2}+A_{4}R\right)g^{\mu\nu}+A_{5}\,\alpha^{\mu}\alpha^{\nu}.$
(12)

Combining derivatives from (11) we obtain

$\begin{cases}\nabla_{\mu}A_{i}=\frac{\partial A_{i}}{\partial
T}\nabla_{\mu}T=A^{\prime}_{i}\,T^{2}\alpha_{\mu},\\\
\nabla_{\mu}\left(u^{\mu}u^{\nu}\right)=T\alpha^{\nu},\\\
\nabla_{\mu}\left(\alpha^{\mu}\alpha^{\nu}\right)=-\Big{(}T\alpha^{2}+\frac{R}{3T}\Big{)}\alpha^{\nu},\\\
\nabla_{\mu}\alpha^{2}=-\frac{R}{6T}\alpha_{\mu}.\end{cases}$ (13)

Then, the derivative of the SET will give us

$\displaystyle\displaystyle\nabla_{\mu}\langle\hat{T}^{\mu\nu}\rangle_{(2)}=\Big{[}(A_{1}\alpha^{2}T+A_{2}RT)-(A^{\prime}_{3}\alpha^{2}T^{2}-A_{3}\frac{R}{6T}+A^{\prime}_{4}RT^{2})+(-A_{5}\alpha^{2}T-A_{5}\frac{R}{3T}+A^{\prime}_{5}\alpha^{2}T^{2})\Big{]}\alpha^{\nu}.$
(14)

Now we have to take into account that (14) is zero and
$\displaystyle\langle\hat{T}^{\mu}_{\mu}\rangle_{(2)}=0$, according to (7).
Collecting terms of the same tensor structure and taking into account that
(14) are satisfied only when each independent term is zero, we obtain the
following system of differential equations

$\begin{cases}A_{1}-A^{\prime}_{3}T-A_{5}+A^{\prime}_{5}T=0,\\\
A_{2}T^{2}+\frac{A_{3}}{6}-A^{\prime}_{4}T^{3}-\frac{A_{5}}{3}=0,\\\
A_{1}-4A_{3}+A_{5}=0,\\\ A_{2}-4A_{4}=0.\end{cases}$ (15)

Since the fields are massless and the only dimensional parameter is
temperature, we know in advance the temperature dependence for $\displaystyle
A_{i}(T)$

$\displaystyle\displaystyle A_{1}=\lambda_{1}T^{4},\quad
A_{2}=\lambda_{2}T^{2},\quad A_{3}=\lambda_{3}T^{4},\quad
A_{4}=\lambda_{4}T^{2},\quad A_{5}=\lambda_{5}T^{4}.$ (16)

where $\displaystyle\lambda_{i}$ are dimensionless constants. Therefore we
transform (15) into a system of algebraic equations

$\begin{cases}\lambda_{1}-4\lambda_{3}+3\lambda_{5}=0,\\\
\lambda_{2}+\frac{\lambda_{3}}{6}-2\lambda_{4}-\frac{\lambda_{5}}{3}=0,\\\
\lambda_{1}-4\lambda_{3}+\lambda_{5}=0,\\\
\lambda_{2}-4\lambda_{4}=0.\end{cases}$ (17)

which can be easily solved. It follows that $\displaystyle\lambda_{5}=0$,
$\displaystyle\lambda_{2}=-\frac{\lambda_{1}}{12}$,
$\displaystyle\lambda_{3}=\frac{\lambda_{1}}{4}$,
$\displaystyle\lambda_{4}=-\frac{\lambda_{1}}{48}$ and finally (12) has the
form

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{(2)}=A\left(a^{2}-\frac{R}{12}\right)T^{2}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)},$
(18)

where $\displaystyle A=\frac{\lambda_{1}}{4}$.

Also, it is necessary to pay some attention to the zeroth order. Since in the
zeroth order the trace anomaly is absent, simply from the conservation
relations we obtain the standard expression

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{(0)}=\sigma
T^{4}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)},$ (19)

in accordance with the formulas (2), (3). The coefficient
$\displaystyle\sigma$ refers us to the Stephan-Boltzmann’s law.

###  3.2 Fourth-order gradients

Now we write out the SET in the fourth order in gradients

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{(4)}=\left(B_{1}\alpha^{4}+B_{2}\alpha^{2}R+B_{3}R^{2}\right)u^{\mu}u^{\nu}-\left(B_{4}\alpha^{4}+B_{5}\alpha^{2}R+B_{6}R^{2}\right)g^{\mu\nu}+\left(B_{7}\alpha^{2}+B_{8}R\right)\alpha^{\mu}\alpha^{\nu}.$
(20)

Using (13) we find the expression for the covariant derivative of the SET

$\displaystyle\displaystyle\nabla_{\mu}\langle\hat{T}^{\mu\nu}\rangle_{(4)}$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\Bigg{[}\Big{(}B_{1}\alpha^{4}T+B_{2}\alpha^{2}RT+B_{3}R^{2}T\Big{)}-\Big{(}B^{\prime}_{4}\alpha^{4}T^{2}-B_{4}\alpha^{2}\frac{R}{3T}+B^{\prime}_{5}\alpha^{2}RT^{2}-B_{5}\frac{R^{2}}{6T}+B^{\prime}_{6}R^{2}T^{2}\Big{)}+$
(21)
$\displaystyle\displaystyle+\Big{(}B^{\prime}_{7}\alpha^{4}T^{2}-B_{7}\alpha^{4}T-B_{7}\alpha^{2}\frac{R}{2T}+B^{\prime}_{8}\alpha^{2}RT^{2}-B_{8}\alpha^{2}RT-
B_{8}\frac{R^{2}}{3T}\Big{)}\Bigg{]}\alpha^{\nu}.$

As in the previous section, substituting (21) into (7) we have the system of
differential equations

$\begin{cases}B_{2}T^{2}+\frac{B_{4}}{3}-B^{\prime}_{5}T^{3}-\frac{B_{7}}{2}+B^{\prime}_{8}T^{3}-B_{8}T^{2}=0,\\\
B_{1}-B^{\prime}_{4}T+B^{\prime}_{7}T-B_{7}=0,\\\
B_{3}T^{2}+\frac{B_{5}}{6}-B^{\prime}_{6}T^{3}-\frac{B_{8}}{3}=0,\\\
B_{1}-4B_{4}+B_{7}=0,\\\ B_{2}-4B_{5}+B_{8}=0,\\\ B_{3}-4B_{6}=k.\end{cases}$
(22)

but now it includes the trace anomaly. Again, we can move on to dimensionless
constants

$\displaystyle\displaystyle B_{1}=b_{1}T^{4},\quad B_{2}=b_{2}T^{2},\quad
B_{3}=b_{3},\quad B_{4}=b_{4}T^{4},\quad B_{5}=b_{5}T^{2},\quad
B_{6}=b_{6},\quad B_{7}=b_{7}T^{4},\quad B_{8}=b_{8}T^{2},\,\,$ (23)

where $\displaystyle b_{i}=\text{const}$, after that we are left with a system
of algebraic equations

$\begin{cases}b_{2}=-b_{1}/6-b_{8},\\\ b_{3}=b_{1}/144+b_{8}/3,\\\
b_{4}=b_{1}/4,\\\ b_{5}=b_{2}/4+b_{8}/4=-b_{1}/24,\\\
b_{6}=b_{3}/4-k/4=b_{1}/576+b_{8}/12-k/4,\\\ b_{7}=0.\end{cases}$ (24)

Therefore (20) can be rewritten as

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{(4)}$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
B\left(a^{2}-\frac{R}{12}\right)^{2}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)}+\frac{k}{4}R^{2}g^{\mu\nu}+b_{8}\bigg{[}\frac{R^{2}}{12}(4u^{\mu}u^{\nu}-g^{\mu\nu})+a^{2}R(\frac{a^{\mu}a^{\nu}}{a^{2}}-u^{\mu}u^{\nu})\bigg{]},$
(25)

where $\displaystyle B=\frac{b_{1}}{4}$. From the relativistic covariance,
(25) should transform to (6) in the vacuum state, which is possible for the
term $\displaystyle R\,a^{\mu}a^{\nu}$ only when $\displaystyle b_{8}=0$, and
therefore, finally we obtain

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{(4)}=B\left(a^{2}-\frac{R}{12}\right)^{2}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)}+\frac{k}{4}R^{2}g^{\mu\nu}.$
(26)

Combining (18), (19) and (26) together, we obtain the final full formula

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle=\Bigg{[}\sigma
T^{4}+A\left(a^{2}-\frac{R}{12}\right)T^{2}+B\left(a^{2}-\frac{R}{12}\right)^{2}\Bigg{]}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)}+\frac{k}{4}R^{2}g^{\mu\nu}.$
(27)

Coefficients $\displaystyle A$ and $\displaystyle B$ for spins $\displaystyle
s=0$ and $\displaystyle s=1/2$ can be defined from (2), (3). Taking into
account equality $\displaystyle a^{\mu}a_{\mu}=-|a|^{2}$, for the case of
$\displaystyle s=0$ expression (2) will take the form

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{s=0}=\Bigg{[}\frac{\pi^{2}}{90}T^{4}-\frac{1}{1440\pi^{2}}\left(|a|^{2}+\frac{R}{12}\right)^{2}\Bigg{]}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)}+\frac{1}{960\pi^{2}}\Big{(}\frac{R}{12}\Big{)}^{2}g^{\mu\nu}\,.$
(28)

And for the case of $\displaystyle s=1/2$ expression (3) will change as

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle_{s=1/2}=\Bigg{[}\frac{7\pi^{2}}{180}T^{4}+\frac{1}{72}\left(|a|^{2}+\frac{R}{12}\right)T^{2}-\frac{17}{2880\pi^{2}}\left(|a|^{2}+\frac{R}{12}\right)^{2}\Bigg{]}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)}+\frac{11}{960\pi^{2}}\Big{(}\frac{R}{12}\Big{)}^{2}g^{\mu\nu}\,.\,\,\,\,\,\,\,$
(29)

The formula (29) matches the result of [20] (that was obtained using quantum
field theory in AdS space-time).

##  4 Discussion

###  4.1 Generalized Unruh effect: accelerated observer in (A)dS space-time

Let us analyze the consequences of (27). First, we should recall that in the
limit $\displaystyle R\to 0$ the usual Unruh effect in flat space imposes the
condition $\displaystyle\langle\hat{T}^{\mu\nu}\rangle(T=T_{U})=0$, as
discussed in Section 2. This allows us to immediately obtain the connection
between $\displaystyle\sigma,A$ and $\displaystyle B$, and (27) takes the form

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle=\Bigg{[}\sigma
T^{4}+A\left(a^{2}-\frac{R}{12}\right)T^{2}+\left(\frac{A}{4\pi^{2}}-\frac{\sigma}{16\pi^{4}}\right)\left(a^{2}-\frac{R}{12}\right)^{2}\Bigg{]}\Big{(}4u^{\mu}u^{\nu}-g^{\mu\nu}\Big{)}+\frac{k}{4}R^{2}g^{\mu\nu}\,,$
(30)

which, of course, both (28) and (29) satisfy. On the other hand, the vacuum
SET should have the form (6). From the general relativistic covariance, the
vacuum SET should have the form (6) in any reference frame, in particular,
accelerated. Now we see that (30) at $\displaystyle T_{UR}$ leads to (6)

$\displaystyle\displaystyle\langle\hat{T}^{\mu\nu}\rangle(T=T_{UR})=\frac{k}{4}R^{2}g^{\mu\nu}\,.$
(31)

This means that the value of temperature $\displaystyle T_{UR}$ corresponds to
vacuum, which therefore, is perceived by the accelerated observer as a heat
bath with a temperature (1). Thus, the generalized Hawking-Unruh effect for
accelerated observer in (A)dS space-time [16, 17, 33] actually follows from
the basic hydrodynamic equations (and the usual Unruh effect in flat space).
This is a somewhat unexpected result, since our analysis essentially concerned
only hydrodynamics in four-dimensional space-time, while (1) is associated
with a five-dimensional acceleration $\displaystyle a_{5}$.

###  4.2 Duality between hydrodynamics and gravity

The result obtained add new elements to the duality between hydrodynamics and
gravity previously discussed in [12]. At first glance, these two approaches
are essentially different - gravitational interaction is fundamental. It is
described, for example, by the vertex $\displaystyle\delta
g_{\mu\nu}T^{\mu\nu}$ in action, where $\displaystyle\delta g_{\mu\nu}$ is the
fluctuation of the space-time metrics. On the other hand, acceleration effects
correspond to the macroscopic interaction $\displaystyle\alpha_{\mu}K^{\mu}$
in the density operator [25] with boost operator $\displaystyle K^{\mu}$.
However, the analysis of conservation relations links the two types of
effects. Indeed, for the SET (5), according to (27) we obtain the relations

$\displaystyle\displaystyle A_{1}=4A_{3}=-12A_{2}T^{2}=-48A_{4}T^{2}\,,$
$\displaystyle\displaystyle
B_{1}=4B_{4}=-6B_{2}T^{2}=-24B_{5}T^{2}=144B_{3}T^{4}=576B_{6}T^{4}+144kT^{4}\,.$
(32)

In this case, $\displaystyle A_{1}$, $\displaystyle B_{1}$, $\displaystyle
A_{3}$ and $\displaystyle B_{4}$ characterize the contribution, associated
only with the acceleration $\displaystyle a_{\mu}$, which survives in the
limit $\displaystyle R\to 0$, while $\displaystyle A_{2}$, $\displaystyle
B_{2}$, $\displaystyle B_{3}$, $\displaystyle A_{4}$, $\displaystyle B_{5}$
and $\displaystyle B_{6}$ describe effects with the finite curvature
$\displaystyle R$ (or mixed hydro-gravity effects). As follows from (32),
these two classes of effects are related to each other.

Since the hydrodynamic gradient expansion and conservation relations are quite
general, our analysis is to be valid for any fluid with massless constituents
with an arbitrary spin. However, at temperatures below the Unruh temperature
$\displaystyle T<T_{U}$ quantum phase transition occurs, a detailed analysis
of which is given in [34], and as the result (3) changes. It should be
expected that a similar constraint holds for (28), (29), and (30) when
$\displaystyle T<T_{UR}$.

## Conclusion

We have considered a relativistic accelerated fluid of massless particles with
an arbitrary spin in a thermodynamic equilibrium in a four-dimensional (A)dS
space-time. We have derived the stress-energy tensor, which takes into account
the effects of both acceleration and constant curvature.

Conservation of the stress-energy tensor and general relativistic covariance
allow us to find the linear equations relating the transport coefficients in
flat and curved space-times. We have verified these relations directly in the
particular case of the Dirac field in the AdS space-time.

The immediate consequence of the obtained formulas is a novel confirmation of
the generalized Unruh effect for accelerated systems in (A)dS space-time: the
temperature of the vacuum measured by the accelerated observer is determined
by acceleration in a flat five-dimensional space-time.

Acknowledgements

The authors are thankful to D. V. Fursaev for stimulating discussions. The
work of G. Yu. P. and V. I. Z. is supported by Russian Science Foundation
Grant No. 22-22-00664. The work of V. I. Z. is partially supported by Grant
No. 0657-2020-0015 of the Ministry of Science and Higher Education of Russia.

## References

  * [1] Thomas Schäfer and Derek Teaney.  Nearly Perfect Fluidity: From Cold Atomic Gases to Hot Quark Gluon Plasmas.  Rept. Prog. Phys., 72:126001, 2009.
  * [2] L Landau and E Lifshitz.  Fluid Mechanics: Landau and Lifshitz: Course of Theoretical Physics.  09 2013.
  * [3] Pavel Kovtun.  Lectures on hydrodynamic fluctuations in relativistic theories.  J. Phys. A, 45:473001, 2012.
  * [4] Dam T. Son and Piotr Surowka.  Hydrodynamics with Triangle Anomalies.  Phys. Rev. Lett., 103:191601, 2009.
  * [5] Valentin I. Zakharov.  Chiral Magnetic Effect in Hydrodynamic Approximation.  2012\.  [Lect. Notes Phys.871,295(2013)].
  * [6] Shi-Zheng Yang, Jian-Hua Gao, and Zuo-Tang Liang.  Constraining Non-Dissipative Transport Coefficients in Global Equilibrium.  Symmetry, 14(5):948, 2022.
  * [7] M. Buzzegoli.  Thermodynamic equilibrium of massless fermions with vorticity, chirality and electromagnetic field.  11 2020.
  * [8] Karl Landsteiner, Eugenio Megias, and Francisco Pena-Benitez.  Gravitational Anomaly and Transport.  Phys. Rev. Lett., 107:021601, 2011.
  * [9] Michael Stone and Jiyoung Kim.  Mixed Anomalies: Chiral Vortical Effect and the Sommerfeld Expansion.  Phys. Rev., D98(2):025012, 2018.
  * [10] Karl Landsteiner, Sergio Morales-Tejera, and Pablo Saura-Bastida.  Anomalous transport from geometry.  Phys. Rev. D, 107(12):125003, 2023.
  * [11] Valeri P. Frolov, Alex Koek, Jose Pinedo Soto, and Andrei Zelnikov.  Chiral anomalies in black hole spacetimes.  Phys. Rev. D, 107(4):045009, 2023.
  * [12] G. Yu. Prokhorov, O. V. Teryaev, and V. I. Zakharov.  Hydrodynamic Manifestations of Gravitational Chiral Anomaly.  Phys. Rev. Lett., 129(15):151601, 2022.
  * [13] W. G. Unruh.  Notes on black hole evaporation.  Phys. Rev., D14:870, 1976.
  * [14] G. E. Volovik.  On the global temperature of the Schwarzschild–de Sitter spacetime.  Pisma Zh. Eksp. Teor. Fiz., 118(1):5–6, 2023.
  * [15] G. W. Gibbons and S. W. Hawking.  Cosmological Event Horizons, Thermodynamics, and Particle Creation.  Phys. Rev. D, 15:2738–2751, 1977.
  * [16] H. Narnhofer, I. Peter, and Walter E. Thirring.  How hot is the de Sitter space?  Int. J. Mod. Phys. B, 10:1507–1520, 1996.
  * [17] Stanley Deser and Orit Levin.  Accelerated detectors and temperature in (anti)-de Sitter spaces.  Class. Quant. Grav., 14:L163–L168, 1997.
  * [18] George Y. Prokhorov, Oleg V. Teryaev, and Valentin I. Zakharov.  Thermodynamics of accelerated fermion gases and their instability at the Unruh temperature.  Phys. Rev., D100(12):125009, 2019.
  * [19] Andrea Palermo, Matteo Buzzegoli, and Francesco Becattini.  Exact equilibrium distributions in statistical quantum field theory with rotation and acceleration: Dirac field.  6 2021.
  * [20] Victor E. Ambrus and Elizabeth Winstanley.  Vortical effects for free fermions on anti-de Sitter space-time.  7 2021.
  * [21] George Y. Prokhorov, Oleg V. Teryaev, and Valentin I. Zakharov.  Unruh effect for fermions from the Zubarev density operator.  Phys. Rev., D99(7):071901(R), 2019.
  * [22] Georgy Y. Prokhorov, Oleg V. Teryaev, and Valentin I. Zakharov.  Unruh effect universality: emergent conical geometry from density operator.  JHEP, 03:137, 2020.
  * [23] M. Buzzegoli, E. Grossi, and F. Becattini.  General equilibrium second-order hydrodynamic coefficients for free quantum fields.  JHEP, 10:091, 2017.  [Erratum: JHEP07,119(2018)].
  * [24] V. I. Zakharov, G. Y. Prokhorov, and O. V. Teryaev.  Acceleration and rotation in quantum statistical theory.  Phys. Scripta, 95(8):084001, 2020.
  * [25] F. Becattini.  Thermodynamic equilibrium with acceleration and the Unruh effect.  Phys. Rev., D97(8):085013, 2018.
  * [26] L. D. Landau and E. M. Lifschits.  The Classical Theory of Fields, volume Volume 2 of Course of Theoretical Physics.  Pergamon Press, Oxford, 1975.
  * [27] Don N. Page.  Thermal Stress Tensors in Static Einstein Spaces.  Phys. Rev. D, 25:1499, 1982.
  * [28] J. S. Dowker and Raymond Critchley.  Effective Lagrangian and Energy Momentum Tensor in de Sitter Space.  Phys. Rev. D, 13:3224, 1976.
  * [29] J. S. Dowker and Raymond Critchley.  The Stress Tensor Conformal Anomaly for Scalar and Spinor Fields.  Phys. Rev. D, 16:3390, 1977.
  * [30] M. J. Duff.  Ultraviolet divergences in extended supergravity.  In First School on Supergravity, 1 1982.
  * [31] F. Becattini.  Thermodynamic equilibrium in relativity: four-temperature, Killing vectors and Lie derivatives.  Acta Phys. Polon. B, 47:1819, 2016.
  * [32] J. M. Luttinger.  Theory of Thermal Transport Coefficients.  Phys. Rev., 135:A1505–A1514, 1964.
  * [33] Stanley Deser and Orit Levin.  Mapping Hawking into Unruh thermal properties.  Phys. Rev. D, 59:064004, 1999.
  * [34] Georgy Yu. Prokhorov, Oleg V. Teryaev, and Valentin I. Zakharov.  Novel phase transition at the Unruh temperature.  4 2023.


# Towards Light Weight Object Detection System

###### Abstract

Transformers are a popular choice for classification tasks and as backbones
for object detection tasks. However, their high latency brings challenges in
their adaptation to light weight object detection systems. We present an
approximation of the self-attention layers used in the transformer
architecture. This approximation reduces the latency of the classification
system while incurring minimal loss in accuracy. We also present a method that
uses a transformer encoder layer for multi-resolution feature fusion. This
feature fusion improves the accuracy of the state-of-the-art light weight
object detection system without significantly increasing the number of
parameters. Finally, we provide an abstraction for the transformer
architecture called Generalized Transformer (gFormer) that can guide the
design of novel transformer-like architectures.

Index Terms—  Vision transformer, self-attention, object detection, deep
neural networks

##  1 Introduction

Convolutional neural networks (CNNs) [1] have been widely used as backbones
for object detection systems. MobileNets [2] use depthwise separable
convolutions to develop light weight CNNs. MobileNetV2 further improves
MobileNets using inverted residuals and linear bottlenecks. It also introduced
efficient ways of applying depthwise separable convolutions to the heads of
Single Shot MultiBox Detector (SSD) [3], which resulted in the light weight
object detection system, SSDLite. Recently, Vision Transformers (ViTs) [4] are
gaining popularity due to their ability to extract global information.
However, they lack the spatial inductive biases present in CNNs. MobileViT [5]
presented a hybrid architecture based on CNNs and ViTs that leverages the
inductive biases of CNNs and also includes global information through ViTs.
MobileViT achieves impressive performance on the ImageNet-1k classification
dataset [6], and its main disadvantage is high latency.

In this work, we propose _Convolution as Transformer_ (CAT): a module that
approximates the self-attention layer in transformers. CAT has low latency and
thus can be used in light weight systems for image classification and object
detection. We replace expensive transformer blocks used in MobileViT with our
CAT blocks, and we show that they are competitive with the self-attention
modules for image classification tasks. Moreover, CAT blocks have complexity
$O(n\times d)$, unlike self-attention that has complexity $O(n^{2}\times d)$,
where $n$ is the sequence length, and $d$ is feature vector size.

Existing light weight systems for object detection [5, 3] mainly consist of a
backbone to extract features from images, followed by heads to extract
features from multiple output resolutions. Predictions on object label and
localization are made directly from these multi-scale features. It is
therefore challenging to learn the relationship between these features from
multiple scales, carrying different semantic information.

To overcome this, we propose the module _Transformer Encoder as Feature
Fusion_ (TAFF): a single layered transformer encoder [7] which fuses features
from multiple resolutions at different scales. We show empirically that the
feature fusion performed by TAFF improves the accuracy of state-of-the-art
object detection models like MobileViT [8].

Finally, we propose _Generalized TransFormer_ (gFormer): a general abstract
architecture that binds multiple variations of attention and transformer
mechanisms under a common umbrella. From this perspective, MetaFormer [9],
Transformer [7], Squeeze and Excitation Networks [10, 11], and our CAT block
are all variations of gFormer.

##  2 System

###  2.1 Convolution as Transformer (CAT)

The baseline for this architecture is the MobileViT architecture [8] that uses
MobileNetV2 blocks along with MobileViT blocks that contain transformer layers
for extracting global information. We refer to [8] for the full architecture
and only show the MobileViT block in Fig. 1.

Fig. 1: MobileViT block used in the MobileVit [8] Fig. 2: Convolution As a
Transformer (CAT) block

The MobileViT architecture extracts global information with transformers. The
major disadvantage of the above method is that it has high latency because of
the self-attention layers used inside the transformer as convolutions block.
We hypothesize and prove empirically that we can extract the global
information using simpler functions (with slight loss in accuracy) that has
lower latency and lower number of computations. Thus, we propose Convolution
as Transformer (CAT) blocks (Fig. 2) that approximates this self-attention for
global feature extraction. We propose to use the following function to
approximate transformers by convolution blocks:

  * •

Depthwise separable convolutional filter to extract global information:
global_information == dep_sep_conv(x) This is a combination of depthwise
convolution with kernel size of (h, w) and a pointwise convolution. This can
also be interpreted as a combination of spatial MLP and channel MLP where
spatial MLP extracts information from spatial domain and channel MLP extracts
information from channel dimension. Thus, the output vector represents
approximate global information from spatial and channel dimension.

  * •

Broadcast global information to same shape as intermediate feature map:
y = reshape(global_information, (H, W, D))

  * •

Elemenentwise product between global information and intermediate feature map:
x = elementwise_product(x, y). Here, $\otimes$ represents elementwise product
between two tensors.

###  2.2 Transformer Encoder as Feature Fusion (TAFF)

Fig. 3 shows the architecture of our TAFF block:

Fig. 3: TAFF module for feature fusion from multiple scales

We extract features from multiple resolution intermediate feature maps. We
extract $d$ dimensional features from each anchor box from multiple resolution
feature maps, we then use Transformer encoder layer that first projects these
features $n*d$ feature tensor into key, query and values tensors. It then
applies a multi-headed attention to these features giving each network head an
ability to not only look at the feature vector at the corresponding location
but also look at other feature vectors to make an informed decision. For
example the locations at the lower level can look at the features of the
semantically higher layers to make a good prediction.

###  2.3 gFormer

MetaFormer [9] abstracts different variations of the transformer architectures
into a general framework. We further generalize multiple variations of the
attention architecture into a common framework that even includes the squeeze
and excite networks. Fig. 4 shows the general transformer (gFormer)
architecture:

Fig. 4: gFormer: generalized abstraction for transformer networks

Where $\oplus$ represents summation, and $\otimes$ represents any pairwise
interaction function (e.g. elementwise product). Thus, following architectures
become special cases of the gFormer abstraction:

  * •

Transformer [7]: Transformer captures pairwise interaction using multi-headed
attention, so we don’t need residual-3 and last $\otimes$. Thus, if we remove
residual-3, use multi-headed attention for spatial mixing, and use MLP for
channel mixing with normalization layers, we recover the Transformer
architecture.

  * •

MetaFormer [9]: If we remove residual-3 and pairwise interaction function we
obtain the MetaFormer.

  * •

CAT: If we remove residual-1 and residual-2, use depthwise convolution or
spatial MLP for spatial mixing, pointwise convolution or MLP for channel
mixing, repeat the global vector to input tensor shape and take the Hadamard
product between two tensors, we arrive at our CAT block.

  * •

Squeeze and excite networks [10, 11]: If we remove residual-1, residual-2, use
global pooling as spatial mixing, use a linear layer with ”MLP-ReLU-MLP-
sigmoid” as channel mixing, repeat the output of this linear layer to original
input tensor shape and take Hadamard product, we obtain squeeze and excite
networks.

  * •

MLP-Mixer [11]: If we remove residual-3, and pairwise interaction function,
use MLP for spatial mixing and MLP for channel mixing, we obtain the MLP-
Mixer.

The gFormer abstraction helps us create new architectures that are optimized
for different needs and still reap the benefits of transformers for capturing
global information.

##  3 Experiments

###  3.1 Classification with CAT

For classification experiments, we use a subset of ImageNet-1k [6] dataset
which consists of 100 balanced classes from ImageNet-1k keeping the original
train-validation split. We used a subset of the dataset because of
computational overhead of training on whole ImageNet-1k. The training dataset
consists of 130k images, while the validation dataset consists of 5k images.
We follow [5] and report accuracy on the validation dataset. We used PyTorch
[12] for our experiments. We trained our model up to 300 epochs with the
stochastic gradient descent (SGD) algorithm with weight decay of $4e-5$ and
momentum of 0.5, at a batch size of 64 per GPU on 4 Nvidia RTX 2080Ti GPUs. We
used the Cosine Annealing scheduler with an initial learning rate of 0.05,
increasing to 0.4 within 7500 iterations, and ultimately decreasing to a
minimum of $2e-4$. We use an input image resolution of $256\times 256$. We
used the Swish activation function [13] as our default activation function.

We compare our results with MobileViT as a baseline method. We also modified
the token mixing algorithm based on Fourier transforms called FNet [14]
architecture used in natural language processing domain and replaced the
MobileViT block with this mechanism. Note that we replaced the multiple
transformer layers used in MobileViT with a single layer of our CAT block.
Also, our CAT block doesn’t use any feedforward layers used by the transformer
layers in MobileViT.

###  3.2 Detection with TAFF

We use COCO [15] dataset for object detection experiments with the same
setting and same hyperparameters as MobileViT [5]. The input image resolution
is $320\times 320$. We evaluated the performance on the validation dataset
using mAP@IOU 0.50:0.05:0.95. We first train MobileViT with SSDLite as the
baseline method.

We then extract features from MobileViT and SSDLite and fuse features from
multiple layers using our TAFF module. The fused features are used to make the
prediction for the class and bounding box. Note that we didn’t even use
positional encoding for the feature fusion that helps in keeping the small
memory footprint when deployed on mobile devices. It’s interesting that the
single layer of transformer encoder was able to fuse features from multiple
scales and multiple anchor boxes. We hypothesize that the projection matrices
in the transformer encoder project features from multiple scales and anchor
boxes to a common domain suitable for feature fusion.

##  4 Results

###  4.1 Classification with CAT

Table 1 shows the accuracy, number of parameters (in Million), number of
floating point operations (FLOPS), and latency (measured for a single image on
Nvidia RTX 2080 Ti by averaging over 1000 runs). The results show that our CAT
block closely approximates the self-attention module of the MobileViT
architecture with slight decrease in accuracy but at almost half number of
parameters, half FLOPs, and half latency.

We also modified the FNet architecture [14] used in the natural language
processing (NLP) domain to approximate the self-attention layers in the
MobileViT architecture and obtained good results. This demonstrates that FNet
is also a viable approach for computer vision along with NLP domain.

Table 1: CAT classification performance on ImageNet dataset

Model Accuracy Parameters FLOPS Latency Complexity MobileViT 84.84 5.7 M 4 G
11.17 ms $O(n^{2}d)$ CAT 83.84 2.4 M 1.27 G 4.69 ms $O(nd)$ FNet 84.34 2 M
1.25 G 3.87 ms $O(nlog(n)d)$

Table 1 also shows the advantage of our method compared to other methods in
terms of run-time complexity of self-attention module. We can see that our
method has linear complexity $O(n)$ with respect to sequence length, while
other methods have either $O(nlogn)$ or $O(n^{2})$ complexity.

###  4.2 Detection with TAFF

Table 2 shows the effectiveness of our TAFF module in object detection tasks
on the COCO validation dataset, which is a challenging object detection
dataset. We demonstrate that adding this module on top of the SSD architecture
with the MobileViT backbone significantly increases the accuracy of object
detection by about 2.5mAP on the COCO validation dataset, while the number of
parameters only grew by $7\%$

Table 2: TAFF detection performance on COCO dataset Method | mAP | Parameters
---|---|---
MobileNetv1 | 22.2 | 5.1 M
MobileNetv2 | 22.1 | 4.3 M
MobileNetv3 | 22.0 | 4.9 M
MobileViT | 27.7 | 5.7 M
TAFF | 30.1 | 6.1 M

##  5 Conclusion

In this paper, we have proposed the CAT block that can decrease the latency
and FLOPs of transformer-based backbones such as MobileViT, while increasing
the inference speed using simple approximation functions composed of
depthwise-separable convolution and Hadamard product. Then, we have proposed
the TAFF module that improves the accuracy of existing anchor-based object
detection systems by fusing features from multiple scales. Finally, we have
presented a general framework called gFormer that helps us design new
architectures.

There are multiple ways to combine these ideas with existing systems to
develop further light weight, accurate, and fast object detection systems. One
such approach would be fusing these ideas to anchor-free object detection
systems. For example, our CAT block and TAFF module can be integrated with
DETR [16]. We explore this approach in future research.

## References

  * [1] Yann LeCun, Yoshua Bengio, et al.,  “Convolutional networks for images, speech, and time series,”  The handbook of brain theory and neural networks, vol. 3361, no. 10, pp. 1995, 1995.
  * [2] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam,  “Mobilenets: Efficient convolutional neural networks for mobile vision applications,”  arXiv preprint arXiv:1704.04861, 2017.
  * [3] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg,  “Ssd: Single shot multibox detector,”  in European conference on computer vision. Springer, 2016, pp. 21–37.
  * [4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in International Conference on Learning Representations, 2021.
  * [5] Sachin Mehta and Mohammad Rastegari,  “Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer,”  in International Conference on Learning Representations, 2022.
  * [6] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.,  “Imagenet large scale visual recognition challenge,”  International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
  * [7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [8] Sachin Mehta and Mohammad Rastegari,  “Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer,”  arXiv preprint arXiv:2110.02178, 2021.
  * [9] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan,  “Metaformer is actually what you need for vision,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10819–10829.
  * [10] Jie Hu, Li Shen, and Gang Sun,  “Squeeze-and-excitation networks,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7132–7141.
  * [11] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al.,  “Mlp-mixer: An all-mlp architecture for vision,”  Advances in Neural Information Processing Systems, vol. 34, pp. 24261–24272, 2021.
  * [12] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.,  “Pytorch: An imperative style, high-performance deep learning library,”  Advances in neural information processing systems, vol. 32, 2019\.
  * [13] Prajit Ramachandran, Barret Zoph, and Quoc V Le,  “Searching for activation functions,”  arXiv preprint arXiv:1710.05941, 2017.
  * [14] Ilya Eckstein, James Patrick Lee-Thorp, Joshua Ainslie, and Santiago Ontanon, Eds.,  FNet: Mixing Tokens with Fourier Transforms, 2022.
  * [15] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick,  “Microsoft coco: Common objects in context,”  in European conference on computer vision. Springer, 2014, pp. 740–755.
  * [16] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko,  “End-to-end object detection with transformers,”  in European conference on computer vision. Springer, 2020, pp. 213–229.