# SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers

Alberto Marchisio1,*, Davide Dura2,*, Maurizio Capra2,*, Maurizio Martina2,
Guido Masera2, Muhammad Shafique3
*These authors contributed equally to this work. 1Technische Universität Wien, Vienna, Austria 2Politecnico di Torino, Turin, Italy 3New York University, Abu Dhabi, UAE Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Transformers’ compute-intensive operations pose enormous challenges for their
deployment in resource-constrained EdgeAI / tinyML devices. As an established
neural network compression technique, quantization reduces the hardware
computational and memory resources. In particular, fixed-point quantization is
desirable to ease the computations using lightweight blocks, like adders and
multipliers, of the underlying hardware. However, deploying fully-quantized
Transformers on existing general-purpose hardware, generic AI accelerators, or
specialized architectures for Transformers with floating-point units might be
infeasible and/or inefficient.

Towards this, we propose SwiftTron, an efficient specialized hardware
accelerator designed for Quantized Transformers. SwiftTron supports the
execution of different types of Transformers’ operations (like Attention,
Softmax, GELU, and Layer Normalization) and accounts for diverse scaling
factors to perform correct computations. We synthesize the complete SwiftTron
architecture in a $65$ nm CMOS technology with the ASIC design flow. Our
Accelerator executes the RoBERTa-base model in $1.83\ ns$, while consuming
$33.64\ mW$ power, and occupying an area of $273\ mm^{2}$. To ease the
reproducibility, the RTL of our SwiftTron architecture is released at
https://github.com/albertomarchisio/SwiftTron.

###### Index Terms:

Hardware Architecture, Transformers, Machine Learning, ASIC, Quantization,
Attention, Softmax, Layer Normalization, GELU.

##  I Introduction

Among advanced Machine Learning (ML) models, Transformers are becoming
mainstream for several applications like natural language processing and
computer vision. However, they involve several compute-intensive operations
like Multi-Head Self Attention and Layer Normalization with massive data
streams. Hence, their execution is highly power-consuming when conducted on
general-purpose hardware. Specialized architectures would be desirable to
accelerate the execution of Transformers’ operations and improve the energy-
efficiency.

Several types of ML accelerators have been recently integrated with the most
common chips to execute massive matrix multiplications that are typical in
convolutional and fully-connected layers [1][2][3][4][5]. However, such
generic architectures do not support some Transformer-specific operations,
such as Attention, Softmax, Gaussian Error Linear Unit (GELU), and Layer
Normalization. Hence, the neural accelerators need to be tailored for such
unique operations involved in Transformers. Towards this, some architectures
have recently been proposed. For instance, OPTIMUS [6] optimizes the execution
of matrix multiplications in transformers, and A3 [7] accelerates the
execution of the Attention operation. However, these architectures only
execute a certain part of a given Transformer, and do not provide a holistic
acceleration platform for the complete Transformer. Note that other functions
like Softmax, GELU, and Layer Normalization involve nonlinear operations that
cannot be easily implemented in integer arithmetics. A common way of handling
these operations is to iteratively quantize the input to compute matrix
multiplication with integers and de-quantize the intermediate results to
calculate the nonlinear operations in floating point arithmetic [8]. A
simplified scheme of this method is depicted in Figure 1a. A similar approach
is also adopted in the recent Transformer Engine of the NVIDIA H100 Tensor
Core GPU [9], which implements the operations in FP8 arithmetic. However, this
approach would require the design of several floating point arithmetic logic
units with significant resource overheads compared to the integer
counterparts.

Figure 1: Simplified diagrams showing the computation flows for (a) the
Quantization-Dequantization scheme of [8], and (b) the quantization-based
scheme adopted in this work that employs Requantization blocks.

As a motivating experimental analysis comparing different arithmetics, we
synthesize an INT8-adder, an INT8-multiplier, an FP32-adder, and an
FP32-multiplier [10] in a 65 nm CMOS technology node with the Synopsys Design
Compiler and evaluate latency, power, and area. Figure 2 shows the respective
overhead of FP32 arithmetic operators compared to their respective INT8
implementations. These experiments show that the potential savings are about
one order of magnitude.

Figure 2: Latency, power, and area overhead of a single adder and a single
multiplier implemented in FP32 arithmetic, compared to their respective INT8
implementations.

On the other hand, computing nonlinear operations with quantized integers
without incurring significant accuracy loss is nontrivial. The state-of-the-
art in this regard is represented by the I-BERT [11], a framework that
implements specific approximations to execute all the nonlinear operations of
the Transformers with integer-only arithmetic. However, this work implemented
the Transformers on general-purpose hardware, i.e., GPU. A specialized
accelerator executing all the Transformer layers, including nonlinear
operations, using only efficient integer arithmetic is missing, and highly
required.

###  I-A Our Novel Contributions

The above-discussed limitations motivate us to propose SwiftTron, an efficient
hardware accelerator that executes quantized Transformers with integer-only
operations (see Figure 1b for our integer-only arithmetic flow), while
focusing on multiple different operations of a given Transformer to provide a
high degree of performance/energy efficiency. Our contributions are discussed
in the following list (see Figure 3).

Figure 3: Overview of our novel contributions in this work.

  * •

We design the SwiftTron hardware architecture, a specialized accelerator
composed of several hardware units to execute different operations of the
Transformers. (Section III)

  * •

We design and implement a quantization scheme for Transformers with scaling
factors to correctly execute the linear operations with INT8 and the nonlinear
operations with INT32 arithmetics. (Section III-A)

  * •

We synthesize the complete SwiftTron architecture in a 65 nm CMOS technology
node with the Synopsys Design Compiler and conduct gate-level simulations to
measure the area and power consumption. (Section IV)

  * •

We compare the key features of our accelerator with the related work to
highlight that our SwiftTron is the first architecture that complies with all
the desired features. (Section IV-D)

  * •

Towards encouraging fast advancements in the neural hardware accelerator and
ML community, and to ease the reproducibility of our experiments, we open-
source the complete RTL of our SwiftTron accelerator architecture at
https://github.com/albertomarchisio/SwiftTron.

##  II Background and Related Work

###  II-A Transformer Models

The Transformer network has been introduced in [12], which is the reference
point of the related works and subsequent models. Transformers are formed of
two main blocks, the encoder and the decoder. They are composed of the
following layers111Note that, since there is little difference between the
encoder and the decoder, the composition of their layers is very similar.:

  * •

Multi-Head Self Attention (MHSA):

    * –

Linear Transformation to compute Query ($Q$), Key ($K$), and Value ($V$)
matrices for each head.

    * –

Attention:

      * *

$Q\cdot K^{T}$ multiplication, where $K^{T}$ denotes the transposed matrix $K$

      * *

$Softmax(Q\cdot K^{T})\cdot V$ multiplication

    * –

Linear Transformation after concatenating every head Attention output.

  * •

Residual Connection & Layer Normalization in the MHSA

  * •

Feed-Forward Network (FFN):

    * –

Linear Transformation

    * –

Activation Function

    * –

Linear Transformation

  * •

Residual Connection & Layer Normalization in the FFN

Figure 4: Architectural model of the Transformer [12], where inputs and
outputs are taken after the positional encoding operations.

When repeated multiple times, these layers form the structure of the encoder
and the decoder, as shown in Figure 4. Note that the typical activation
functions for Transformers222There exist a few other options, but these are
the most commonly used activation functions in Transformers. are the ReLU [13]
and the GELU [14]. The baseline Transformer [12] is composed of $N=6$ Encoder
layers and $N=6$ Decoder layers, while many different architectures have been
proposed. The work of [15] proposed a tunable Transformer model where the same
Encoder/Decoder layer is used until the computation reaches the desired
results. This design led to having only one instance of these layers and the
same parameter set for each iteration, which either can be fixed a priori or
adapted dynamically during inference. On the other hand, the BERT-like
architectures [16] rely only on the Encoder part of the network, achieving
state-of-the-art performances in several tasks.

The Attention is the key operation of a Transformer, unleashing overall better
performance than other ML architectures. However, due to the necessary non-
linearities, its compute-intensive operations challenge typical neural
hardware accelerators. Moreover, Transformers’ massive sizes, like the OpenAI
GPT-3 [17] with $96$ hidden layers and $175\ B$ parameters, lead to large
memory footprint, latency, and power consumption, making their hardware
execution extremely resource-demanding.

Some works [18][19] proposed mathematical manipulations for reducing the
complexity of the needed operators in functions like Softmax and Layer
Normalization. Other works inspired by compression techniques like pruning or
knowledge distillation reduce the model’s operations and parameters [20]. Note
that these approaches are orthogonal to our work.

A potential technique that has the higher impact on implementing Transformer
networks onto hardware devices is quantization, which transforms the floating-
point values, universally used in the Transformer models, into integer values
while trying to minimize the consequent precision loss. For instance, the
I-BERT [11] implements the entire BERT network [16] with integer operations.
This process consists of an efficient way to have simpler operators and
lighter number representation, helping both the resources and memory
constraints for an efficient accelerator design. Similarly, the I-ViT [21]
quantizes the Vision Transformer [22] for image classifiation. However, these
works are deployed on GPUs, while our focus is on specialized hardware
accelerators.

###  II-B AI Hardware Accelerators Executing Transformers’ Operations

Generic AI hardware accelerators are based on arrays of Processing Elements
(PEs) [23]. Adapting an existing accelerator to compute the dot product
between the query, key, and value matrices in the MHSA mechanism would require
several modifications in its architecture and would not be straightforward.
Therefore, designing specialized hardware architectures for Transformers is
highly desirable.

Recent works proposed hardware designs for executing some of the Transformer’s
operations efficiently. However, they primarily use floating-point
representation, which is complex and expensive in terms of hardware resources.
To the best of our knowledge, there are no accelerators in prior works
implementing the entire Transformer network using only integer computations
and simple approximations for nonlinear operations. Still, the work in [24]
presents an interesting hardware architecture for MHSA and FFN. Besides
several novel additions, our work also takes some inspiration from these works
regarding the column-oriented computations. In fact, our data flow is designed
to use one column at a time of the matrices under processing. Since the matrix
multiplication, which is the most used operation in Transformers, requires the
input to be read column-by-column according to its algorithm, this data flow
structure enables a simple interface between blocks. The architecture proposed
in [6] proposes an optimization for the hardware matrix multiplication in the
Transformer. The work in [25] analyzes the network design considering the
hardware latency in the process. The work in [26] evaluates the performance of
a floating-point implementation on CPUs and GPUs.

The main drawbacks of these related works from the hardware execution
perspective are the following:

  1. 1.

The Floating Point representation is employed in some computations
[8][27][28]. This is a simulated quantization process where almost every
variable is quantized. However, the GELU, Softmax, and LayerNorm are computed
with floating-point operations. It not only increases the hardware complexity,
but it also requires both quantization and dequantization layers to convert
data between blocks.

  2. 2.

Complex operators like exponential and square root are implemented with
expensive LUTs [29][30], or using different approaches that involve FFTs [31].

To overcome these issues, our proposed SwiftTron architecture executes all the
layers and functions in transformers using only integer computations.

##  III SwiftTron: Hardware Architecture Design

This section describes each hardware block designed for executing the
Transformers’ layers (as discussed in Section II-A). The layers are mainly
composed of linear transformations and nonlinear functions. Since the linear
layers are based on matrix multiplications, a MatMul block is designed. On the
other hand, to execute nonlinear functions, namely the Softmax, GELU, and
Square Root for the Normalization, our work deploys second-order polynomial
approximations and recursive implementation, which are based on the concepts
from [11]. A top-level view of our proposed SwiftTron architecture is shown in
Figure 5.

Figure 5: Top-level overview of our SwiftTron architecture.

The Transformer’s main parameters are the model dimension $d$, the number of
heads $k$, the sentence length $m$, and the feed-forward dimension $d_{ff}$
(usually two or four times the value of $d$).

###  III-A Quantization and Scaling Factor Design

Before going into detail of each component, it is important to highlight that
quantized values have their corresponding scaling factors derived during the
quantization process.

Formally, given $a$ a floating-point value and $q_{a}$ its quantized value,
the scaling factor $S_{a}$ is defined such that $a=Q_{a}S_{a}$. Scaling
factors allow the transformations from floating point to integer and vice
versa, and determine the correct operation between two integers. For example,
two numbers with different scaling factors cannot be directly summed together,
but an extra component is needed for the Residual Connection. Furthermore, the
scaling factors are fundamental for computing the algorithm’s coefficients,
especially in the nonlinear functions. Some constraints on the scaling factor
values are applied to have representable coefficients and to limit the risk of
overflow. A scaling factor is a floating-point number that is not directly
included in the architecture to avoid FP operators, but its value is fixed for
each layer at design time. Another critical aspect to consider is the data
representation in the architecture. The matrix multiplications are conducted
with INT8 inputs and INT32 accumulators. The nonlinear functions operate on
INT32 to avoid excessive accuracy loss. Therefore, a Requantization block is
needed to bring the INT32 values back to INT8 as input to the subsequent
MatMul operations.

Dealing with the scaling factor in linear operations is straightforward. For
instance, the multiplication between to numbers ($a$ and $b$) with different
scaling factors is defined as $a\cdot b=q_{a}S_{a}\cdot q_{b}S_{b}=(q_{a}\cdot
q_{b})(S_{a}\cdot S_{b})$. This property holds for matrix multiplications
(MatMul), since its resulting expression is $MatMul(Sq)=S\cdot MatMul(q)$.

However, transformers contain several nonlinear operations, such as GELU,
Softmax, and Layer Normalization, for which this property does not hold. For
this reason, these operations can be either approximated using a second-order
polynomial, as in the case of GELU and Softmax, or computed iteratively, like
for the square root of the Layer Normalization.

###  III-B Hardware Architecture of the MatMul Block

The MatMul block is extremely important since it is used extensively in the
following designs. It is formed by $\\#rows$ $\times$ $\\#columns$ Multiply-
and-Accumulate (MAC) elements that, at every iteration, receive their
corresponding row and column input and update the accumulator. Figure 6
reports an example with a $4\times 4$ MAC array. After all the inputs are
scanned, they store the final output that can be read column-by-column by
tuning the selector of the output multiplexer. In the following layers,
several MatMul operations are required with different dimensions, but
depending on what type of Transformer workload to execute, these components
can be shared and/or reused.

Figure 6: Architectural view of a $4\times 4$ MatMul block with bias.

The bias addition for the linear transformations can be incorporated into the
component, like in this example. The bias is added when reading the output
matrix. Hence, a different value is added to each column. For multiplications
that do not require bias, this process can be ignored.

###  III-C Hardware Architecture of the Requantization Unit

A scaling factor change is necessary to reduce the precision from $32$ bits to
$8$ bits. To perform this transformation, the Dyadic Numbers concept [32] is
involved. Starting from the $32$-bit representation of a value $a$, denoted
with its quantized value $q_{a}$ and its quantization scale $S_{a}$ such that
$a=q_{a}S_{a}$, the final representation should be $o=q_{o}S_{o}$ with $q_{o}$
on $8$ bits. Hence, equalling $a$ and $o$ since the real value must remain
unchanged, the formula is derived in Equation 1.

$q_{a}S_{a}=q_{o}S_{o}\longrightarrow q_{o}=q_{a}\frac{S_{a}}{S_{o}}$ (1)

Remembering that the scaling factors are not strictly integers but can assume
any real value, this expression cannot be implemented directly on integer-only
resources. Hence, the scaling factor ratio is represented with a dyadic
number, a rational number in the format of $b/2^{c}$, where $b$ and $c$ are
two integer numbers. The final expression is shown in Equation 2.

$q_{o}=q_{a}\frac{S_{a}}{S_{o}}=q_{a}DN(\frac{S_{a}}{S_{o}})=q_{a}*\frac{b}{2^{c}}$
(2)

This convention also avoids the need to use dividers, as the required
resources are only an INT32 multiplication and a one-bit shifting, as shown in
Figure 7.

Figure 7: Architecture details of the Requantization (Req) unit.

###  III-D Hardware Architecture of the MHSA Unit

The MHSA block is responsible for computing the correspondent operation in a
Transformer. A single head, whose architecture is shown in Figure 8, is
composed of three MatMul blocks with inputs connected to Query, Keys, and
Values, and an Attention operator. Figure 9 shows an example of the complete
MHSA architecture composed of $4$ heads and another MatMul block that
generates the outputs.

Figure 8: Architecture details of a single head in the MHSA, composed of three
MatMul blocks, performing computations on Query ($Q$), Key ($K$), and Value
($V$) matrices, and one Attention block. Figure 9: Architecture details of the
complete MHSA unit with multiple heads. The example in the figure contains $4$
heads and another MatMul block that generates the outputs.

The choice of the number of heads to be computed in parallel depends on the
available hardware resources. Different architectural configurations can be
designed, from processing one head at a time to computing all heads
concurrently. Therefore, data can be either processed concurrently or
sequentially by reusing the head blocks. Consequently, the computations of the
final MatMul can be conducted with multiple batches of data. Whenever it
receives as input one head output (coming in order), it updates its
accumulators.

###  III-E Hardware Architecture of the Attention Unit

The Attention architecture is shown in Figure 10. It is composed of two plain
MatMuls with a Softmax in between. The Scale simply consists of a division by
the model dimension $d$. If the value of $d$ is a multiple of $2$, it becomes
a simple shift operation. The Requantization is required to keep the input of
the second MatMul to INT8.

Figure 10: Attention architecture, composed of MatMul, Scale, Softmax, and
Requantization operators.

###  III-F Hardware Architecture of the Softmax Unit

Since the Softmax is performed along the rows of the $Q\times K^{T}$ matrix,
$m$ Softmax components are instantiated and work concurrently. A single
Softmax operator is shown in Figure 11. Its computation requires three phases,
namely maximum search, exponential computation, and output generation.

Figure 11: Architecture of the Softmax operator. According to [11],
$q_{1}=\lfloor b/S_{pe}\rfloor$, $q_{2}=\lfloor c/aS_{pe}^{2}\rfloor$,
$q_{3}=\lfloor ln2/S_{e}\rfloor$, and $q_{4}=\lfloor-1/q_{3}\rfloor$, where
$S_{e}$ is the scaling factor of the exponential computation input (relative
to $q_{e}$), $S_{pe}$ is the scaling factor of the polynomial approximation
input (relative to $q_{pe}$) and $a,b,c$ are the coefficient of the second-
order polynomial that approximates the function, i.e., $a(x+b)^{2}+c$.
Therefore, $q_{1,2,3,4}$ can be computed at design time and provided as
constant values to the SwiftTron architecture.

The implementation approach of the Softmax unit (see Figure 12) aims at
restricting the range of values in which the exponential function needs to be
computed. Following the property described in Equation 3, as in the Softmax
inputs are limited by their maximum value, subtracting the maximum value leads
to dealing with non-positive real numbers, which can be decomposed [11].
Consequently, the exponential function must be computed only for the
restricted range of $[-ln2,\ 0]$ and can be approximated with a second-order
polynomial.

$Softmax(\textbf{x}_{i})=\frac{exp(x_{i})}{\sum_{j=1}^{k}exp(x_{j})}=\frac{exp(x_{i}-x_{max})}{\sum_{j=1}^{k}exp(x_{j}-x_{max})}$

(3)

Figure 12: Approach used for manipulating the Softmax operator to allow a
second-order polynomial approximation of the exponential function in a
restricted range of values.

It is evident that with this approximation, only simple operators are
involved, like a comparator for the maximum, adders, and multipliers. The most
complex operator is the divider, whose implementation consumes relatively more
resources.

###  III-G Hardware Architecture of the Feed-Forward Network Unit

This sub-layer has two linear transformations separated by an activation
function. The transformations, implemented with MatMul blocks, are the biggest
of the Transformer architecture, as $d_{ff}$ is usually $4\times$ the
dimension $d$. The FFN architecture is depicted in Figure 13. The activation
function used in our architecture is the GELU [14], which despite being more
complex, has better performances than the ReLU.

Figure 13: Architecture of the Feed-Forward Network unit, composed of MatMul,
GELU, and Requantization operators.

###  III-H Hardware Architecture of the GELU Unit

The GELU operator, shown in Figure 14, contains the computation of the error
function (erf). This nonlinear function is linearized through another second-
order polynomial with limited input. With this approximation, the resulting
operators are only adders and multipliers, with some sign-handling operations
that complete the execution.

Figure 14: Architecture of the GELU operator. According to [11], $q_{5}=-b/S$,
$q_{6}=\lfloor b/S\rfloor$ $q_{7}=\lfloor c/aS^{2}\rfloor$, and $q_{8}=\lfloor
1/S_{erf}\rfloor$, where $S$ is the scaling factor of the GELU input (relative
to $q$), $S_{erf}$ is the scaling factor of the error function output
(relative to $q_{erf}$) and $a,b,c$ are the coefficient of the second-order
polynomial that approximates the function, i.e., $a(x+b)^{2}+c$. Therefore,
$q_{5,6,7,8}$ can be computed at design time and provided as constant values
to the SwiftTron architecture.

###  III-I Hardware Architecture of the Residual Connection and Layer
Normalization Units

Since the MHSA and FFN are residual blocks, their outputs are added to the
original inputs. As we are dealing with quantized values, the two addends need
to have the same scaling factor before being added together. This
transformation is achieved using a Dyadic unit, already discussed in the
Requantization unit (recall Equation 2), implemented with a multiplication by
a coefficient and a right-shifting. It is a combinatorial block that is
replicated by the number of rows, as it receives one column at a time coming
from the previous sub-blocks.

After the residual connection, the Layer Normalization (LayerNorm) is
required. Its architecture is depicted in Figure 15. Similarly to the Softmax
component, since the LayerNorm operation works on the row elements, $d$
instantiations are needed. Moreover, it is composed of three phases, the mean
value calculation, the standard deviation calculation, and the output
generation.

Figure 15: Layer Normalization architecture, where the Square Root unit
implements a recursive algorithm. The $Valid$ and $z$ signals are flags for
assisting the control unit with the correct timing.

The only nonlinear operation is the square root, which is implemented using an
iterative algorithm as proposed in [33], also adopted in [11]. It is a
recursive algorithm that needs multiple cycles to compute the output. It
includes combinatorial operators and registers to store intermediate values
and break the loop.

It has a constant initial value, defined as $x_{0}$. At every iteration, the
partial result $x_{i}$ is compared to the partial result of the next iteration
$x_{i+1}$ that is equal to $(x_{i}+x_{i}/n)/2$. Note that the division by $2$
is implemented through a simple one-position right shift. The algorithm
iterates until $x_{i+1}$ is larger than or equal to $x_{i}$. When this
happens, the final result is saved into the dedicated register. Since the
number of cycles needed is unknown a priori, the $Valid$ and $z$ signals are
flags for assisting the control unit and generate the correct timing. In the
special case when the square root input is zero, the output goes directly to
zero, and no iterations are needed.

###  III-J Hardware Architecture of the Control Unit

At each stage of the Transformers’ process, the control unit generates
different control signals for all the components of the SwiftTron
architecture, according to the operations needed. Its functionality is
depicted in Figure 16. For the three major operations, which are MHSA,
LayerNorm, and FFN, dedicated Finite State Machines (FSMs) generate the
respective control signals. A set of handshake signals (e.g., $Start$, $Done$,
$Valid$) is devised to interact between different FSMs and guarantee the
correct timing of the operations in all stages of the Transformers’ inference.

Figure 16: Functionality of the control unit in our SwiftTron architecture,
composed of dedicated Finite State Machines for MHSA, LayerNorm, and FFN
operations.

##  IV Evaluation of our SwiftTron Architecture

###  IV-A Experimental Setup

We implement the complete design of our SwiftTron architecture in RTL (VHDL)
and evaluate it for the RoBERTa architecture [34] on the GLUE benchmark [35]
and for the DeiT [36] on the ImageNet dataset [37]. We synthesize the
SwiftTron architecture in a $65\ nm$ CMOS technology node using the ASIC
design flow with the Synopsys Design Compiler. We conduct functional and
timing validation through gate-level simulations using Mentor Graphics
QuestaSim. With the synthesized netlist, we obtain area, power, and
performance of our design. We also run the inference on an Nvidia GeForce RTX
2080 Ti GPU for latency comparison.

For validation, we use the pre-trained Transformer models from the HuggingFace
library [38], and implement them on the PyTorch framework using the
quantization algorithms of I-BERT [11]. The complete flow is shown in Figure
17, where the grey boxes are the inputs, the orange boxes are the outputs, and
the green boxes represent the main tools used. Note that this hardware design
and validation flow are well-adopted by the hardware design community.

Figure 17: Experimental setup and tool flow for conducting the experiments.

###  IV-B SwiftTron Synthesis Results

For evaluating the complete SwiftTron design, we evluate the architecture with
$d=768$, $k=12$, $m=256$, and $d_{ff}=3072$, to efficiently execute the
RoBERTa-base architecture [34]. Note that these values are arbitrary
parameters in our design and can be tuned during design time to support the
execution of different Transformer architectures. The clock period has been
set to $7\ ns$, which corresponds to a clock frequency of $\approx 143\ MHz$.
To comply with the timing requirements, the computations of the Softmax and
LayerNorm units have been partitioned into three pipeline stages. Each
component of the SwiftTron architecture has been tested separately and
simulated to validate its outputs compared to the software-level
implementation of [11]. Area and power consumption have been evaluated based
on the reports obtained by the Synopsys Design Compiler tool, while the
latency has been measured with a cycle-accurate simulator333The simulator
considers the worst-case scenario in terms of clock cycles computed by the
square root operator of the LayerNorm unit.. Table I summarizes the key
results obtained from the synthesis. Table II reports the accuracy and latency
of different Transformer models executed on our SwiftTron architecture and the
speedup of our accelerator compared to an Nvidia GeForce RTX 2080 Ti GPU with
CUDA $10$. Our experiments run more than $3\times$ faster than the GPU
implementations.

TABLE I: Summary of synthesis results of our proposed SwiftTron architecture.

Clock Frequency | $143\ MHz$ |  | Technology Node | $65\ nm$
---|---|---|---|---
Power Consumption | $33.64\ W$ |  | Area | $273.0\ mm^{2}$

TABLE II: Accuracy and inference latency for RoBERTa-base and RoBERTa-large
models on the STT-2 task of the GLUE benchmark with sequence length $m=256$
and for DeiT-S on the ImageNet dataset with resolution $224\times 224$
executed on our SwiftTron architecture. The last column reports the speedup
w.r.t. their execution on the RTX 2080 Ti GPU.

Model | Accuracy | Latency | Speedup w.r.t. GPU
---|---|---|---
RoBERTa-base on STT-2 | $95.2\%$ | $1.83\ ms$ | $3.81\times$
RoBERTa-large on STT-2 | $96.4\%$ | $45.70\ ms$ | $3.90\times$
DeiT-S on ImageNet | $79.11\%$ | $1.13\ ms$ | $3.58\times$

TABLE III: Summary of comparisons between the related works and our proposed SwiftTron architecture. Work | HW Implementation | Bit-width | Complete Architecture | Nonlinear Function Computation
---|---|---|---|---
OPTIMUS [6] |  ✓ ASIC 28 nm | INT16 |  ✗ No |  ✗ N/A
A3 [7] |  ✓ ASIC 40 nm |  ✓ INT8 |  ✗ No |  ✓ Integers (approximated)
FTRANS [31] |  ✓ Xilinx FPGA | INT16 |  ✓ Yes |  ✗ Integers using FFT
Lu et al. [24] |  ✓ Xilinx FPGA |  ✓ INT8 |  ✗ No |  ✓ Integers (approximated)
EFA-Trans [29] |  ✓ Xilinx FPGA |  ✓ INT8 |  ✓ Yes |  ✗ LUT
FQ-BERT [30] |  ✓ Xilinx FPGA |  ✓ INT8 |  ✓ Yes |  ✗ LUT
Lin et al. [8] |  ✗ TITAN V GPU |  ✓ INT8 |  ✓ Yes |  ✗ FP32
I-BERT [11] |  ✗ Tesla T4 GPU |  ✓ INT8 |  ✓ Yes |  ✓ Integers (approximated)
I-ViT [21] |  ✗ RTX 2080 Ti GPU |  ✓ INT8 |  ✓ Yes |  ✓ Integers (approximated)
Transformer Engine [9] |  ✓ ASIC 4 nm (inside H100 GPU) |  ✗ FP8 |  ✓ Yes |  ✗ FP16 / FP32
SwiftTron (ours) |  ✓ ASIC 65 nm |  ✓ INT8 |  ✓ Yes |  ✓ Integers (approximated)

###  IV-C Area and Power Breakdown

The total values of the area and power consumption of the complete SwiftTron
architecture are reported in the respective lines of Table I, while Figure 18
analyzes in detail the breakdown for each component. It is evident that the
MatMul block is responsible for the majority ($55\%$) of the area of the
entire architecture. The difference between MatMul and other components
becomes even larger for the power consumption. While the Softmax unit occupies
$17\%$ of the total area, its contribution to the total power is only $14\%$.
An even more significant difference is noted for the LayerNorm unit, which
occupies $25\%$ area, but its power consumption is $6\%$. As expected, the
GELU unit is a small component with only $3\%$ area and $1\%$ power
consumption.

Figure 18: (a) Area and (b) power breakdown of our SwiftTron architecture.

###  IV-D Comparison with Related Works

While it is known that a specialized accelerator brings immense advantages
compared to executing the same task on a general-purpose hardware device,
e.g., CPU or GPU, it is difficult to compare metrics like power and
performance across completely different hardware platforms of the related
works, which include GPUs, FPGAs, and other ASIC architectures synthesized for
different technology nodes. Hence, we identify key features of a hardware
architecture for Transformers that push its efficiency to the upper boundary.
Not only the hardware device on which it is implemented is important, but also
the bit-width plays a key role in determining its energy efficiency. This is
because even simple operators like adders and multipliers are relatively
lightweight when implemented using integer arithmetic and low bit-width, like
for INT8. On the other hand, their floating-point implementation incurs a
significant complexity overhead.

Table III summarizes the comparison between our SwiftTron architecture and the
related works, considering these important features for a hardware
architecture for Transformers. From the table, it is clear that our work
complies with all the requirements, while all the related works have at least
one missing feature. Some works [8][11][21] implement their design on GPUs,
other works [6][7][24] accelerate only part of a complete Transformer, and
other works do not use efficient computations for their nonlinear functions.
The design proposed in [31] uses integer computations, but its complexity is
high due to the presence of FFT transforms. The architectures presented in
[29][30] use LUTs for computing some nonlinear operations, while the works in
[8][9] performs the nonlinear computations using FP16 or FP32 arithmetics.

##  V Conclusion

In this paper, we present SwiftTron, a specialized accelerator for
Transformers that executes all the operations, including the nonlinear
operations, in integer arithmetic. The correct computations between integers
are achieved through a specialized quantization scheme that accounts for
diverse scaling factors. Dedicated designs implement approximated versions of
the nonlinear units, like Softmax, GELU, and Layer Normalization. The
SwiftTron architecture synthesized using the ASIC design flow shows efficient
area, power, and performance while complying with all the desired features for
a Transformer accelerator. Our design and thorough analyses pave the way for
future developments of efficient Transformer architectures.

## Acknowledgment

This work has been supported in part by the Doctoral College Resilient
Embedded Systems, which is run jointly by the TU Wien’s Faculty of Informatics
and the UAS Technikum Wien. This work was also supported in parts by the
NYUAD’s Research Enhancement Fund (REF) Award on “eDLAuto: An Automated
Framework for Energy-Efficient Embedded Deep Learning in Autonomous Systems”,
and by the NYUAD Center for Artificial Intelligence and Robotics (CAIR),
funded by Tamkeen under the NYUAD Research Institute Award CG010.

## References

  * [1] M. Shafique, T. Theocharides, C. Bouganis, M. A. Hanif, F. Khalid, R. Hafiz, and S. Rehman, “An overview of next-generation architectures for machine learning: Roadmap, opportunities and challenges in the iot era,” in 2018 Design, Automation & Test in Europe Conference & Exhibition, (DATE), pp. 827–832, IEEE, 2018.
  * [2] A. Marchisio, M. A. Hanif, F. Khalid, G. Plastiras, C. Kyrkou, T. Theocharides, and M. Shafique, “Deep learning for edge computing: Current trends, cross-layer optimizations, and open research challenges,” in 2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI), pp. 553–559, IEEE, 2019.
  * [3] M. Capra, B. Bussolino, A. Marchisio, M. Shafique, G. Masera, and M. Martina, “An updated survey of efficient hardware architectures for accelerating deep convolutional neural networks,” Future Internet, vol. 12, no. 7, p. 113, 2020.
  * [4] M. Shafique, A. Marchisio, R. V. W. Putra, and M. A. Hanif, “Towards energy-efficient and secure edge AI: A cross-layer framework ICCAD special session paper,” in IEEE/ACM International Conference On Computer Aided Design, ICCAD 2021, Munich, Germany, November 1-4, 2021, pp. 1–9, IEEE, 2021.
  * [5] S. Dave, A. Marchisio, M. A. Hanif, A. Guesmi, A. Shrivastava, I. Alouani, and M. Shafique, “Special session: Towards an agile design methodology for efficient, reliable, and secure ML systems,” in 40th IEEE VLSI Test Symposium, VTS 2022, San Diego, CA, USA, April 25-27, 2022, pp. 1–14, IEEE, 2022.
  * [6] J. Park, H. Yoon, D. Ahn, J. Choi, and J. Kim, “OPTIMUS: optimized matrix multiplication structure for transformer neural network accelerator,” in Proceedings of Machine Learning and Systems 2020 (MLSys), 2020.
  * [7] T. J. Ham, S. Jung, S. Kim, Y. H. Oh, Y. Park, Y. Song, J. Park, S. Lee, K. Park, J. W. Lee, and D. Jeong, “A${}^{\mbox{3}}$: Accelerating attention mechanisms in neural networks with approximation,” in IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 328–341, 2020.
  * [8] Y. Lin, Y. Li, T. Liu, T. Xiao, T. Liu, and J. Zhu, “Towards fully 8-bit integer inference for the transformer model,” in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI), pp. 3759–3765, 2020.
  * [9] “Nvidia h100 tensor core gpu architecture whitepaper,” https://resources.nvidia.com/en-us-tensor-core/gtc22-whitepaper-hopper.
  * [10] G. Marcus, P. Hinojosa, A. Avila, and J. Nolazco-FIores, “A fully synthesizable single-precision, floating-point adder/substractor and multiplier in vhdl for general and educational use,” in Proceedings of the Fifth IEEE International Caracas Conference on Devices, Circuits and Systems, 2004., vol. 1, pp. 319–323, 2004.
  * [11] S. Kim, A. Gholami, Z. Yao, M. W. Mahoney, and K. Keutzer, “I-BERT: integer-only BERT quantization,” in Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 5506–5518, 2021.
  * [12] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30 (NeurIPS), pp. 5998–6008, 2017.
  * [13] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th International Conference on Machine Learning (ICML), pp. 807–814, 2010.
  * [14] D. Hendrycks and K. Gimpel, “Bridging nonlinearities and stochastic regularizers with gaussian error linear units,” CoRR, vol. abs/1606.08415, 2016.
  * [15] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser, “Universal transformers,” in 7th International Conference on Learning Representations (ICLR), 2019.
  * [16] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pp. 4171–4186, 2019.
  * [17] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in Advances in Neural Information Processing Systems 33 (NeurIPS), 2020.
  * [18] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” CoRR, vol. abs/2006.04768, 2020.
  * [19] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992–10002, IEEE, 2021.
  * [20] W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity,” CoRR, vol. abs/2101.03961, 2021.
  * [21] Z. Li and Q. Gu, “I-vit: Integer-only quantization for efficient vision transformer inference,” CoRR, vol. abs/2207.01405, 2022.
  * [22] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in 9th International Conference on Learning Representations (ICLR), 2021.
  * [23] M. Capra, B. Bussolino, A. Marchisio, G. Masera, M. Martina, and M. Shafique, “Hardware and software optimizations for accelerating deep neural networks: Survey of current trends, challenges, and the road ahead,” IEEE Access, vol. 8, pp. 225134–225180, 2020.
  * [24] S. Lu, M. Wang, S. Liang, J. Lin, and Z. Wang, “Hardware accelerator for multi-head attention and position-wise feed-forward in the transformer,” in 33rd IEEE International System-on-Chip Conference (SoCC), pp. 84–89, IEEE, 2020.
  * [25] H. Wang, Z. Wu, Z. Liu, H. Cai, L. Zhu, C. Gan, and S. Han, “HAT: hardware-aware transformers for efficient natural language processing,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7675–7688, 2020.
  * [26] H. Wang et al., Efficient algorithms and hardware for natural language processing.  PhD thesis, Massachusetts Institute of Technology, 2020.
  * [27] A. Bhandare, V. Sripathi, D. Karkada, V. Menon, S. Choi, K. Datta, and V. Saletore, “Efficient 8-bit quantization of transformer neural machine language translation model,” CoRR, vol. abs/1906.00532, 2019.
  * [28] G. Prato, E. Charlaix, and M. Rezagholizadeh, “Fully quantized transformer for machine translation,” in Findings of the Association for Computational Linguistics (EMNLP), pp. 1–14, 2020.
  * [29] X. Yang and T. Su, “Efa-trans: An efficient and flexible acceleration architecture for transformers,” Electronics, vol. 11, no. 21, 2022.
  * [30] Z. Liu, G. Li, and J. Cheng, “Hardware acceleration of fully quantized BERT for efficient natural language processing,” in Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 513–516, IEEE, 2021\.
  * [31] B. Li, S. Pandey, H. Fang, Y. Lyv, J. Li, J. Chen, M. Xie, L. Wan, H. Liu, and C. Ding, “FTRANS: energy-efficient acceleration of transformers using FPGA,” in ACM/IEEE International Symposium on Low Power Electronics and Design (ISLPED), pp. 175–180, ACM, 2020.
  * [32] Z. Yao, Z. Dong, Z. Zheng, A. Gholami, J. Yu, E. Tan, L. Wang, Q. Huang, Y. Wang, M. W. Mahoney, and K. Keutzer, “HAWQ-V3: dyadic neural network quantization,” in Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 11875–11886, 2021.
  * [33] C. P. Richard Crandall, Prime Numbers: A Computational Perspective, vol. 182.  Springer New York, NY, 2006.
  * [34] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” CoRR, vol. abs/1907.11692, 2019.
  * [35] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in 7th International Conference on Learning Representations (ICLR), 2019.
  * [36] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 10347–10357, 2021.
  * [37] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems 25 (NeurIPS), pp. 1106–1114, 2012.
  * [38] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, 2020.


# Violation of Diagonal Non-Invasiveness: A Hallmark of Quantum Memory Effects

Adrián A. Budini  Consejo Nacional de Investigaciones Científicas y Técnicas
(CONICET), Centro Atómico Bariloche, Avenida E. Bustillo Km 9.5, (8400)
Bariloche, Argentina, and Universidad Tecnológica Nacional (UTN-FRBA), Fanny
Newbery 111, (8400) Bariloche, Argentina

###### Abstract

An operational (measurement based) scheme that connects in a univocal way
measurement invasivity and the presence of memory effects is defined. Its
underlying theoretical basis relies on a non-invasive measurability of
(memoryless) Markovian dynamics when the corresponding observable is diagonal
in the same basis as the system density matrix. In contrast, (operational
defined) quantum memory effects always lead to violation of diagonal non-
invasiveness. Related conditions for violation of Leggett-Garg inequality due
to non-Markovian memory effects are also established.

Several well understood physical principles allow distinguishing classical
from quantum realms. For example, the principle of locality, valid in a
classical regime, is violated in presence of quantum entanglement. This
property is captured by Bell inequality (BI), which introduces a severe
constraint on the (measurement) correlations that can be established between
two spatially-separated systems bell ; nobel . Non-invasive measurability is
another principle that distinguish classical and quantum systems. In fact, a
quantum measurement process leads in general to an unavoidable modification of
the system state. Added to macroscopic realism, this feature is the basis of
Leggett-Garg inequality (LGI) leggett ; noriRep . It defines a constraint on
the correlations that can be established between measurements performed over a
single system at two different times. Similarly to BI, a broad class of
theoretical an experimental related results were analyzed and proposed in the
literature mahler ; huelga ; korotkov ; buttiker ; caslav ; jordan ; chen ;
dressel ; FHuelga ; RMN ; nori ; brukner ; groot ; budro ; emary ; alt ; guo ;
quiroga ; haliwell ; cunha , being from different perspectives a topic of
current interest pagliara ; laflame ; vitale ; reid ; arani ; yama .

Both BI and LGI assume a similar structure after mapping the distance between
the measured systems in the former case with the time-interval between
measurements in the last case. For this reason LGIs are also termed as
“temporal Bell Inequalities” noriRep ; mahler . Nevertheless, there is an
intrinsic unavoidable difference. Temporal correlations can only be studied
after introducing a non-trivial (non-vanishing) system dynamics. In fact,
discussion about the influence of the (open) system time-evolution can be
found in the original LGI presentation leggett .

From a simplified point of view, one can affirm that both closed and open
quantum systems are always altered by a measurement process, leading to LGI
violation. This kind of simplification contrast with the strong advancements
achieved in the last years in the classification and understanding of open
quantum dynamics alicki ; breuerbook ; vega . In particular, the usual
association between memory effects and time-convoluted contributions in the
density matrix evolution was abandoned. Instead, memory effects are defined
from two complementary perspectives. In non-operational approaches
BreuerReview ; plenioReview ; BreuerFirst ; cirac ; rivas ; DarioSabrina ;
fisher ; geometrical ; canonicalCresser , memory effects are related to
departures of the system time-evolution from a (memoryless) Markovian Lindblad
master equation breuerbook ; alicki . Alternatively, in operational approaches
the system is subjected to a set of measurement processes modi ; budiniCPF ;
pollock ; pollockInfluence ; bonifacio ; han ; ban ; rio ; hefei ; goan ; BIF
. Thus, non-Markovianity is defined in a standard probabilistic way vanKampen
from the corresponding outcome statistics.

Given the previous advancements in the characterization of open system
dynamics, it is compelling to find out if there exist any general relation
between measurement invasivity and the presence of memory effects. Studies
along these lines were performed previously diogo ; ali ; shao ; plenio .
Nevertheless, not any clear boundary in the properties of measurement
invasivity seems to be defined by the presence or absence of memory effects in
the system dynamics, independently of which definition is taken. In fact, the
possible relations turn up to be strongly model-dependent. Hence, a criteria
that allow relating measurement invasivity with the properties of the system
dynamics, Markovian or non-Markovian, is still lacking.

The aim of this work is to establishing a clear and rigorous general relation
between measurement invasivity and the presence of memory effects. It provides
a fundamental step forward in the understanding and characterization of
measurement processes in open quantum system dynamics. The main theoretical
ingredient relies on a diagonal non-invasiveness (DNI) of Markovian dynamics,
which applies when the measurement observable commutates with the (pre-
measurement) system density matrix. Hence, the observable and system state are
diagonal in the same basis of states. In contrast, memory effects, as defined
in operational approaches modi ; budiniCPF , break the previous condition. An
operational scheme, based on performing three consecutive system measurement
processes, allow to witnessing these properties. In addition, these results
enable us to establish under which conditions violations of LGI can be related
univocally to the presence of memory effects.

Measurement Invasivity.—In operational approaches, memory effects can be
witnessed with (a minimum of) three consecutive system measurement processes.
Consequently, measurement invasivity is defined from the same (operational)
basis. The measurements are performed at times $0,$ $t,$ and $t+\tau,$
delivering correspondingly the outcomes $\\{x\\},$ $\\{y\\},$ and $\\{z\\}.$
Their joint probability is denoted as $P_{3}(z,y,x),$ where the subindex
indicates the number of performed measurements. Margination over the
intermediate measurement outcomes lead to

$P_{3}(z,x)\equiv\sum\nolimits_{y}P_{3}(z,y,x).$ (1)

Alternatively, one can perform only two measurements at times $0$ and
$t+\tau,$ which defines the joint probability $P_{2}(z,x).$ For quantum
systems, measurement invasivity implies that $P_{2}(z,x)\neq P_{3}(z,x).$ In
order to quantify this disagreement, we use a Kolmogorov (trace) distance
nielsen

$I\equiv\sum\nolimits_{zx}|P_{3}(z,x)-P_{2}(z,x)|.$ (2)

For classical systems $I=0,$ while $I>0$ is a direct witness of measurement
invasiveness. This property is valid independently of the system dynamics,
closed or open, Markovian or non-Markovian. In the following analysis, the
quantifier $I$ is studied by assuming different underlying system-environment
($s$-$e$) dynamics.

Markovian dynamics.—Similarly to closed systems, a Markovian dynamics is
defined by a density matrix propagator $\Lambda_{t,t^{\prime}}$ that is
completely independent of the system or environment initial states modi ;
budiniCPF . Hence, in the case of two measurement processes it is simple to
obtain

$\frac{P_{2}(z,x)}{P_{1}(x)}=\mathrm{Tr}_{s}(E_{z}\Lambda_{t+\tau,0}[\rho_{x}]).$
(3)

Similarly, when performing three measurements it follows

$\frac{P_{3}(z,y,x)}{P_{1}(x)}=\mathrm{Tr}_{s}(E_{z}\Lambda_{t+\tau,t}[\rho_{y}])\
\mathrm{Tr}_{s}(E_{y}\Lambda_{t,0}[\rho_{x}]).$ (4)

In both cases, $P_{1}(x)=\mathrm{Tr}_{s}(E_{x}\rho_{0}),$ where $\rho_{0}$ is
the initial system state. $\mathrm{Tr}(\bullet)$ is the trace operation.
$\\{E_{m}\\}$ and $\\{\rho_{m}\\}$ $(m=x,y,z)$ are the (positive) measurement
operators and system post-measurement states respectively breuerbook ; nielsen
. For Hermitian observables, both $\\{E_{m}\\}$ and $\\{\rho_{m}\\}$ are the
projectors associated to each observable spectral representation. Notice that
the Markov property $P_{3}(z,y,x)=P_{3}(z|y)P_{2}(y|x)P_{1}(x)$ is fulfilled.
$P(b|a)$ denotes the conditional probability of $b$ given $a.$

From Eq. (3) and (4) it follows that in general $I\neq 0.$ In fact,
measurement invasivity applies when the system dynamics is Markovian.
Nevertheless, given that the measurements are arbitrary ones, the intermediate
$y$-measurement can be chosen such that $[\rho_{t|x},E_{y}]=0,$ where
$\rho_{t|x}\equiv\Lambda_{t,0}[\rho_{x}].$ Thus,
$\\{\mathrm{Tr}_{s}(E_{y}\Lambda_{t,0}[\rho_{x}])\\}$ can be read as the
eigenvalues of $\rho_{t|x}$ implying
$\sum\nolimits_{y}\rho_{y}\mathrm{Tr}_{s}(E_{y}\Lambda_{t,0}[\rho_{x}])=\rho_{t|x}.$
Consequently,

$I_{D}\overset{M}{=}0.$ (5)

This property defines the DNI of Markovian dynamics: a measurement process at
a given time is non-invasive if the corresponding observable commutates with
the pre-measurement system density matrix. In the three measurement scheme DNI
at (any) time $t$ is valid for arbitrary $x$\- and $z$-measurements. These are
central results for the developed scheme. They are violated in presence of
memory effects.

Stochastic Hamiltonian models.—Here the open system is driven by random
fluctuations such that
$\Lambda_{t,t^{\prime}}=\overline{\Lambda_{t,t^{\prime}}^{st}},$ where
$\Lambda_{t,t^{\prime}}^{st}$ is the stochastic propagator for each noise
realization while the overline denotes the corresponding average. In
similarity with Eq. (3) it follows

$\frac{P_{2}(z,x)}{P_{1}(x)}=\mathrm{Tr}_{s}\overline{(E_{z}\Lambda_{t+\tau,0}^{st}[\rho_{x}])},$
(6)

while in similarity with Eq. (4),

$\frac{P_{3}(z,y,x)}{P_{1}(x)}=\mathrm{Tr}_{s}\overline{(E_{z}\Lambda_{t+\tau,t}^{st}[\rho_{y}])\
\mathrm{Tr}_{s}(E_{y}\Lambda_{t,0}^{st}[\rho_{x}])}.$ (7)

Hence, DNI is not valid in general. It is recovered when the average defining
$P_{3}(z,y,x)$ split in two terms as in Eq. (4). This property is only valid
for white noise fluctuations vanKampen , that is, when the dynamics is
Markovian budiniCPF .

Completely positive system-environment dynamics.—We consider an open dynamics
where the system $(s)$ and its environment $(e)$ obey a completely positive
dynamics. Taking separable initial conditions, given the (bipartite)
propagator $\mathcal{G}_{t,t^{\prime}},$ it follows

$\frac{P_{2}(z,x)}{P_{1}(x)}=\mathrm{Tr}_{se}(E_{z}\mathcal{G}_{t+\tau,0}[\rho_{x}\otimes\sigma_{0}]).$
(8)

where $\sigma_{0}$ is the environment initial state. Furthermore,

$\frac{P_{3}(z,y,x)}{P_{1}(x)}=\mathrm{Tr}_{se}(E_{z}\mathcal{G}_{t+\tau,t}[\rho_{y}\otimes\mathrm{Tr}_{s}(E_{y}\mathcal{G}_{t,0}[\rho_{x}\otimes\sigma_{0}])]).$
(9)

From these expressions it follows that DNI is hardly satisfied. Additionally,
$P_{3}(z,y,x)\neq P_{3}(z|y)P_{2}(y|x)P_{1}(x).$

For unitary $s$-$e$ interactions, when a Born-Markov approximation applies
breuerbook ,
$\mathcal{G}_{t,0}[\rho_{0}\otimes\sigma_{0}]\approx\rho_{t}\otimes\sigma_{0},$
DNI and Markovianity are simultaneously recovered. Complementarily, memory
effects lead to violation of DNI. Alternatively, one can also consider non-
unitary $s$-$e$ (dissipative) Lindblad dynamics alicki ; breuerbook . The same
relation between violation of DNI and non-Markovianity remains valid.

From Eqs. (8) and (9), it is simple to check that even in presence of memory
effects (for example for environments of finite dimension) DNI is accidentally
valid if the bipartite (pre-measurement) state
$\rho_{t|x}^{se}\equiv\mathcal{G}_{t,0}[\rho_{x}\otimes\sigma_{0}]$ has
($\forall t$) a null discord ZureckDiscord ; henderson ; modiCorrelator ,
$\rho_{t|x}^{se}=\sum_{c}|c_{t}\rangle\langle c_{t}|\otimes\sigma_{t}^{(c)},$
where $\\{|c_{t}\rangle\\}$ is a (in general time-dependent) complete
orthogonal basis of system states while $\\{\sigma_{t}^{(c)}\\}$ are bath
states. The intermediate $y$-measurement must be defined by the projectors
$\\{|c_{t}\rangle\langle c_{t}|\\}.$ This case is consistent with the results
of Ref. plenio , which relate classicality with a bipartite state with
vanishing discord. Nevertheless, only for very specific interactions and
particular $s$-$e$ initial conditions bipartite evolutions (unitary or non-
unitary) lead to a state with vanishing discord $\forall t$ almost ; poland ;
cero . Thus, for the proposed continuous-in-time open system dynamics, the
relation between violation of DNI and memory effects is always recovered by
considering the arbitrariness of the $x$\- and $z$-measurements. In fact, an
underlying local-in-time completely positive bipartite quantum dynamics cannot
lead to a vanishing discord state $\forall t$ when considering arbitrary
system initial conditions almost .

Leggett-Garg inequality—By denoting with $\\{O_{m}\\}$ the observable values
associated to each outcome, $m=x,y,z,$ LGI reads noriRep

$-3\leq\langle O_{y}O_{x}\rangle+\langle O_{z}O_{y}\rangle-\langle
O_{z}O_{x}\rangle\leq 1.$ (10)

This result is valid for dichotomic observables, $O_{m}=\pm 1.$ The
correlators, which are obtained by performing only two measurements, are
$\langle O_{j}O_{k}\rangle\equiv\sum_{jk}O_{j}O_{k}P_{2}(j,k).$ Eq. (10) can
be derived from $P_{2}(y,x)=\sum_{z}P_{3}(z,y,x),$ and by assuming that
$P_{2}(z,y)\overset{LGI}{=}\sum_{x}P_{3}(z,y,x),$ and
$P_{2}(z,x)\overset{LGI}{=}\sum_{y}P_{3}(z,y,x)$ noriRep . Due to measurement
invasiveness and independently of the system dynamics (Markovian or non-
Markovian), these equalities are not valid in general. Nevertheless, from the
present analysis [DNI, Eq. (5)] we conclude that Markovian dynamics always
obey LGI if at each pair of measurement processes the observables commutate
with the pre-measurement system density matrix. On the other hand, even under
this election of measurement processes, non-Markovian dynamics may or may not
obey LGI. In this sense, the distance $I$ [Eq. (2)] provides a deeper
characterization of measurement invasivity.

Examples.—The previous results establish a univocal relation between the
presence of memory effects and violation of DNI. Here we study different
$s$-$e$ interactions that lead to a dephasing non-Markovian dynamics. The
system is a qubit (two-level system) with basis of states $|\pm\rangle.$ Its
density matrix reads

$\rho_{t}=\left(\begin{array}[]{cc}\langle+|\rho_{0}|+\rangle&d(t)\langle+|\rho_{0}|-\rangle\\\
d(t)^{\ast}\langle-|\rho_{0}|+\rangle&\langle-|\rho_{0}|-\rangle\end{array}\right).$
(11)

The populations remain constant while the coherences are characterized by the
decay function $d(t).$

In correspondence with the analyzed dynamics, the first evolution is set by a
stochastic Hamiltonian anderson ; GaussianNoise

$\frac{d\rho_{t}^{st}}{dt}=-i\xi(t)[\sigma_{z},\rho_{t}^{st}].$ (12)

Here, $\sigma_{z}$ is the $z$-Pauli matrix. The noise has a null average
$\overline{\xi(t)}=0$ and correlation
$\overline{\xi(t)\xi(t^{\prime})}=(\gamma/2\tau_{c})\exp[-(t-t^{\prime})/\tau_{c}].$
From the system density matrix $\rho_{t}=\overline{\rho_{t}^{st}},$ the
coherence decay reads
$d(t)=\exp\left[-2\gamma(t-\tau_{c}(1-e^{-t/\tau_{c}})\right].$ In the limit
of a vanishing correlation time, $\tau_{c}/\gamma\rightarrow 0,$ an
exponential (Markovian) decay is recovered.

A unitary $s$-$e$ model is defined by a spin bath ZurekRMP ; Paz ,

$\frac{d\rho_{t}^{se}}{dt}=-ig\Big{[}\sigma_{z}\otimes\sum\nolimits_{j=1}^{n}\sigma_{z}^{(j)},\rho_{t}^{se}\Big{]}.$
(13)

Here, $g$ is a coupling parameter while $\sigma_{z}^{(j)}$ is the $z$-Pauli
matrix of the $j$ environment spin. The system density matrix
$\rho_{t}=\mathrm{Tr}_{e}[\rho_{t}^{se}]$ follows by tracing the bipartite
$s$-$e$ state. Assuming that each bath spin begins in an identity state, the
system coherence decay reads $d(t)=[\cos(2gt)]^{n}.$

A dissipative $s$-$e$ model is set by a non-diagonal multipartite Lindblad
equation Clerck ; noDiag

$\frac{d\rho_{t}^{se}}{dt}=\sum_{j,k=1}^{n}\Gamma_{jk}(\sigma_{z}^{(j)}\rho_{t}^{se}\sigma_{z}^{(k)}-\frac{1}{2}\\{\sigma_{z}^{(k)}\sigma_{z}^{(j)},\rho_{t}^{se}\\}_{+}),$
(14)

where $\Gamma_{jk}=(\gamma-\chi)\delta_{jk}+\chi(i)^{j-1}(-i)^{k-1}.$ When
$\chi=0,$ each qubit obey a Markovian dephasing evolution with rate $\gamma.$
When $\chi\neq 0$ all subsystems are coupled to each other, leading to the
development of memory effects. Taking the first qubit as the system of
interest, the coherence decay reads $d(t)=e^{-2\gamma t}[\cos(2\chi
t)]^{\bar{n}},$ where $\bar{n}=\mathrm{Int}(n/2)$ is the integer part of
$n/2.$ This result relies on assuming that all environment qubit subsystems
begin in a completely mixed state noDiag .

For the three previous models it is possible to calculate $P_{3}(z,y,x)$ in an
exact analytical way, where $z=\pm 1,$ $y=\pm 1,$ and $x=\pm 1.$ We assume
that three measurement processes are performed successively in the Bloch
directions $\hat{x}-\hat{n}-\hat{x},$ where the vector
$\hat{n}=\hat{n}(\theta,\phi)$ is defined by polar angles $(\theta,\phi).$
Using that $d(t)=d(t)^{\ast},$ we get

$\frac{P_{3}(z,y,x)}{P_{1}(x)}=\frac{1}{4}[1+yxf(t)+zyf(\tau)+zxf(t,\tau)],$
(15)

where $f(t)=\sin(\theta)\cos(\phi)d(t),$ while

$f(t,\tau)=\frac{1}{2}\sin^{2}(\theta)[d(t+\tau)+\cos(2\phi)d(t,\tau)].$ (16)

The function $d(t)$ is the coherence decay of each model, while $d(t,\tau)$
differ in each case suple . In all cases $P_{3}(z,y,x)$ does not fulfill (in
general) a Markovian property.

Figure 1: Invasivity measure $I(t,\tau)$ at equal time-intervals $(\tau=t)$
for different underlying open system dynamics, where the measurements are
performed in Bloch directions $\hat{x}-\hat{n}(\theta,\phi)-\hat{x}$ with
$\phi=0.$ (a) and (b) Stochastic Hamiltonian model [Eq. (12)]. (c) Hamiltonian
$s$-$e$ model [Eq. (13)]. (d) Dissipative model [Eq. (14)]. The parameters are
indicated in each plot. The DNI Bloch-direction is $\theta=\pi/2,$ $\phi=0.$

From Eqs. (1) and (15) it is easy to obtain
$P_{3}(z,x)=[1+zxf(t,\tau)]P_{1}(x)/2.$ Given that the last measurement is
performed in the $\hat{x}$-Bloch direction, $P_{2}(z,x)$ can also be obtained
from Eq. (15) under the steps $\sum_{z},$ the replacements $y\rightarrow z,$
$t\rightarrow t+\tau,$ and taking $\theta=\pi/2,$ $\phi=0,$ which deliver
$P_{2}(z,x)=[1+zxd(t+\tau)]P_{1}(x)/2.$ The invasivity distance Eq. (2)
therefore is

$I=I(t,\tau)=|f(t,\tau)-d(t+\tau)|.$ (17)

This expression for $I$ is valid for an arbitrary intermediate measurement
defined by the angles $(\theta,\phi).$ The DNI of Markovian dynamics [Eq. (5)]
is valid when this measurement is performed in the same basis where
$\rho_{t|x}$ is diagonal. Given that the first measurement is performed in the
$\hat{x}$-Bloch direction, Eq. (11) implies
$\langle\pm|\rho_{t|x}|\pm\rangle=1/2$ and
$\langle\pm|\rho_{t|x}|\mp\rangle=xd(t)/2.$ Defining
$M_{\hat{\imath}}\equiv\mathrm{Tr}_{s}[\sigma_{i}\rho_{t|x}]$ where
$\sigma_{i}$ are the Pauli matrixes, we get $M_{\hat{x}}=xd(t),$ while
$M_{\hat{y}}=M_{\hat{z}}=0.$ Hence, $\rho_{t|x}$ is diagonal ($\forall t$) in
the $\hat{x}$-Bloch direction blum . Consequently, the intermediate observable
commutates with $\rho_{t|x}$ when $\theta=\pi/2$ and $\phi=0.$

In Fig. 1 we plot $I(t,\tau)$ for the different open dynamics. The
intermediate measurement is in the $\hat{z}$-$\hat{x}$ Bloch plane: $\phi=0.$
Fig. 1(a) and (b) correspond to the stochastic Hamiltonian evolution [Eq.
(12)]. In (a) the parameters approach a Markovian white noise limit,
$\tau_{c}/\gamma\simeq 0\Rightarrow d(t)\simeq\exp(-2\gamma t)\Rightarrow
P_{3}(z,y,x)\simeq P_{3}(z|y)P_{2}(y|x)P_{1}(x)$ [Eq. (15)]. Invasiveness is
clearly observed, $I(t,\tau)\neq 0.$ Nevertheless, when
$\theta\rightarrow\pi/2,$ the DNI of Markovian dynamics is corroborated
$I(t,\tau)$ $\rightarrow 0.$ In contrast, in (b) when $\tau_{c}/\gamma>0,$
even when the intermediate observable commutates with the system density
matrix $(\theta=\pi/2$ and $\phi=0),$ consistently with our results, due to
the presence of memory effects $I(t,\tau)$ does not vanish.

In Fig. 1(c) we plot $I(t,\tau)$ for the spin environment model [Eq. (13)].
Given that the number of spins is finite all behaviors are periodic in time.
Again, even when the measurement and system state commutate $(\theta=\pi/2$
and $\phi=0),$ DNI is violated, $I(t,\tau)>0.$ The same result is valid for
the dissipative model [Eq. (14)] when $\chi/\gamma\neq 0,$ Fig. 1(d). When
$\chi/\gamma\rightarrow 0,$ a Markovian regime is approached. The behavior of
$I(t,\tau)$ becomes indistinguishable from that shown in Fig. 1(a),
corroborating the DNI of Markovian dynamics in this alternative model.

Figure 2: LGI parameter $K(t,\tau)\equiv d(t)+d(\tau)-d(t+\tau)$ [Eq. (18)]
for equal time-intervals. (a) Stochastic Hamiltonian model [Eq. (12)]. (b)
Hamiltonian $s$-$e$ model [Eq. (13)]. (c) Dissipative model [Eq. (14)], while
(d) shows the associated system coherence decay. The dotted line corresponds
to the Markovian limit $d(t)=\exp[-2\gamma t].$

Assuming that $\rho_{0}$ is diagonal in the $\hat{x}$-Bloch direction, the
density matrix [Eq. (11)] remains diagonal in that base. Hence, violation of
LGI due to memory effects can be checked by choosing the three observables
$\\{O_{m}\\}$ $(m=x,y,z)$ as diagonal in the same $\hat{x}$-direction. Given
that only two measurements are explicitly performed, for the three studied
$s$-$e$ models, Eq. (10) $(O_{m}=m)$ can be expressed in terms of the
corresponding coherence decay,

$-3\leq d(t)+d(\tau)-d(t+\tau)\leq 1.$ (18)

In Fig. 2 we study the validity of this inequality. For the stochastic
Hamiltonian model (a), LGI is only valid when a Markovian white noise limit is
attained, that is $\tau_{c}/\gamma=0.$ For the unitary $s$-$e$ model (b),
given the absence of a Markovian limit (in probability), LGI is violated
independently of the number of environment spins. The dissipative model (c)
presents a memory induced transition. In fact, for $\chi/\gamma\lesssim 0.17$
LGI is valid, which includes the Markovian case $\chi/\gamma=0.$ Nevertheless,
LGI is violated for $\chi/\gamma\gtrsim 0.17.$ In (d) we show the
corresponding system coherence decay $d(t).$ All of them are quasi-monotonic.
Thus, the memory induced transition, as defined in the present operational
approach [Eq. (15)], does not relies on any revival in the coherence decay.

Conclusions.—A deep relation between measurement invasivity and the presence
of memory effects in open quantum systems has been established. Based on an
operational (measurement based) approach, it was found that non-Markovian
dynamics are intrinsically modified by a measurement process even when the
corresponding observable commutates with the system state. This violation of
DNI disappears when the dynamic approaches a memoryless Markovian regime.

A measure of the previous relation was introduced. It relies on performing
three system measurement processes, where the observable of the intermediate
one must to commutate with the (pre-measurement) system density matrix. By
taking into account the arbitrariness of the first and last measurements, the
invasiveness indicator only vanishes in a Markovian regime. In stretched
relation, we found that LGI is always obeyed by Markovian dynamics if the
pairs of involved measurements commutate with the system state. The studied
models support the main conclusions. All of them can be implemented in
different experimental platforms. The DNI measurement-basis can be determine
from standard tomographic techniques. The examples also allowed us to
demonstrate that pure memory effects can drive a transition in the validity of
LGI.

Acknowledgments.—The author thanks Mónica M. Guraya for a critical reading of
the manuscript. This paper was supported by Consejo Nacional de
Investigaciones Científicas y Técnicas (CONICET), Argentina.

## References

  * (1) J. S. Bell, Speakable and unspeakable in quantum mechanics, (Cambridge University Press, 1987).
  * (2) Scientific Background on the Nobel Prize in Physics 2022, The Nobel Committee for Physics, The Royal Swedish Academy of Sciences; https://www.aps.org/newsroom/pressreleases/nobel2022. cfm
  * (3) A. J. Leggett and A. Garg, Quantum Mechanics versus Macroscopic Realism: Is the Flux There when Nobody Looks?, Phys. Rev. Lett. 54, 857 (1985).
  * (4) C. Emary, N. Lambert, and F. Nori, Leggett-Garg Inequalities, Rep. Prog. Phys. 77, 016001 (2014); arXiv:1304.5133v3.
  * (5) J. P. Paz and G. Mahler, Proposed Test for Temporal Bell Inequalities, Phys. Rev. Lett. 71, 3235 (1993).
  * (6) S. F. Huelga, T. W. Marshall, and E. Santos, Proposed test for realist theories using Rydberg atoms coupled to a high-Q resonator, Phys. Rev. A 52, R2497 (1995).
  * (7) R. Ruskov, A. N. Korotkov, and A. Mizel, Signatures of Quantum Behavior in Single-QubitWeak Measurements, Phys. Rev. Lett. 96, 200404 (2006).
  * (8) A. N. Jordan, A. N. Korotkov, and M. Büttiker, Leggett-Garg Inequality with a Kicked Quantum Pump, Phys. Rev. Lett. 97, 026805 (2006).
  * (9) J. Kofler and C. Brukner, Conditions for Quantum Violation of Macroscopic Realism, Phys. Rev. Lett. 101, 090403 (2008).
  * (10) N. S. Williams and A. N. Jordan, Weak Values and the Leggett-Garg Inequality in Solid-State Qubits, Phys. Rev. Lett. 100, 026804 (2008).
  * (11) N. Lambert, C. Emary, Y.-N. Chen, and F. Nori, Distinguishing Quantum and Classical Transport through Nanostructures, Phys. Rev. Lett. 105, 176801 (2010).
  * (12) J. Dressel, C. J. Broadbent, J. C. Howell, and A. N. Jordan, Experimental Violation of Two-Party Leggett-Garg Inequalities with Semiweak Measurements, Phys. Rev. Lett. 106, 040402 (2011).
  * (13) G. Waldherr, P. Neumann, S. F. Huelga, F. Jelezko, and J. Wrachtrup, Violation of a Temporal Bell Inequality for Single Spins in a Diamond Defect Center, Phys. Rev. Lett. 107, 090401 (2011).
  * (14) V. Athalye, S. Singha Roy, and T. S. Mahesh, Investigation of the Leggett-Garg Inequality for Precessing Nuclear Spins, Phys. Rev. Lett. 107, 130402 (2011).
  * (15) C.-M. Li, N. Lambert, Y.-N. Chen, G.-Y. Chen, and F. Nori, Witnessing Quantum Coherence: from solid-state to biological systems, Sci. Rep. 2, 885 (2012).
  * (16) J. Kofler and C. Brukner, Condition for macroscopic realism beyond the Leggett-Garg inequalities, Phys. Rev. A 87, 052115 (2013).
  * (17) J. P. Groen, D. Ristè, L. Tornberg, J. Cramer, P. C. de Groot, T. Picot, G. Johansson, and L. DiCarlo, Partial-Measurement Backaction and Nonclassical Weak Values in a Superconducting Circuit, Phys. Rev. Lett. 111, 090506 (2013).
  * (18) C. Budroni and C. Emary, Temporal Quantum Correlations and Leggett-Garg Inequalities in Multilevel Systems, Phys. Rev. Lett. 113, 050401 (2014); R. Uola, G. Vitagliano, and C. Budroni, Leggett-Garg macrorealism and the quantum nondisturbance conditions, Phys. Rev. A 100, 042117 (2019).
  * (19) G. Schild and C. Emary, Maximum violations of the quantum-witness equality, Phys. Rev. A 92, 032101 (2015).
  * (20) C. Robens, W. Alt, D. Meschede, C. Emary, and A. Alberti, Ideal Negative Measurements in Quantum Walks Disprove Theories Based on Classical Trajectories, Phys. Rev. X 5, 011003 (2015).
  * (21) Z.-Q. Zhou, S. F. Huelga, C.-F. Li, and G.-C. Guo, Experimental Detection of Quantum Coherent Evolution through the Violation of Leggett-Garg-Type Inequalities, Phys. Rev. Lett. 115, 113002 (2015).
  * (22) F. J. Gómez-Ruiz, J. J. Mendoza-Arenas, F. J. Rodríguez, C. Tejedor, and L. Quiroga, Universal two-time correlations, out-of-time-ordered correlators, and Leggett-Garg inequality violation by edge Majorana fermion qubits, Phys. Rev. B 97, 235134 (2018); F. J. Gómez-Ruiz, J. J. Mendoza-Arenas, F. J. Rodríguez, C. Tejedor, and L. Quiroga, Quantum phase transitions detected by a local probe using time correlations and violations of Leggett-Garg inequalities, Phys. Rev. B 93, 035441 (2016).
  * (23) J. J. Halliwell, Decoherent histories and measurement of temporal correlation functions for Leggett-Garg inequalities, Phys. Rev. A 94, 052131 (2016).
  * (24) S. V. Moreira and M. Terra Cunha, Quantifying quantum invasiveness, Phys. Rev. A 99, 022124 (2019).
  * (25) F. Giacosa and G. Pagliara, Leggett-Garg inequalities and decays of unstable systems, Phys. Rev. A 104, 052225 (2021).
  * (26) S. Majidy, J. J. Halliwell, and Raymond Laflamme, Detecting violations of macrorealism when the original Leggett-Garg inequalities are satisfied, Phys. Rev. A 103, 062212 (2021).
  * (27) A. Santini and V. Vitale, Experimental violations of Leggett-Garg inequalities on a quantum computer, Phys. Rev. A 105, 032610 (2022).
  * (28) M. Thenabadu and M. D. Reid, Bipartite Leggett-Garg and macroscopic Bell-inequality violations using cat states: Distinguishing weak and deterministic macroscopic realism, Phys. Rev. A 105, 052207 (2022); M. Thenabadu and M. D. Reid, Macroscopic delayed choice and retrocausality: Quantum eraser, Leggett-Garg, and dimension witness tests with cat states, Phys. Rev. A 105, 062209 (2022).
  * (29) M. M. Ettefaghi, R. Ramazani Arani, and Z. S. Tabatabaei Lotfi, Gravitational effects on quantum coherence in neutrino oscillation, Phys. Rev. D 105, 095024 (2022).
  * (30) A. Matsumura, Y. Nambu, and K. Yamamoto, Leggett-Garg inequalities for testing quantumness of gravity, Phys. Rev. A 106, 012214 (2022).
  * (31) R. Alicki and K. Lendi, Quantum Dynamical Semigroups and Applications, Lect. Notes Phys. 717 (Springer, Berlin Heidelberg, 1987).
  * (32) H. P. Breuer and F. Petruccione, The theory of open quantum systems, (Oxford University Press, 2002).
  * (33) I. de Vega and D. Alonso, Dynamics of non-Markovian open quantum systems, Rev. Mod. Phys. 89, 015001 (2017).
  * (34) H. P. Breuer, E. M. Laine, J. Piilo, and V. Vacchini, Colloquium: Non-Markovian dynamics in open quantum systems, Rev. Mod. Phys. 88, 021002 (2016).
  * (35) A. Rivas, S. F. Huelga, and M. B. Plenio, Quantum non-Markovianity: characterization, quantification and detection, Rep. Prog. Phys. 77, 094001 (2014).
  * (36) H. P. Breuer, E. M. Laine, and J. Piilo, Measure for the Degree of Non-Markovian Behavior of Quantum Processes in Open Systems, Phys. Rev. Lett. 103, 210401 (2009).
  * (37) M. M. Wolf, J. Eisert, T. S. Cubitt, and J. I. Cirac, Assessing Non-Markovian Quantum Dynamics, Phys. Rev. Lett. 101, 150402 (2008).
  * (38) A. Rivas, S. F. Huelga, and M. B. Plenio, Entanglement and Non-Markovianity of Quantum Evolutions, Phys. Rev. Lett. 105, 050403 (2010).
  * (39) D. Chruściński and S. Maniscalco, Degree of Non-Markovianity of Quantum Evolution, Phys. Rev. Lett. 112, 120404 (2014).
  * (40) X.-M. Lu, X. Wang, and C. P. Sun, Quantum Fisher information flow and non-Markovian processes of open systems, Phys. Rev. A 82, 042103 (2010).
  * (41) S. Lorenzo, F. Plastina, and M. Paternostro, Geometrical characterization of non-Markovianity, Phys. Rev. A 88, 020102(R) (2013).
  * (42) M. J. W. Hall, J. D. Cresser, L. Li, and E. Andersson, Canonical form of master equations and characterization of non-Markovianity, Phys. Rev. A 89, 042120 (2014).
  * (43) F. A. Pollock, C. Rodríguez-Rosario, T. Frauenheim, M. Paternostro, and K. Modi, Operational Markov Condition for Quantum Processes, Phys. Rev. Lett. 120, 040405 (2018); F. A. Pollock, C. Rodríguez-Rosario, T. Frauenheim, M. Paternostro, and K. Modi, Non-Markovian quantum processes: Complete framework and efficient characterization, Phys. Rev. A 97, 012127 (2018).
  * (44) A. A. Budini, Quantum Non-Markovian Processes Break Conditional Past-Future Independence, Phys. Rev. Lett. 121, 240401 (2018); A. A. Budini, Conditional past-future correlation induced by non-Markovian dephasing reservoirs, Phys. Rev. A 99, 052125 (2019).
  * (45) P. Taranto, F. A. Pollock, S. Milz, M. Tomamichel, and K. Modi, Quantum Markov Order, Phys. Rev. Lett. 122, 140401 (2019); P. Taranto, S. Milz, F. A. Pollock, and K. Modi, Structure of quantum stochastic processes with finite Markov order, Phys. Rev. A 99, 042108 (2019).
  * (46) M. R. Jørgensen and F. A. Pollock, Exploiting the Causal Tensor Network Structure of Quantum Processes to Efficiently Simulate Non-Markovian Path Integrals, Phys. Rev. Lett. 123, 240602 (2019).
  * (47) M. Bonifacio and A. A. Budini, Perturbation theory for operational quantum non-Markovianity, Phys. Rev. A 102, 022216 (2020).
  * (48) L. Han, J. Zou, H. Li, and B. Shao, Non-Markovianity of A Central Spin Interacting with a Lipkin–Meshkov–Glick Bath via a Conditional Past–Future Correlation, Entropy 22, 895 (2020).
  * (49) M. Ban, Operational non-Markovianity in a statistical mixture of two environments, Phys. Lett. A 397, 127246 (2021).
  * (50) T. de Lima Silva, S. P. Walborn, M. F. Santos, G. H. Aguilar, and A. A. Budini, Detection of quantum non-Markovianity close to the Born-Markov approximation, Phys. Rev. A 101, 042120 (2020).
  * (51) S. Yu, A. A. Budini, Y.-T. Wang, Z.-J. Ke, Y. Meng, W. Liu, Z.-P. Li, Q. Li, Z.-H. Liu, J.-S. Xu, J.-S. Tang, C.-F. Li, and G.-C. Guo, Experimental observation of conditional past-future correlations, Phys. Rev. A 100, 050301(R) (2019).
  * (52) Y. -Y. Hsieh, Z. -Y. Su, and H. -S. Goan, Non-Markovianity, information backflow, and system-environment correlation for open-quantum-system processes, Phys. Rev. A 100, 012120 (2019).
  * (53) A. A. Budini, Detection of bidirectional system-environment information exchanges, Phys. Rev. A 103, 012221 (2021).
  * (54) N. G. van Kampen, Stochastic Processes in Physics and Chemistry, (North-Holland, Amsterdam, 1981).
  * (55) A. M. Souza, J. Li, D. O. Soares-Pinto, R. S. Sarthour, I. S. Oliveira, S. F. Huelga, M. Paternostro, and F. L. Semião, Experimental Demonstration of non-Markovian Dynamics via a Temporal Bell-like Inequality, arXiv:1308.5761v1 (2013).
  * (56) P.-W. Chen and Md. M. Ali, Investigating Leggett-Garg inequality for a two level system under decoherence in a non-Markovian dephasing environment, Sci. Rep. 4, 6165 (2014).
  * (57) H. Li , J. Zou, and B. Shao, Enhanced quantumness via non-Markovianity, Phys. Rev. A 104, 052201 (2021).
  * (58) S. Milz, D. Egloff, P. Taranto, T. Theurer, M. B. Plenio, A. Smirne, and S. F. Huelga, When Is a Non-Markovian Quantum Process Classical?, Phys. Rev. X 10, 041049 (2020).
  * (59) M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum Information (Cambridge University Press, Cambridge, England, 2000).
  * (60) H. Ollivier and W. H. Zurek, Quantum Discord: A Measure of the Quantumness of Correlations, Phys. Rev. Lett. 88, 017901 (2002).
  * (61) L. Henderson, V. Vedral, Classical, quantum and total correlations, J. Phys. A 34, 6899 (2001); B. Dakic, V. Vedral, and C. Brukner, Necessary and Sufficient Condition for Nonzero Quantum Discord, Phys. Rev. Lett. 105, 190502 (2010).
  * (62) K. Modi, A. Brodutch, T. Paterek, and V. Vedral, The classical-quantum boundary for correlations: Discord and related measures, Rev. Mod. Phys. 84, 1655 (2012).
  * (63) A. A. Budini, Almost all dynamics generate quantum discord, to be published; A. Ferraro, L. Aolita, D. Cavalcanti, F. M. Cucchietti, and A. Acín, Almost all quantum states have nonclassical correlations, Phys. Rev. A 81, 052318 (2010).
  * (64) Cywiński, Equivalence of qubit-environment entanglement and discord generation via pure dephasing interactions and the resulting consequences, Phys. Rev. A 97, 012306 (2018).
  * (65) X. Hu, Y. Gu, Q. Gong, and G. Guo, Necessary and sufficient condition for Markovian-dissipative-dynamics-induced quantum discord, Phys. Rev. A 84, 022113 (2011).
  * (66) P. W. Anderson and P. R. Weiss, Exchange Narrowing in Paramagnetic Resonance, Rev. Mod. Phys. 25, 269 (1953); P. T. Callaghan, Principles of Nuclear Magnetic Resonance Microscopy, (Clarendom Press, Oxford, 1991).
  * (67) A. A. Budini, Quantum systems subject to the action of classical stochastic fields, Phys. Rev. A 64, 052110 (2001).
  * (68) W. H. Zurek, Decoherence, einselection, and the quantum origins of the classical, Rev. Mod. Phys. 75, 715 (2003).
  * (69) F. M. Cucchietti, J. P. Paz, and W. H. Zurek, Decoherence from spin environments, Phys. Rev. A 72, 052113 (2005).
  * (70) A. Seif , Y.-X. Wang , and A. A. Clerk, Distinguishing between Quantum and Classical Markovian Dephasing Dissipation, Phys. Rev. Lett. 128, 070402 (2022).
  * (71) A. A. Budini, Memory effects in multipartite systems coupled by non-diagonal dephasing mechanisms, arXiv: 2209.00400 (2022).
  * (72) For the model Eq. (12), $d(t,\tau)=d(t+\tau)\exp[4\gamma\tau_{c}(1-e^{-t/\tau_{c}})(1-e^{-\tau/\tau_{c}})],$ for the model Eq. (13), $d(t,\tau)=d(t-\tau),$ and for the model Eq. (14), $d(t,\tau)=\exp[-2\gamma(t+\tau)]\cos[2\chi(t-\tau)]^{\bar{n}}.$
  * (73) K. Blum, Density Matrix Theory and Applications, (Plenum Press, New York, 1996).