# Jetfire: Efficient and Accurate Transformer Pretraining with
INT8 Data Flow and Per-Block Quantization

Haocheng Xi  Yuxiang Chen  Kang Zhao  Kaijun Zheng  Jianfei Chen  Jun Zhu

###### Abstract

Pretraining transformers are generally time-consuming. Fully quantized
training (FQT) is a promising approach to speed up pretraining. However, most
FQT methods adopt a quantize-compute-dequantize procedure, which often leads
to suboptimal speedup and significant performance degradation when used in
transformers due to the high memory access overheads and low-precision
computations. In this work, we propose Jetfire, an efficient and accurate INT8
training method specific to transformers. Our method features an INT8 data
flow to optimize memory access and a per-block quantization method to maintain
the accuracy of pretrained transformers. Extensive experiments demonstrate
that our INT8 FQT method achieves comparable accuracy to the FP16 training
baseline and outperforms the existing INT8 training works for transformers.
Moreover, for a standard transformer block, our method offers an end-to-end
training speedup of 1.42x and a 1.49x memory reduction compared to the FP16
baseline.

Machine Learning, ICML

##  1 Introduction

Recently, large-scale pre-trained transformer-based models such as GPT-4
(OpenAI, 2023), LLAMA (Touvron et al., 2023), and PaLM (Anil et al., 2023)
have attained significant breakthroughs in multiple fields, including natural
language processing and computer vision. However, pre-training transformers
from scratch are extremely resource-intensive since they require numerous
computations and high-bandwidth memory for updating weights and accessing huge
amounts of training tokens, respectively.

Figure 1: Visualization of INT8 data flow. (a) Floating point training with FP
data flow. (b) Existing works on quantized training with FP data flow. (c)
Ours INT8 training forward process, with INT8 data flow.
$\boldsymbol{\mathbf{X}}$ refers to the activation, and
$\boldsymbol{\mathbf{S}}$ refers to the corresponding quantization scale
factors.

To accelerate the pre-training of transformers, fully quantized training (FQT)
has emerged as a promising technique to speed up both the forward and backward
passes. FQT integrates quantizers and dequantizers into the original full-
precision computational graph. In this way, the expensive floating-point
operations during training are replaced with cheaper low-precision
alternatives, and activations saved for the backward pass are stored with
fewer bits. Thus, both computations and memory bandwidths are largely reduced.
Typical FQT works include (Banner et al., 2018; Wang et al., 2018b;
Micikevicius et al., 2018; Chen et al., 2020; Wortsman et al., 2023; Xi et
al., 2023; Sun et al., 2020; Chmiel et al., 2021; Sun et al., 2019).

However, the existing FQT studies still have three limitations: 1) Existing
FQT methods are not accurate enough for Transformer models. Previous FQT
methods were mainly designed for CNNs (Zhu et al., 2020; Zhao et al., 2021),
and directly applying these methods to transformer models will result in
significant accuracy degradation. Those few papers that focus on transformers
often encounter significant quantization errors when computing weight
gradients. Therefore, they leave this part in floating-point precision, which
limits its overall speedup. 2) Most FQT methods only focus on the reduction of
computations, regardless of data access overheads (Wortsman et al., 2023).
Nevertheless, for a transformer block, only the linear layers are compute-
bounded; other layers, such as LayerNorm and activation functions, are
normally memory-bounded. Failing to optimize the memory access leads to
suboptimal training acceleration. 3) Some FQT techniques require specialized
hardware and are not applicable to general computing platforms. For instance,
FP8 (Peng et al., 2023; Perez et al., 2023) training is only supported on GPUs
with Hopper architecture (nvi, 2022). Not to mention that those hybrid-format
quantized training methods rely on application-specific integrated circuits to
deliver the desired performance.

To address these limitations, in this work, we propose Jetfire, an INT8
pretraining method for transformers. Specifically, to improve training
efficiency, we propose using INT8 data flow. As shown in Fig. 1, the INT8 data
flow simply refers to the utilization of 8-bit integers for data movement
among operators. Compared to the FP16 data flow, the INT8 data flow is 2x
faster in theory. In particular, the INT8 data flow considerably enhances the
speed of memory-constrained operators, including Layernorm and GELU.

In addition to INT8 flow, we propose per-block quantization that is
specialized for transformer pretraining. On one hand, compared to per-tensor
or per-token quantization (Wortsman et al., 2023), our per-block quantization
better preserves the accuracy of pretrained transformers. On the other hand,
per-block quantization brings practical training speedup on tensor cores
compared to per-channel quantization. Furthermore, our method is applicable to
a wide range of computing platforms supporting INT8 matrix multiplications
(MMs).

We validate our INT8 FQT method for transformers across a diverse range of
tasks, including machine translation, image classification, and generative
model pretraining. Jetfire consistently attains comparable accuracy with the
FP16 training baseline and has superior accuracy compared with the existing
works of INT8 training (Wortsman et al., 2023). On NVIDIA RTX 4090 GPUs, our
custom linear and non-linear operators achieve speedups of 1.4x and 1.8x,
respectively, compared to the FP16 baselines. Besides, our Jetfire achieves a
speedup of 1.42x for a single transformer block and 1.49x memory reduction
compared with the FP16 baseline.

##  2 Related Work

#### Post-Training Quantization and Quantization-Aware Training

Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) aim to
find a good low-precision representation for a full-precision model. Post-
Training Quantization (PTQ) (Chee et al., 2023; Xiao et al., 2023; Dettmers et
al., 2022; Kim et al., 2023; Lin et al., 2023; Kim et al., 2021; Jacob et al.,
2018; Liu et al., 2021) converts the pre-trained model’s weights to lower-bit
representations directly. Quantization-Aware Training (QAT) (Dong et al.,
2019b, a; Shen et al., 2019; Zhang et al., 2020; Bai et al., 2020; Tang et
al., 2022; Esser et al., 2019) involves retraining the model to adapt its
weights and regain accuracy after the quantization process.

Table 1: Comparison with related works. SB refers to SwitchBack (Wortsman et
al., 2023), TE refers to TransformerEngine (Nvidia, 2022).

Support | Jetfire (Ours) | SB | TE | FP8-LM | DAQ
---|---|---|---|---|---
Transformers | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$
INT8 quantization | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\checkmark$
8-bit gradient | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\times$
8-bit data flow | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$

#### Fully Quantized Training

Fully Quantized Training (FQT)(Wang et al., 2018b; Banner et al., 2018; Xi et
al., 2023; Perez et al., 2023; Wortsman et al., 2023; Zhu et al., 2020; Zhao
et al., 2021; Micikevicius et al., 2018; Nvidia, 2022) has been introduced as
a technique to accelerate the training process of neural networks. FQT
requires quantizing both the forward propagation and backward propagation to
actually accelerate the whole training process. Nowadays 16-bit quantization
has been commonly employed with float16 and bfloat16 data formats in training.
It introduces loss scaling to prevent underflow and overflow problem.

For INT8 training, the majority of the work focuses on quantization of CNNs
(Zhu et al., 2020; Zhao et al., 2021; Zhou et al., 2021). SwitchBack (Wortsman
et al., 2023) introduces per-token quantization and successfully applies INT8
training to CLIP models for the first time, but still leaves the calculation
of weight gradient in FP. For LLM pre-training, per-token quantization still
results in significant accuracy loss due to channel-wise outliers.

With the introduction of the Hopper architecture, FP8 training has also gained
attention. TransformerEngine (Nvidia, 2022) incorporates per-layer scaling to
reduce quantization errors and proposes using E4M3 during forward and E5M2
during backward passes to adapt. (Perez et al., 2023) explores adjusting per-
tensor scaling biases to improve accuracy, while (Peng et al., 2023)
investigates further quantizing optimizer states and the weight’s master copy
to FP8. However, these methods rely on GPUs with the Hopper architecture and
cannot be applied to a wider range of GPUs.

As summarized in Table. 1, our method supports INT8 quantization, 8-bit
gradient, and 8-bit data flow at the same time, compared to other FQT methods.

##  3 INT8 Data Flow

In this section, we introduce our approach for INT8 training with INT8 data
flow. We begin by defining the concept of Fully Quantized Training (FQT).

###  3.1 Fully Quantized Training

Consider a network consisting of linear and nonlinear layers. In the forward
pass, these layers can be represented as
$\boldsymbol{\mathbf{Y}}=\boldsymbol{\mathbf{F}}(\boldsymbol{\mathbf{X}},\boldsymbol{\mathbf{W}})$,
where $\boldsymbol{\mathbf{X}}$ is the activation, $\boldsymbol{\mathbf{W}}$
is the weight, and $\boldsymbol{\mathbf{Y}}$ is the output, also the next
layer’s activation. In the backward pass, each layer takes the gradient
$\nabla_{\boldsymbol{\mathbf{Y}}}$, $\boldsymbol{\mathbf{X}}$, and
$\boldsymbol{\mathbf{W}}$ as inputs and computes the activation gradient and
weight gradient by
$\nabla_{\boldsymbol{\mathbf{X}}},\nabla_{\boldsymbol{\mathbf{W}}}=\boldsymbol{\mathbf{d}}\boldsymbol{\mathbf{F}}(\nabla_{\boldsymbol{\mathbf{Y}}},\boldsymbol{\mathbf{X}},\boldsymbol{\mathbf{W}})$.

Quantization accelerates training by utilizing low-precision computing units
on hardware. One notable example is matrix multiplication (MM) in the form of
$\boldsymbol{\mathbf{Y}}=\boldsymbol{\mathbf{X}}\boldsymbol{\mathbf{W}}^{\top}$.
When both input matrices are in low-precision format, the MM can have 2x
theoretical flops relative to an MM with full-precision inputs, where in this
paper we assume that the full-precision format is FP16 and the low-precision
format is INT8. Most FQT methods utilize such low-precision MM by a _quantize-
compute-dequantize (QCD) approach_ : (1) temporarily converting FP16 input
matrices to INT8 with a _quantizer_ $Q(\cdot)$; (2) perform the INT8 MM to get
an INT32 output; and (3) convert the output matrix back to FP16 with a
_dequantizer_ $Q^{-1}(\cdot)$. With QCD, a MM operator can be formulized as
$\boldsymbol{\mathbf{Y}}=\text{QCD-
MM}(\boldsymbol{\mathbf{X}},\boldsymbol{\mathbf{W}})=Q^{-1}(Q(\boldsymbol{\mathbf{X}})Q(\boldsymbol{\mathbf{W}}^{\top}))$.
As the QCD-MM operator has identical interface to FP16 MMs (i.e., both input
and output are still in FP16), we can accelerate training by simply replacing
all FP16 MM operators with QCD MMs.

However, QCD only reduces the _computing precision_ to INT8, while leaving the
_data flow precision_ in FP16. That is, MMs are performed under INT8, but the
input, output, and data transferred between layers are still in FP16, as
illustrated in Fig. 1. The practical speedup of QCD is limited by the _memory
bandwidth_. Modern GPUs have excessive computational power, while the GPU
memory bandwidth is scarce. An algorithm must have a high arithmetic intensity
(i.e., ratio of computing operations to memory accesses) to run fast on GPUs.
Unfortunately, the QCD approach’s arithmetic intensity is low: the computation
is cut by half, but the memory access is not reduced as much, since data are
still represented in FP16. More specifically, QCD has three drawbacks:

1\. Frequent quantization and dequantization operations incur additional
memory access overhead.
2\. Nonlinear operators cannot be accelerated.
3\. GPU memory consumption and communication costs remain high.

###  3.2 FQT with INT8 Data Flow

To address these challenges, we directly utilize INT8 _data flow_ throughout
the network. That is, we employ the INT8 data format for activations, weights,
and gradients, and both our linear and non-linear operators directly take INT8
matrices as inputs and get INT8 tensors as outputs.

To achieve this, we directly represent activation, weight, and gradient
tensors in a custom INT8 format defined in Sec. 4. Then, we redesign and
implement all operators used in transformer training, including linear
operators (Sec. 5) and nonlinear operators (Sec. 6), allowing them to directly
use our custom INT8 format as inputs/outputs rather than FP16. The custom INT8
format is carefully designed to ensure that the operators can be implemented
efficiently on GPUs, while maintaining accuracy. Such INT8 data flow is
compared with QCD in Fig. 1.

With the INT8 data flow, we reduced the amount of memory access in the
training algorithm, resulting in better efficiency. In a nutshell, our
operators read/write INT8 data from global memory in a block-wise fashion, and
perform the quantize/dequantize/compute operations on chip within shared
memory and registers. In this way, both computation and memory access can be
reduced by half, and the arithmetic intensity remains high. A direct
consequence is that, our method can accelerate nonlinear operators, since
their memory access is also cut by half. Finally, as the data are stored in
INT8 format, the activation memory consumption and amount of communication
(tensor / pipeline parallelism) can be also cut by half, effectively avoiding
memory capacity and communication bottlenecks.

##  4 Per Block Quantization

(a)

(b)

Figure 2: (a) Channel-wise outliers in activation distribution. (b) Non-linear
operator is memory-bounded.

In this section, we introduce our INT8 numerical format. Typically, we can
approximate an FP16 matrix with an INT8 matrix
$\boldsymbol{\mathbf{X}}^{\text{INT8}}$ and a FP16 scale factor
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}}^{\text{FP16}}$, that is
$\boldsymbol{\mathbf{X}}^{\text{INT8}},\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}}^{\text{FP16}}=Q(\boldsymbol{\mathbf{X}}^{\text{FP16}})$.
Depending on the shape of the scale factor, there are different quantization
methods, including per-tensor quantization, per-token quantization, and per-
channel quantization. The INT8 numerical format must accurately support the
following three MMs of a linear layer in forward and back propagation:

$\displaystyle\boldsymbol{\mathbf{Y}}=\boldsymbol{\mathbf{X}}\boldsymbol{\mathbf{W}}^{\top},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\nabla_{\boldsymbol{\mathbf{X}}}=\nabla_{\boldsymbol{\mathbf{Y}}}\boldsymbol{\mathbf{W}},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\nabla_{\boldsymbol{\mathbf{W}}}=\nabla_{\boldsymbol{\mathbf{Y}}}^{\top}\boldsymbol{\mathbf{X}}.$

Researchers have observed that activations in transformers are difficult to
quantize (Dettmers et al., 2022; Xiao et al., 2023) due to the presence of
channel-wise outliers. We visualize this problem in Fig. 2a. Per-token
quantization assigns different scale factors for different tokens and often
results in large quantization errors since outliers appear channel-wise. On
the other hand, per-channel quantization assigns different scale factors for
different channels and has relatively lower quantization errors, as shown in
Sec. 7.3. In addition, gradient outliers also appear along the token axis
(Chen et al., 2020; Xi et al., 2023), which poses challenges for computing the
weight gradient
$\nabla_{\boldsymbol{\mathbf{W}}}=\nabla_{\boldsymbol{\mathbf{Y}}}^{\top}\boldsymbol{\mathbf{X}}$.
In this case, per-token quantization should be applied to the output gradient
$\nabla_{\boldsymbol{\mathbf{Y}}}$ to avoid large quantization error.

However, applying per-channel quantization for forward propagation or applying
per-token quantization for computing weight gradients both pose challenges in
practical hardware implementations. For a MM in the form
$\boldsymbol{\mathbf{C}}=\boldsymbol{\mathbf{A}}\boldsymbol{\mathbf{B}}$, we
call the 0th axis of $\boldsymbol{\mathbf{A}}$ and the 1st axis of
$\boldsymbol{\mathbf{B}}$ to be outer axes, as $\boldsymbol{\mathbf{C}}$ has
them; the other two axes are inner axes. INT8 MMs are performed with tensor
core WMMA (Warp Matrix Multiply-Accumulate) operations (Markidis et al.,
2018), and scaling can only be performed at the outer axis of MM if we want to
utilize tensor core. As a compromise, (Wortsman et al., 2023) only use per-
token quantization for forward propagation, sacrificing accuracy; and fall
back to FP16 when computing weight gradients, sacrificing speed.

We propose _per-block quantization_ to achieve computational efficiency and
preserve accuracy at the same time. For a matrix
$\boldsymbol{\mathbf{X}}\in\mathbb{R}^{N\times C}$, we partition
$\boldsymbol{\mathbf{X}}$ into blocks
$\boldsymbol{\mathbf{X}}_{ij}\in\mathbb{R}^{B\times B}$ along both row axis
and column axis, where $B$ is quantization block size, $i,j$ is the index of
quantization block along the token and channel axis. We assign a scale factor
$\boldsymbol{\mathbf{s}}_{ij}$ for each block $\boldsymbol{\mathbf{X}}_{ij}$
that corresponds to the maximum absolute value in the block. The method can be
formulated as:

$\displaystyle\small
Q(\boldsymbol{\mathbf{X}}_{ij})=\left\lceil\frac{\boldsymbol{\mathbf{X}}_{ij}}{\boldsymbol{\mathbf{s}}_{ij}}\right\rfloor,\leavevmode\nobreak\
\leavevmode\nobreak\
Q^{-1}(\boldsymbol{\mathbf{X}}_{ij}^{\text{INT8}},\boldsymbol{\mathbf{s}}_{ij})=\boldsymbol{\mathbf{X}}_{ij}^{\text{INT8}}\boldsymbol{\mathbf{s}}_{ij},$
(1)

where $\lceil\cdot\rfloor$ is the round operator. We visualize this method in
Fig. 3 for better understanding. Since our per-block quantization method
partitions along the inner axis, it restricts the impact of an outlier
channel/token within a block. Therefore the quantization error is controlled.
We will demonstrate in the next section that per-block quantization can be
also efficiently implemented on GPUs.

(a)  Figure 3: Visualization of the per-block quantization methodology. When
the original tensor has some outliers, our method can restrict its effect to a
$B\times B$ block.

##  5 Linear Layer Operator

In this section, we mainly discuss how our per-block quantization method
should be applied to linear layers. We highlight that our linear operator
adopts INT8 data flow, that takes INT8 as input and produces INT8 as output.

(a)  Figure 4: Different quantization methods for linear layer.

###  5.1 Notations

We consider the CUDA implementation of the following MM as an example in this
section:

(2)

which dimensions are represented as $N\times C\times D$.

In our MM operator, each input and output matrix is represented in per-block
INT8 format: a INT8 matrix and a FP16 scale matrix, as defined in Sec. 4. In
this case, we have INT8 input denoted as $\boldsymbol{\mathbf{X}}$ and
$\boldsymbol{\mathbf{W}}$, and we have scale factors denoted as
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}}\in\mathbb{R}^{L_{N}\times
L_{C}},\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{W}}}\in\mathbb{R}^{L_{D}\times
L_{C}},$ where $L_{N}=\tfrac{N}{B},L_{C}=\tfrac{C}{B},L_{D}=\tfrac{D}{B}$ is
the number of quantization blocks along every axis, and $B$ is the
quantization block size in Eq. (1). We utilize tensor cores to perform INT8
WMMA. For a single INT8 WMMA instruction, the inputs are two INT8 matrices of
shape $16\times 16$ and the output is an INT32 matrix of shape $16\times 16$.

###  5.2 3-Level Tiling of MM

An efficient MM implementation must organize the computation into blocks
(“tiling”) based on the GPU architecture. We tile the computation in 3 levels.
The block dimensions are listed in Table 2.

#### CUDA block level

When implementing our MM operator in CUDA, we first parallelize the
computation along the $N$ and $D$ axis. Every time we only calculate a
submatrix
$\boldsymbol{\mathbf{B}}_{\boldsymbol{\mathbf{Y}}}\in\mathbb{R}^{B_{N}\times
B_{D}}$ of the output matrix $\boldsymbol{\mathbf{Y}}$. We further divide $C$
into small segments of size $B_{C}$, and accumulate along this axis. The CUDA
block size $B_{N}\times B_{C}\times B_{D}$ is architecture specific. Depending
on shared memory capacity and number of threads, typical values are 32, 64,
128, or 256. We define $T_{N}=\left\lceil
N/B_{N}\right\rceil,T_{C}=\left\lceil C/B_{C}\right\rceil,T_{D}=\left\lceil
D/B_{D}\right\rceil$ to be the number of blocks along each axis of the MM.

For every iteration, we load submatrix
$\boldsymbol{\mathbf{X}}_{ik}\in\mathbb{R}^{B_{N}\times B_{C}}$ and
$\boldsymbol{\mathbf{W}}_{jk}\in\mathbb{R}^{B_{D}\times B_{C}}$ from global
memory to shared memory and compute the output submatrix
$\boldsymbol{\mathbf{Y}}_{ij}\in\mathbb{R}^{B_{N}\times B_{D}}$, where $1\leq
i\leq T_{N},1\leq j\leq T_{D},1\leq k\leq T_{C}$ are the CUDA block index
along the $N,D,C$ axis.

#### Quantization block level

We set $B_{N}$ and $B_{D}$ to be multiples of the quantization block size $B$,
and set $B_{C}=B$. In this case, $\boldsymbol{\mathbf{X}}_{ik}$ consists of
$R_{N}=\left\lceil B_{N}/B\right\rceil$ quantization blocks along its 0th-
axis, and we use $\boldsymbol{\mathbf{X}}_{ik,p}$ to denote the $p$-th
quantization block. Similarly, $\boldsymbol{\mathbf{W}}_{jk}$ consists of
$R_{D}=\left\lceil B_{D}/B\right\rceil$ quantization blocks along its 0th-axis
with $\boldsymbol{\mathbf{W}}_{jk,q}$ as the $q$-th block.

We use two nested _for loops_ to iterate over $R_{N}$ and $R_{D}$, load
$\boldsymbol{\mathbf{X}}_{ik,p}\in\mathbb{R}^{B\times B}$ and
$\boldsymbol{\mathbf{W}}_{jk,q}\in\mathbb{R}^{B\times B}$ from shared memory
to register and performing INT8 WMMA separately to get INT32 output
$\boldsymbol{\mathbf{Y}}_{ij,pq}\in\mathbb{R}^{B\times B}$, where $1\leq p\leq
R_{N},1\leq q\leq R_{D}$ is the quantization block index along $R_{N},R_{D}$
axis.

#### WMMA operation level

Within the computation of single quantization blocks, we utilize the INT8 WMMA
instruction for computation on register. Therefore, when we set $B=32$ as an
example, we need to perform $2^{3}=8$ WMMA instructions to complete the
computation, since a single WMMA instruction can only compute $16\times
16\times 16$ MM.

In summary, we divide the implementation of the MM operator into three levels.
First, at the CUDA block level, we divide the operator into sizes of
$B_{N}\times B_{C}\times B_{D}$ for computation. Then, at the quantization
block level, we further divide each CUDA block into sizes of $B\times B\times
B$. Finally, at the WMMA operation level, we divide the computation of each
quantization block based on the dimensions of the WMMA operation.

###  5.3 Quantize and Dequantize

We now discuss how to integrate the quantize and dequantize operators in our
algorithm. Since different quantization blocks have different scale factors,
after every INT8 WMMA operation, we need to dequantize the INT32 output into
FP32 and accumulate in FP32. By applying the same index notation as the
previous section, we have

$\displaystyle\boldsymbol{\mathbf{Y}}_{ij,pq}^{\text{INT32}}=\boldsymbol{\mathbf{X}}_{ik,p}\boldsymbol{\mathbf{W}}_{jk,q}^{\top},\leavevmode\nobreak\
\leavevmode\nobreak\
\boldsymbol{\mathbf{Y}}_{ij,pq}^{\text{FP32}}=\sum_{k=1}^{T_{C}}\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{X}}_{ik,p}}^{\text{FP16}}\boldsymbol{\mathbf{Y}}_{ij,pq}^{\text{INT32}}{\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{W}}_{jk,q}}^{\text{FP16}}},$

where
$\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{X}}},\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{W}}}$
is scale factor and both $\boldsymbol{\mathbf{Y}}$s are accumulators.

After the calculation of $\boldsymbol{\mathbf{Y}}_{ij,pq}^{\text{FP32}}$, we
quantize it to get a INT8 submatrix
$\boldsymbol{\mathbf{Y}}_{ij,pq}^{\text{INT8}}$ and a scale factor
$\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{Y}}_{ij,pq}}$.

We formalize our algorithm in Algorithm 1. In the algorithm, we have omitted
the details of the quantization block level and WMMA operation level for
simplicity. We highlight the overhead introduced by our method in red. We
further compare it with per-tensor quantization MM (Banner et al., 2018) and
per-token quantization MM (Wortsman et al., 2023) in Fig. 4a.

Our algorithm accurately quantizes channel-wise outliers while introducing
only a small amount of overhead for dequantize and quantize operations. We
calculate the overhead within the computation of a submatrix
$\boldsymbol{\mathbf{Y}}_{ij}$ and compare our method with basic INT8 MM and
SwitchBack. Results are reported in Table 3. The time complexity of MM is
$O(B_{N}*C*B_{D})$. while our method’s overhead time complexity is
$O(B_{N}*T_{C}*B_{D})+(BN+B_{D})C$. Since $\frac{C}{T_{C}}=B_{C}$ is typically
set to 32 or 64 and $B_{N},B_{D}$ is 128 or 256, the overhead is negligible.

Table 2: Meaning of Key Constants. $B_{N}/B_{C}/B_{D}$ | CUDA block size in MM
---|---
$T_{N}/T_{C}/T_{D}$ | Number of CUDA blocks along each axis
$B$ | Quantization block size
$R_{N}/R_{C}/R_{D}$ | Number of quantization blocks
| in a CUDA block along each axis
Table 3: Time complexity of different operations in MM.

| Method
---|---
Operation | Basic INT8 | SwitchBack | Ours
MM | $B_{N}B_{D}C$ | $B_{N}B_{D}C$ | $B_{N}B_{D}C$
16-bit Load/Store | $(B_{N}+B_{D})C+B_{N}B_{D}$ | $\frac{B_{N}B_{C}}{T_{D}}+\frac{B_{D}B_{C}}{T_{N}}+B_{N}B_{D}$ | -
8-bit Load/Store | - | $(B_{N}+B_{D})C$ | $(B_{N}+B_{D})C+B_{N}B_{D}$
Dequantize | - | $B_{N}B_{D}$ | $B_{N}B_{D}T_{C}$
Quantize | - | $\frac{B_{N}B_{C}}{T_{D}}+\frac{B_{D}B_{C}}{T_{N}}$ | $B_{N}B_{D}$

Algorithm 1 INT8 Linear Layer

0: INT8 Matrices $\boldsymbol{\mathbf{X}}\in\mathbb{R}^{N\times
C},\boldsymbol{\mathbf{W}}\in\mathbb{R}^{D\times C}$, FP16 scale matrices
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}}\in\mathbb{R}^{L_{N}\times
L_{C}},\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{W}}}\in\mathbb{R}^{L_{D}\times
L_{C}}$, CUDA Block size $B_{N}\times B_{C}\times B_{D}$

1: Define $T_{N}=\left\lceil\frac{N}{B_{N}}\right\rceil$,
$T_{C}=\left\lceil\frac{C}{B_{C}}\right\rceil$,
$T_{D}=\left\lceil\frac{D}{B_{D}}\right\rceil$

2: Define
$R_{N}=\left\lceil\frac{B_{N}}{B}\right\rceil,R_{C}=\left\lceil\frac{B_{C}}{B}\right\rceil
R_{D}=\left\lceil\frac{B_{D}}{B}\right\rceil$

3: for 1 $\leq$ i $\leq T_{N}$ do

4: for 1 $\leq$ j $\leq T_{D}$ do

5: Initialize accumulator $\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}$,
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT32}}$

6: for 1 $\leq$ k $\leq T_{C}$ do

7: Load INT8 Block $\boldsymbol{\mathbf{X}}_{ik}\in\mathbb{R}^{B_{N}\times
B_{C}}$ and scale factor
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}_{ik}}\in\mathbb{R}^{R_{N}\times
R_{C}}$

8: Load INT8 Block
$\boldsymbol{\mathbf{W}}_{jk}^{\top}\in\mathbb{R}^{B_{C}\times B_{D}}$ and
scale factor
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{W}}_{jk}^{\top}}\in\mathbb{R}^{R_{C}\times
R_{D}}$

9: On chip, compute INT8 Matmul:
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT32}}=\boldsymbol{\mathbf{X}}_{ik}\boldsymbol{\mathbf{W}}_{jk}^{\top}$

10: On chip, dequantize to FP32 and accumulate:
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}\leftarrow\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}+$
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}_{ik}}\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT32}}\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{W}}_{jk}^{\top}}$

11: end for

12: On Chip, quantize the output
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}$ to get
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT8}}\in\mathbb{R}^{B_{N}\times
B_{D}}$ and scale $\boldsymbol{\mathbf{S}}_{ij}\in\mathbb{R}^{R_{N}\times
R_{D}}$

13: Save $\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT8}}$ and
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{Y}}_{ij}}$ to global memory.

14: end for

15: end for

##  6 Non-Linear Operator

In this section, we mainly discuss how our per-block quantization method
should be applied to non-linear layers. By reducing the precision of the input
and output to INT8, we can achieve acceleration for these operators as well.

###  6.1 Non-Linear Operators are Memory-Bounded

We have observed that non-linear operators are memory-bounded, which means
that the speed of these operators is primarily limited by memory bandwidth,
rather than by computation. We validate this by manipulating the data format
(INT8, FP16, FP32) for global memory read/write operations in the GELU
operator, while internally converting them to FP32 for computation. Fig. 2b
illustrates that even computations are kept in FP32, simply reducing the
read/write precision can already obtain near-linear speedup. As our method
reduces the data flow precision from FP16 to INT8, we anticipate $\sim$2x
speedup for all nonlinear operators. In contrast, QCD cannot accelerate
nonlinear operators.

###  6.2 Triton Implementation

Based on the observations above, our main idea is to load/write in INT8 and
leave all calculations within the shared memory through kernel fusion.
Specifically, after loading the INT8 input into shared memory, we dequantize
it to FP32 and apply the non-linear operators, then quantize the FP32 output
back to INT8 format before writing the data into global memory.

We primarily focus on non-linear operators like GELU (Hendrycks & Gimpel,
2016), LayerNorm (Ba et al., 2016), Dropout (Fan et al., 2019), and Add (He et
al., 2016), and implement them with Triton (Tillet et al., 2019).

We define $f$ to be the element-wise operator,
$\boldsymbol{\mathbf{X}},\boldsymbol{\mathbf{Y}}\in\mathbb{R}^{N\times C}$ to
be the INT8 input and output,
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}},\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{Y}}}\in\mathbb{R}^{L_{N}\times
L_{C}}$ are scale factors, where $L_{N}=\frac{N}{B},L_{C}=\frac{C}{B}$ are
number of quantization blocks along each axis and $B$ is the quantization
block size. Similar to CUDA, we also do tiling to parallelize the computation.
For a single block (whose shape is defined as Triton Block Size) we denote
$\boldsymbol{\mathbf{X}}_{ij}^{\textbf{INT8}}$ to be the input tensor and
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}_{ij}}^{\textbf{FP16}}$ to be
the scale. The computation process can be represented as

$\displaystyle\small\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}=f(Q^{-1}(\boldsymbol{\mathbf{X}}_{ij}^{\textbf{INT8}},\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{X}}_{ij}}^{\textbf{FP16}}));\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT8}},\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{Y}}_{ij}}^{\textbf{FP16}}=Q(\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}),$

where $Q^{-1}$ and $Q$ is the dequantizer and quantizer,
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT8}}$ is the output tensor, and
$\boldsymbol{\mathbf{s}}_{\boldsymbol{\mathbf{Y}}_{ij}}^{\textbf{FP16}}$ is
the scale factor. This algorithm can be expressed as Algorithm 2, where we
omit the quantization block level for simplicity.

##  7 Experiments

(a)  Figure 5: Quantization error for different quantization methods. Per-
Block refers to our Jetfire quantization method.

(a)

(b)

Figure 6: Speed test of GELU and GEMM operator. (a) Triton kernel speedup with
different Triton block sizes.(b) GEMM CUDA kernel speed with different CUDA
block sizes. (a)  Figure 7: Speed comparision between our INT8 non-linear
operator and pytorch FP16 implementation. (a)  Figure 8:  Matrix
Multiplication Speed test for Different Methods in different settings
($B$=Batch Size, $N$=Sequence Length). Table 4: Results on machine
translation, deit pretraining, GPT2 pretraining, and GLUE fine-tuning result
based on the pretrained model. FP refers to floating-point, SwitchBack refers
to per-token quantization. ’–’ means the model does not converge.

|  |  | Baseline | Ours
---|---|---|---|---
Model |  $\\#$Params(M) | METRIC | FP | Switchback | Per-Tensor | Jetfire
Transformer-Base | 61 | BLEU | 26.49 | 26.46 | 26.04 | 26.49
Deit-tiny | 5 | Top1 Acc | 64.08 | 63.55 | – | 63.95
Deit-small | 22 | 73.43 | 72.80 | – | 73.31
Deit-base | 86 | 75.67 | 75.62 | – | 76.03
GPT2-base | 124 | Valid loss | 2.9074 | 3.0796 | 3.1638 | 2.8597
GPT2-medium | 350 | 2.6612 | 2.9141 | 3.1795 | 2.4195
GPT2-large | 774 | 2.5993 | 3.0512 | 2.9775 | 2.4696
GPT2-base | 124 | GLUE score | $78.50_{0.45}$ | $78.15_{0.15}$ | $76.33_{0.94}$ | $\boldsymbol{78.18_{0.34}}$
GPT2-medium | 350 | $82.00_{0.50}$ | $80.03_{0.15}$ | $79.19_{0.22}$ | $\boldsymbol{81.60_{0.26}}$
GPT2-large | 774 | $83.01_{0.24}$ | $78.74_{0.24}$ | $75.88_{0.35}$ | $\boldsymbol{82.94_{0.70}}$

###  7.1 Settings

We evaluate our INT8 training algorithm Jetfire on a wide variety of tasks
including machine translation, image classification, and generative model
pretraining. We adopt default architectures, optimizers, and schedulers for
all the evaluated models. We adopt the default hyperparameter except for
generative model pretraining.

We quantize all of the linear layers in the MLP and attention module and non-
linear layers (including GELU, LayerNorm and Dropout) to INT8, and leave
multi-head attention in FP16 by employing FlashAttention (Dao et al., 2022).
We compare our method with floating point training baseline (denoted as FP),
per-tensor quantization, and SwitchBack (Wortsman et al., 2023)). We do not
compare with FP8 training algorithms as they require specialized Hopper
architecture GPU to run, making them less accessible. We emphasize that only
our method adopts an INT8 data flow and quantizes non-linear layers.

We implement our linear operators with CUDA and implement non-linear operators
with Triton. CUDA block size is set to $128\times 32\times 128$ and Triton
block size is set to $64\times 64$. The quantization block size is set to
$B=32$.

###  7.2 Converged Model Accuracy

#### Machine Translation

We validate our Jetfire’s effectiveness on the translation task. We train a
Transformer-base model on WMT 14 En-De dataset (Bojar et al., 2014) based on
Nvidia’s recipe 111https://github.com/NVIDIA/DeepLearningExamples
/tree/master/PyTorch/Translation/Transformer. In Table 4 we report the BLEU
(Papineni et al., 2002) score result. Our method has no degradation compared
with the FP baseline, while the SwitchBack baseline has $0.03\%$ BLEU score
degradation, and the per-tensor quantization baseline has $0.4\%$ degradation.

#### Image Classification

We do pretraining for Deit-Tiny, Deit-Small, and Deit-Base (Touvron et al.,
2021) model on ImageNet1K (Deng et al., 2009) for 90 epochs based on facebook
research’s recipe 222https://github.com/facebookresearch/deit. Results are
listed on Table 4. In all experiments, Our method has less than $0.1\%$
accuracy degradation compared with the floating-point baseline, and for Deit-
base, our method shows $0.4\%$ improvement. For Deit-tiny and Deit-small
models, Switchback has over $0.5\%$ accuracy degradation, and per-tensor
quantization does not converge. This indicates that our method can accurately
quantize channel-wise outliers.

#### Generative Model Pretraining

We evaluate our method by training three GPT2 (Radford et al., 2019) models
with different sizes: GPT2-base for 300k steps, GPT2-medium for 200k steps,
and GPT2-large for 120k steps on the OpenWebText (Peterson et al., 2019)
dataset based on NanoGPT333https://github.com/karpathy/nanoGPT.
(Hyperparameters: Learning Rate = $1.5\times 10^{-4}$, Weight Decay =
$10^{-1}$). We report the validation loss and the fine-tuning average accuracy
on the GLUE (Wang et al., 2018a) dataset over 3 seeds. The results are shown
in Table 4.

We found that SwitchBack resulted in 0.1 valid loss degradation on GPT-base
and led to 0.3-0.4 valid degradation on GPT-medium and GPT-large. Our method
achieves even lower valid loss compared to the FP baseline, which may be
attributed to the regularization effect of quantization.

For fine-tuning, our method shows less than 0.3% degradation compared to
baseline, while SwitchBack has a degradation of 0.3% on GPT2-base, 1.8% on
GPT2-medium, and 4.3% on GPT2-large. This indicates that for LLM pretraining,
the influence of channel-wise outliers is significant, and our quantization
method effectively preserves accuracy.

###  7.3 Ablation Study

#### Quantization Error

We study the quantization error of different quantization methods on four
different sizes of GPT2 models to show our method’s effectiveness. We focus on
the activation of the final layer and calculate the mean squared error (MSE)
and the mean error after quantization. The results are shown in Fig. 5. For
all models, per-channel quantization consistently resulted in smaller
quantization errors compared to per-token quantization. Jetfire (ours)
achieves lower quantization error than per-token quantization while performing
on par with per-channel quantization.

#### CUDA kernel and Triton kernel block size

We have found that the selection of the block size for Triton and CUDA kernels
is crucial. A large block size leads to a decrease in parallelism, while a
small block size results in low utilization of bandwidth and computational
resources. Both cases can result in low kernel speed. In Fig. 6a 6b, we test
the kernel’s speed under different block sizes and find that optimal
efficiency is achieved when we set Triton block size$\leavevmode\nobreak\
=64\times 64$ and CUDA block size$\leavevmode\nobreak\ =128\times 32\times
128$.

###  7.4 Operator and End-to-End experiments

#### Linear layer speedup

We test the speedup of our custom linear layer. We analyzed the time
consumption of each component in forward and backward passes and compared the
speed of our implementation with FP16 and SwitchBack linear layers. The
results are shown in Fig. 8. Our MM operator provides about 60% speed
improvement compared to FP16. Other overhead components like quantizing and
reshaping have a relatively minor impact. Our method achieves 40% overall
speedup (forward + backward), which is comparable to the acceleration result
of SwitchBack, where SwitchBack leaves the calculation of weight gradient in
FP. The speedup becomes larger when the matrix size increases since the
overhead proportion decreases.

#### Non-linear operator speedup

We also test the speedups offered by our custom non-linear layers, which is
the first quantized training work to achieve acceleration for these non-linear
operators.

Our INT8 GELU operator achieves 80% speedup in both forward and backward
passes compared to PyTorch’s FP16 operators. Our INT8 LayerNorm operator
achieves 40% speed up in its forward pass and up to 90% speedup in its
backward pass when $\text{hidden size}=8192$ but does not accelerate when the
hidden size is small. These results indicate that the global memory access is
indeed the bottleneck for these non-linear operators, and our INT8 data flow
can effectively solve the bottleneck, resulting in near-ideal speedup.

#### End-to-end speedup

We experimented with GPT2 models and varied the network hidden size to show
the end-to-end speedup for our Jetfire method over PyTorch’s FP16 training. We
integrated all linear and non-linear operators and reported the speedup of a
transformer layer. We compared the forward, backward, and overall runtime
speedup with the SwitchBack layer. Results in Table 5 showed that our method
achieved comparable or improved acceleration compared to SwitchBack. This is
primarily because our linear operators in backpropagation are faster than
SwitchBack, and we can accelerate all of the non-linear operators in both
forward and backward propagation.

Table 5: Acceleration ratios for End-to-end comparison (SB refers to
SwitchBack basic version)

| Forward | Backward | Overall
---|---|---|---
Hidden Size | SB | Ours | SB | Ours | SB | Ours
4096 | 1.50 | 1.32 | 1.18 | 1.46 | 1.27 | 1.42
2048 | 1.53 | 1.29 | 1.24 | 1.41 | 1.32 | 1.37
1024 | 0.94 | 0.97 | 1.14 | 1.11 | 1.07 | 1.07

#### End-to-End Memory Reduction

We experimented with GPT2 models and varied the network depth and batch size
to show the memory reduction of our method. We report the reduction ratio of
activation memory. The results are shown in Table 6. Our method achieved up to
1.49x activation memory reduction, which is better than SwitchBack since we
reduced the memory footprint of non-linear operators.

Table 6: Activation memory reduction ratios for End-to-end comparison (SB
refers to SwitchBack Memeory Efficient version)

| BS=1 | BS=2 | BS=4
---|---|---|---
Layer Num | SB | Ours | SB | Ours | SB | Ours
12 | 1.19 | 1.33 | 1.14 | 1.31 | 1.11 | 1.29
24 | 1.24 | 1.49 | 1.18 | 1.47 | 1.14 | 1.45

##  8 Conclusion

In this work, we propose Jetfire, an INT8 pretraining method for transformer
models. For the first time, we propose to use INT8 data flow in pretraining to
reduce computation, memory access, memory usage, and communication at the same
time. We also propose to use per-block quantization for all of the
activations, weights, and gradients for both linear and non-linear layers to
preserve accuracy. Extensive experiments demonstrate that our proposed method
performs on par with FP baselines, and can effectively accelerate the training
speed and reduce the memory footprint.

## Acknowledgements

The authors would like to thank Bingrui Li, Ziteng Wang, Jiayi Zhong, Cheng Lu
for the helpful discussions.

##  9 Broader Impact

Our INT8 fully quantized training (FQT) method significantly improves the
efficiency of deep learning by reducing computations and memory usage of
training transformers. This contributes substantially to energy conservation
and emission reduction, and aligns with the objective of global
sustainability. Besides, our method promotes the democratization of artificial
intelligence (AI) by making transformer training more accessible to cheap and
low-resource platforms. Nevertheless, this method could also be misused to
expedite the training of ”evil models” designed to generate harmful content.

## References

  * nvi (2022) Nvidia h100 tensor core gpu architecture.  https://resources.nvidia.com/en-us-tensor-core, 2022.
  * Anil et al. (2023) Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Ba et al. (2016) Ba, J. L., Kiros, J. R., and Hinton, G. E.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Bai et al. (2020) Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., and King, I.  Binarybert: Pushing the limit of bert quantization.  _arXiv preprint arXiv:2012.15701_ , 2020.
  * Banner et al. (2018) Banner, R., Hubara, I., Hoffer, E., and Soudry, D.  Scalable methods for 8-bit training of neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 5145–5153, 2018.
  * Bojar et al. (2014) Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., et al.  Findings of the 2014 workshop on statistical machine translation.  In _Proceedings of the ninth workshop on statistical machine translation_ , pp. 12–58, 2014.
  * Chee et al. (2023) Chee, J., Cai, Y., Kuleshov, V., and De Sa, C.  Quip: 2-bit quantization of large language models with guarantees.  _arXiv preprint arXiv:2307.13304_ , 2023.
  * Chen et al. (2020) Chen, J., Gai, Y., Yao, Z., Mahoney, M. W., and Gonzalez, J. E.  A statistical framework for low-bitwidth training of deep neural networks.  _Advances in neural information processing systems_ , 33:883–894, 2020.
  * Chmiel et al. (2021) Chmiel, B., Banner, R., Hoffer, E., Yaacov, H. B., and Soudry, D.  Logarithmic unbiased quantization: Practical 4-bit training in deep learning.  _arXiv preprint arXiv:2112.10769_ , 2021.
  * Dao et al. (2022) Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ , 35:16344–16359, 2022.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dettmers et al. (2022) Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L.  Llm. int8 (): 8-bit matrix multiplication for transformers at scale.  _arXiv preprint arXiv:2208.07339_ , 2022.
  * Dong et al. (2019a) Dong, Z., Yao, Z., Cai, Y., Arfeen, D., Gholami, A., Mahoney, M. W., and Keutzer, K.  Hawq-v2: Hessian aware trace-weighted quantization of neural networks.  _arXiv preprint arXiv:1911.03852_ , 2019a.
  * Dong et al. (2019b) Dong, Z., Yao, Z., Gholami, A., Mahoney, M., and Keutzer, K.  Hawq: Hessian aware quantization of neural networks with mixed-precision.  _ICCV_ , 2019b.
  * Esser et al. (2019) Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S.  Learned step size quantization.  _arXiv preprint arXiv:1902.08153_ , 2019.
  * Fan et al. (2019) Fan, A., Grave, E., and Joulin, A.  Reducing transformer depth on demand with structured dropout.  _arXiv preprint arXiv:1909.11556_ , 2019.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hendrycks & Gimpel (2016) Hendrycks, D. and Gimpel, K.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Jacob et al. (2018) Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2704–2713, 2018.
  * Kim et al. (2021) Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Keutzer, K.  I-bert: Integer-only bert quantization.  In _International conference on machine learning_ , pp. 5506–5518. PMLR, 2021.
  * Kim et al. (2023) Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K.  Squeezellm: Dense-and-sparse quantization.  _arXiv preprint arXiv:2306.07629_ , 2023.
  * Lin et al. (2023) Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S.  Awq: Activation-aware weight quantization for llm compression and acceleration.  _arXiv preprint arXiv:2306.00978_ , 2023.
  * Liu et al. (2021) Liu, Z., Wang, Y., Han, K., Zhang, W., Ma, S., and Gao, W.  Post-training quantization for vision transformer.  _Advances in Neural Information Processing Systems_ , 34:28092–28103, 2021.
  * Markidis et al. (2018) Markidis, S., Der Chien, S. W., Laure, E., Peng, I. B., and Vetter, J. S.  Nvidia tensor core programmability, performance & precision.  In _2018 IEEE international parallel and distributed processing symposium workshops (IPDPSW)_ , pp. 522–531. IEEE, 2018.
  * Micikevicius et al. (2018) Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al.  Mixed precision training.  In _International Conference on Learning Representations_ , 2018.
  * Nvidia (2022) Nvidia.  Nvidia transformer engine.  https://docs.nvidia.com/deeplearning/transformer-engine/index.html, 2022.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Papineni et al. (2002) Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pp. 311–318, 2002.
  * Peng et al. (2023) Peng, H., Wu, K., Wei, Y., Zhao, G., Yang, Y., Liu, Z., Xiong, Y., Yang, Z., Ni, B., Hu, J., et al.  Fp8-lm: Training fp8 large language models.  _arXiv preprint arXiv:2310.18313_ , 2023.
  * Perez et al. (2023) Perez, S. P., Zhang, Y., Briggs, J., Blake, C., Levy-Kramer, J., Balanca, P., Luschi, C., Barlow, S., and Fitzgibbon, A. W.  Training and inference of large language models using 8-bit floating point.  _arXiv preprint arXiv:2309.17224_ , 2023.
  * Peterson et al. (2019) Peterson, J., Meylan, S., and Bourgin, D.  Open clone of openai’s unreleased webtext dataset scraper, 2019.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Shen et al. (2019) Shen, S., Dong, Z., Ye, J., Ma, L., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K.  Q-bert: Hessian based ultra low precision quantization of bert.  _arXiv preprint arXiv:1909.05840_ , 2019.
  * Sun et al. (2019) Sun, X., Choi, J., Chen, C.-Y., Wang, N., Venkataramani, S., Srinivasan, V. V., Cui, X., Zhang, W., and Gopalakrishnan, K.  Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 4901–4910, 2019.
  * Sun et al. (2020) Sun, X., Wang, N., Chen, C.-Y., Ni, J., Agrawal, A., Cui, X., Venkataramani, S., El Maghraoui, K., Srinivasan, V. V., and Gopalakrishnan, K.  Ultra-low precision 4-bit training of deep neural networks.  In _Advances in Neural Information Processing Systems_ , volume 33, 2020.
  * Tang et al. (2022) Tang, H., Zhang, X., Liu, K., Zhu, J., and Kang, Z.  Mkq-bert: Quantized bert with 4-bits weights and activations.  _arXiv preprint arXiv:2203.13483_ , 2022.
  * Tillet et al. (2019) Tillet, P., Kung, H.-T., and Cox, D.  Triton: an intermediate language and compiler for tiled neural network computations.  In _Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages_ , pp. 10–19, 2019.
  * Touvron et al. (2021) Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pp. 10347–10357. PMLR, 2021.
  * Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wang et al. (2018a) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_ , 2018a.
  * Wang et al. (2018b) Wang, N., Choi, J., Brand, D., Chen, C.-Y., and Gopalakrishnan, K.  Training deep neural networks with 8-bit floating point numbers.  In _Advances in Neural Information Processing Systems_ , pp. 7675–7684, 2018b.
  * Wortsman et al. (2023) Wortsman, M., Dettmers, T., Zettlemoyer, L., Morcos, A., Farhadi, A., and Schmidt, L.  Stable and low-precision training for large-scale vision-language models.  _arXiv preprint arXiv:2304.13013_ , 2023.
  * Xi et al. (2023) Xi, H., Li, C., Chen, J., and Zhu, J.  Training transformers with 4-bit integers.  _arXiv preprint arXiv:2306.11987_ , 2023.
  * Xiao et al. (2023) Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In _International Conference on Machine Learning_ , pp. 38087–38099. PMLR, 2023.
  * Zhang et al. (2020) Zhang, W., Hou, L., Yin, Y., Shang, L., Chen, X., Jiang, X., and Liu, Q.  Ternarybert: Distillation-aware ultra-low bit bert.  _arXiv preprint arXiv:2009.12812_ , 2020.
  * Zhao et al. (2021) Zhao, K., Huang, S., Pan, P., Li, Y., Zhang, Y., Gu, Z., and Xu, Y.  Distribution adaptive int8 quantization for training cnns.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 3483–3491, 2021.
  * Zhou et al. (2021) Zhou, Q., Guo, S., Qu, Z., Guo, J., Xu, Z., Zhang, J., Guo, T., Luo, B., and Zhou, J.  Octo: INT8 training with loss-aware compensation and backward quantization for tiny on-device learning.  In _2021 USENIX Annual Technical Conference (USENIX ATC 21)_ , pp. 177–191, 2021.
  * Zhu et al. (2020) Zhu, F., Gong, R., Yu, F., Liu, X., Wang, Y., Li, Z., Yang, X., and Yan, J.  Towards unified int8 training for convolutional neural network.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1969–1979, 2020.

##  Appendix A Triton Implementation of Non-Linear Operators

For the GELU function, its forward and backward operator is:

$\mathrm{GELU}(x)=x\cdot\Phi(x),\leavevmode\nobreak\ \leavevmode\nobreak\
\frac{\mathrm{d}\mathrm{GELU}(x)}{\mathrm{d}x}=\frac{x}{\sqrt{2\pi}}e^{-\frac{x^{2}}{2}}+\Phi(x).$

For Dropout, its forward and backward operator is:

$\mathrm{Drop}(x)=\frac{1}{1-p}x\circ m,\leavevmode\nobreak\
\leavevmode\nobreak\
\frac{\mathrm{d}\mathrm{Drop}(x)}{\mathrm{d}x}=\frac{1}{1-p}m.$

For Add, when we calculate the residual connection $y=x+f(x)$, we also need to
perform $\mathrm{d}x=\mathrm{d}f(y)+\mathrm{d}y$ in the backward process. This
addition operator can be represented as:

$\mathrm{Add}(x_{1},x_{2})=x_{1}+x_{2}.$

Algorithm 2 INT8 Non-Linear Operator

0: INT8 Matrix $\boldsymbol{\mathbf{X}}\in\mathbb{R}^{N\times C}$, FP16 scale
matrix
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}}\in\mathbb{R}^{L_{N}\times
L_{C}}$, element-wise function $f$

1: Define $T_{N}=\left\lceil\frac{N}{B_{N}}\right\rceil$,
$T_{C}=\left\lceil\frac{C}{B_{C}}\right\rceil$

2: Define
$R_{N}=\left\lceil\frac{B_{N}}{B}\right\rceil,R_{C}=\left\lceil\frac{B_{C}}{B}\right\rceil$

3: for 1 $\leq$ i $\leq T_{N}$ do

4: for 1 $\leq$ j $\leq T_{C}$ do

5: Load INT8 block $\boldsymbol{\mathbf{X}}_{ij}\in\mathbb{R}^{B_{N}\times
B_{C}}$,
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}_{ij}}\in\mathbb{R}^{R_{N}\times
R_{C}}$

6: Dequantize $\boldsymbol{\mathbf{X}}_{ij}$ and
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{X}}_{ij}}$ to get
$\boldsymbol{\mathbf{X}}_{ij}^{\textbf{FP32}}$

7: Operate:
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}=f(\boldsymbol{\mathbf{X}}_{ij}^{\textbf{FP32}})$

8: Quantize $\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{FP32}}$ to get
$\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT32}}\in\mathbb{R}^{B_{N}\times
B_{C}}$ and scale factor
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{Y}}_{ij}}\in\mathbb{R}^{R_{N}\times
R_{C}}$

9: Save $\boldsymbol{\mathbf{Y}}_{ij}^{\textbf{INT32}}$ and
$\boldsymbol{\mathbf{S}}_{\boldsymbol{\mathbf{Y}}_{ij}}$ to global memory.

10: end for

11: end for

Differing from non-linear operators above, LayerNorm involves interactions
between elements. Therefore, performing calculations separately for each
$B_{N}\times B_{C}$ block is not feasible. In order to solve the problem, we
observed that both pre-norm and post-norm models encountered the ADD operator
before LayerNorm.

We make the following modifications to our ADD operator: We will calculate the
mean and sum of squares for each row of the block $(B_{N},B_{C})$ and store
these values. We will then get the mean matrix and sum of squares matrix of
size $N\times\frac{C}{B_{C}}$, which reduces the amount of data we need to
load and store by $\frac{1}{B_{C}}$. Before the LayerNorm operator, we use
these values to compute the mean and variance for each row, which size is
$N\times 1$. This allows the LayerNorm to directly access these values. The
implementation of the remaining part of LayerNorm is similar to that of GELU.

##  Appendix B Detailed Results of GLUE Fine-Tuning Test

Table 7: Detailed Results of GLUE fine-tuning test based on the pretrained
model. FP refers to floating-point, SwitchBack refers to per-token
quantization. ’–’ means the model does not converge.

|  |  | Baseline | Ours
---|---|---|---|---
Model |  $\\#$Params(M) | METRIC | FP | Switchback | Per-Tensor | Jetfire
GPT2-BASE | 124 | COLA | $43.97_{3.32}$ | $\boldsymbol{44.68_{1.06}}$ | $41.43_{2.23}$ | $41.98_{2.51}$
STSB | $83.20_{0.95}$ | $81.08_{1.39}$ | $75.63_{5.35}$ | $\boldsymbol{81.66_{0.45}}$
RTE | $64.02_{1.37}$ | $64.74_{0.83}$ | $64.14_{0.75}$ | $\boldsymbol{65.22_{0.91}}$
MRPC | $85.48_{0.78}$ | $84.30_{0.49}$ | $83.68_{0.61}$ | $\boldsymbol{85.01_{0.49}}$
SST2 | $91.40_{0.30}$ | $\boldsymbol{92.09_{0.50}}$ | $90.79_{0.26}$ | $91.93_{0.35}$
QNLI | $87.97_{0.35}$ | $\boldsymbol{87.58_{0.19}}$ | $86.64_{0.25}$ | $87.57_{0.19}$
QQP | $90.07_{0.001}$ | $89.94_{0.07}$ | $88.84_{0.09}$ | $\boldsymbol{90.14_{0.03}}$
MNLI | $81.84_{0.22}$ | $80.76_{0.24}$ | $79.51_{0.36}$ | $\boldsymbol{81.92_{0.18}}$
GPT2-MEDIUM | 350 | COLA | $53.55_{2.89}$ | $50.97_{0.58}$ | $48.82_{1.43}$ | $\boldsymbol{53.38_{1.08}}$
STSB | $86.17_{0.78}$ | $83.21_{1.61}$ | $81.66_{0.50}$ | $\boldsymbol{84.98_{0.32}}$
RTE | $68.23_{1.44}$ | $63.78_{0.91}$ | $67.27_{0.21}$ | $\boldsymbol{68.47_{0.21}}$
MRPC | $88.16_{0.91}$ | $84.79_{0.68}$ | $84.98_{0.15}$ | $\boldsymbol{86.74_{1.36}}$
SST2 | $93.73_{0.29}$ | $\boldsymbol{94.00_{0.18}}$ | $91.78_{0.13}$ | $93.46_{0.23}$
QNLI | $90.49_{0.14}$ | $89.61_{0.20}$ | $88.05_{0.16}$ | $\boldsymbol{90.15_{0.17}}$
QQP | $90.98_{0.15}$ | $90.54_{0.15}$ | $90.02_{0.08}$ | $\boldsymbol{90.93_{0.08}}$
MNLI | $84.69_{0.37}$ | $83.37_{0.36}$ | $80.93_{0.31}$ | $\boldsymbol{84.65_{0.11}}$
GPT2-LARGE | 774 | COLA | $52.22_{0.15}$ | $48.70_{1.20}$ | $35.85_{1.23}$ | $\boldsymbol{54.06_{1.62}}$
STSB | $84.99_{1.37}$ | $79.49_{0.15}$ | $80.00_{0.15}$ | $\boldsymbol{84.99_{1.37}}$
RTE | $76.65_{1.67}$ | $65.10_{1.10}$ | $64.26_{2.60}$ | $\boldsymbol{74.73_{2.53}}$
MRPC | $86.70_{0.92}$ | $83.75_{1.00}$ | $83.09_{0.45}$ | $\boldsymbol{87.06_{1.21}}$
SST2 | $94.84_{0.50}$ | $92.47_{0.07}$ | $90.86_{0.24}$ | $\boldsymbol{94.65_{0.18}}$
QNLI | $91.35_{0.23}$ | $88.42_{0.12}$ | $85.25_{0.36}$ | $\boldsymbol{91.19_{0.52}}$
QQP | $91.40_{0.08}$ | $89.92_{0.19}$ | $89.21_{0.03}$ | $\boldsymbol{91.19_{0.02}}$
MNLI | $85.88_{0.14}$ | $82.05_{0.20}$ | $78.53_{0.15}$ | $\boldsymbol{85.61_{0.30}}$


Contribution to JETP special issue in honor of E. I. Rashba’s 95th birthday

# Spin and valley effects on the quantum phase transition in two dimensions

A. A. Shashkina and S. V. Kravchenkob aInstitute of Solid State Physics,
Chernogolovka, Moscow District, 142432, Russia
bPhysics Department, Northeastern University, Boston, Massachusetts 02115, USA

###### Abstract

Using several independent methods, we find that the metal-insulator transition
occurs in the strongly-interacting two-valley two-dimensional electron system
in ultra-high mobility SiGe/Si/SiGe quantum wells in zero magnetic field. The
transition survives in this system in parallel magnetic fields strong enough
to completely polarize the electrons’ spins, thus making the electron system
“spinless”. In both cases, the resistivity on the metallic side near the
transition increases with decreasing temperature, reaches a maximum at a
temperature $T_{\text{max}}$, and then decreases. The decrease reaches more
than an order of magnitude in zero magnetic field. The value of
$T_{\text{max}}$ in zero magnetic field is found to be close to the
renormalized Fermi temperature. However, rather than increasing along with the
Fermi temperature, the value $T_{\text{max}}$ decreases appreciably for
spinless electrons in spin-polarizing magnetic fields. The observed behavior
of $T_{\text{max}}$ cannot be described by existing theories. The results
indicate the spin-related origin of the effect. At the same time, the low-
temperature resistivity drop in both spin-unpolarized and spinless electron
systems is described quantitatively by the dynamical mean-field theory.

###### pacs:

71.10.Hf, 71.27.+a, 71.10.Ay

##  I Introduction

Spin and valley degrees of freedom in two-dimensional (2D) electron systems
have recently attracted much attention due to rapidly developing fields of
spintronics and valleytronics (see, e.g., Refs. Behnia (2012); Zhu et al.
(2012); Schaibley et al. (2016); Zhu et al. (2017)). The existence of the
zero-magnetic-field metallic state and the metal-insulator transition (MIT) in
strongly interacting 2D electron systems is intimately related to the
existence of these degrees of freedom Lee and Ramakrishnan (1985); Punnoose
and Finkel’stein (2001, 2005); Fleury and Waintal (2008). The MIT in two
dimensions was theoretically envisioned based on the renormalization group
analysis (see Ref. Lee and Ramakrishnan (1985) for a review). It was first
experimentally observed in a strongly-interacting 2D electron system in
silicon metal-oxide-semiconductor field-effect transistors (MOSFETs)
Zavaritskaya and Zavaritskaya (1987); Kravchenko et al. (1994, 1995); Popović
et al. (1997) and subsequently reported in a wide variety of 2D electron and
hole systems: $p$-type SiGe heterostructures, $p$\- and $n$-type GaAs/AlGaAs
heterostructures, AlAs heterostructures, ZnO-related heterostructures, etc.
(for recent reviews, see Refs. Shashkin and Kravchenko (2019, 2021)). Now it
is widely accepted that the driving force behind the MIT is the strong
correlations between carriers. Here we study the metal-insulator transition
and non-monotonic temperature-dependent resistivity on the metallic side near
the MIT in the strongly-interacting two-valley 2D electron system in ultra-
high mobility SiGe/Si/SiGe quantum wells in zero and spin-polarizing magnetic
fields.

Measurements reported here were performed on ultra-high mobility SiGe/Si/SiGe
quantum wells similar to those described in Refs. Melnikov et al. (2015);
Melnikov et al. (2017a). The peak electron mobility, $\mu$, in these samples
reaches 240 m2/Vs. It is important to note that judging by the appreciably
higher quantum electron mobility ($\sim 10$ m2/Vs) in the SiGe/Si/SiGe quantum
wells compared to that in Si MOSFETs, the residual disorder related to both
short- and long-range random potential is drastically smaller in the samples
used here. The approximately 15 nm wide silicon (001) quantum well is
sandwiched between Si0.8Ge0.2 potential barriers. The samples were patterned
in Hall-bar shapes with the distance between the potential probes of 150
$\mu$m and width of 50 $\mu$m using standard photo-lithography. Measurements
were carried out in an Oxford TLM-400 dilution refrigerator. Data on the
metallic side of the transition were taken by a standard four-terminal lock-in
technique in a frequency range 1–10 Hz in the linear response regime. On the
insulating side of the transition, the resistance was measured with dc using a
high input impedance electrometer. Since in this regime, the current-voltage
($I$-$V$) curves are strongly nonlinear, the resistivity was determined from
${\rm d}V/{\rm d}I$ in the linear interval of $I$-$V$ curves, as $I\rightarrow
0$.

##  II Quantum phase transition in zero magnetic field

An important metric defining the MIT is the magnitude of the resistance drop
in the metallic regime. Until recently, the strongest drop of the resistance
with decreasing temperature (up to a factor of 7) was reported in Si MOSFETs
Kravchenko et al. (1994, 1995). In contrast, in spite of much lower level of
disorder in $p$\- and $n$-GaAs-based structures, the low-temperature drop of
the resistance in those systems has never exceeded a factor of about three
Hanein et al. (1998). This discrepancy has been attributed primarily to the
fact that electrons in Si MOSFETs have two almost degenerate valleys, which
further enhances the correlation effects Punnoose and Finkel’stein (2001,
2005). The importance of these strong interactions in 2D systems has been
confirmed recently in the observation of the formation of a quantum electron
solid in Si MOSFETs Brussarski et al. (2018). It has been found that the
effective electron mass in Si MOSFET 2D electron systems strongly increases as
the electron density is decreased, with a tendency to diverge at a density
that lies close to, but is consistently below, the critical density for the
MIT Shashkin et al. (2002); Mokashi et al. (2012).

The resistivity, $\rho$, as a function of temperature, $T$, is shown in Fig.
1(a) for different electron densities, $n_{\text{s}}$, on both sides of the
metal-insulator transition. While at the highest temperature the difference
between the resistivities measured at the lowest and highest densities differ
by less than two orders of magnitude, at the lowest temperature this
difference exceeds six orders of magnitude. We identify the transition point
at $n_{\text{c}}(0)=0.88\pm 0.02\times 10^{10}$ cm-2, based on the ${\rm
d}\rho/{\rm d}T$ sign-change criterion taking account of the tilted separatrix
Punnoose and Finkel’stein (2005). The low-temperature drop of the resistivity
in the ultra-clean 2D electron system in SiGe/Si/SiGe quantum wells reaches a
factor of about 12. This is the highest value reported in any 2D electron
system Melnikov et al. (2019).

Figure 1: (a) Temperature dependences of the resistivity in a SiGe/Si/SiGe
quantum well in zero magnetic field. The electron densities in units of
$10^{10}$ cm-2 (top to bottom) are: 0.37, 0.43, 0.49, 0.55, 0.61, 0.68, 0.74,
0.80, 0.85, 0.88, 0.92, 0.98, 1.17, 1.35, 1.54, 1.72, 1.90, 2.09, 2.27, 2.64,
3.01, 3.38, and 3.75. The solid line corresponds to the separatrix. The inset
shows a close-up view of $\rho(T)$ at $n_{\text{s}}=3.01\times 10^{10}$ cm-2
displaying a drop of the resistivity by a factor of 12. (b) Activation energy
and the square root of the threshold voltage as a function of the electron
density in zero magnetic field. Vertical error bars correspond to the
experimental uncertainty. The solid lines are linear fits yielding
$n_{\text{c}}(0)=0.87\pm 0.02\times 10^{10}$ cm-2. Top inset: Current-voltage
characteristic measured at a temperature of 30 mK. Bottom inset: Arrhenius
plots of the resistivity in the insulating phase for two electron densities.
The densities in both insets are indicated in cm-2.

The location of the MIT point can also be determined by studying the
insulating side of the transition, where the resistance has an activated form,
as shown in the bottom inset of Fig. 1(b); note that the activation energy,
$E_{\text{a}}$, can be determined provided $E_{\text{a}}>k_{\text{B}}T$.
Figure 1(b) shows the activation energy in temperature units,
$E_{\text{a}}/k_{\text{B}}$, as a function of the electron density. Near the
critical point, this dependence corresponds to the constant thermodynamic
density of states and should be linear; the relative accuracy of determination
of $E_{\text{a}}$ increases with increasing activation energy, and the linear
fit should be drawn through all data points. The activation energy
extrapolates to zero at $n_{\text{c}}(0)=0.87\pm 0.02\times 10^{10}$ cm-2
which coincides, within the experimental uncertainty, with the value of
$n_{\text{c}}$ determined from the temperature derivative criterion.
Furthermore, in the insulating state, a typical low-temperature $I$-$V$ curve
shows a step-like function: the voltage rises abruptly at low currents and
then almost saturates, as seen in the top inset of Fig. 1(b). The magnitude of
the step is $2\,V_{\text{c}}$, where $V_{\text{c}}$ is the threshold voltage.
The threshold behavior of the $I$-$V$ curves has been explained Polyakov and
Shklovskii (1993); Shashkin et al. (1994) within the concept of the breakdown
of the insulating phase that occurs when the localized electrons at the Fermi
level gain enough energy to reach the mobility edge in an electric field,
$V_{\text{c}}/d$, over a distance of the localization length, $L$ (here $d$ is
the distance between the potential probes). The values
$E_{\text{a}}/k_{\text{B}}$ and $V_{\text{c}}$ are related via the
localization length, which is temperature-independent and diverges near the
transition as $L(E_{\text{F}})\propto(E_{\text{c}}-E_{\text{F}})^{-s}$ with
exponent $s$ close to unity Shashkin et al. (1994) (here $E_{\text{c}}$ is the
mobility edge and $E_{\text{F}}$ is the Fermi level). This corresponds to a
linear dependence of the square root of $V_{\text{c}}$ on $n_{\text{s}}$ near
the MIT, as seen in Fig. 1(b). The dependence extrapolates to zero at the same
electron density as $E_{\text{a}}/k_{\text{B}}$. A similar analysis has been
previously performed Shashkin et al. (2001) in a 2D electron system in Si
MOSFETs and has yielded similar results, thus adding confidence that the MIT
in 2D is a genuine quantum phase transition.

The critical electron density for the MIT is almost an order of magnitude
smaller than that in the least-disordered Si MOSFETs, where
$n_{\text{c}}(0)\approx 8\times 10^{10}$ cm-2. Such a difference can indeed be
expected for an interaction-driven MIT. The interaction parameter,
$r_{\text{s}}$, is defined as the ratio of the Coulomb and Fermi energies,
$r_{\text{s}}=g_{\text{v}}/(\pi n_{\text{s}})^{1/2}a_{\text{B}}$, where
$g_{\text{v}}=2$ is the valley degeneracy and $a_{\text{B}}$ is the effective
Bohr radius in the semiconductor. We compare the value of the interaction
parameter at the critical density $n_{\text{c}}$ in SiGe/Si/SiGe quantum wells
with that in Si MOSFETs (where $r_{\text{s}}\approx 20$). The two systems
differ by the level of the disorder, the thickness of the 2D layer, and the
dielectric constant (7.7 in Si MOSFETs and 12.6 in SiGe/Si/SiGe quantum
wells). Due to the higher dielectric constant, the interaction parameter at
the same electron density is smaller in SiGe/Si/SiGe quantum wells by
approximately 1.6. In addition, the effective $r_{\text{s}}$ value is reduced
further due to the much greater thickness of the 2D layer in the SiGe/Si/SiGe
quantum wells, which results in a smaller form-factor Ando et al. (1982).
Assuming that the effective mass in the SiGe barrier is $\approx
0.5\,m_{\text{e}}$ and estimating the barrier height at $\approx 25$ meV, we
evaluate the penetration of the wave function into the barrier and obtain the
effective thickness of the 2D layer $\approx 200$ Å compared to $\approx 50$ Å
in Si MOSFETs. This yields the additional suppression of $r_{\text{s}}$ in the
SiGe/Si/SiGe quantum wells compared to Si MOSFETs by a factor of about 1.3.
Thus, the electron densities $n_{\text{c}}$ correspond to $r_{\text{s}}\approx
20$ in both Si MOSFETs and SiGe/Si/SiGe quantum wells, which is consistent
with the results of Ref. Shashkin et al. (2007).

##  III Quantum phase transition in spinless electron system

Figure 2: (a) Resistivity of an electron system in a SiGe/Si/SiGe quantum well
placed in the spin-polarizing magnetic field $B^{*}$ as a function of
temperature for electron densities (from top to bottom) 0.55, 0.61, 0.68,
0.74, 0.80, 0.86, 0.92, 1.01, 1.11, 1.22, 1.35, 1.54, 1.72, 2.09, 2.64, and
$3.75\times 10^{10}$ cm-2. The solid line corresponds to the separatrix. The
magnetic fields used are spanned in the range between approximately 1 and 2 T.
The inset shows a closeup view of $\rho(T)$ for $n_{\text{s}}=2.09\times
10^{10}$ cm-2. (b) Activation energy, $E_{\text{a}}$, and square root of the
threshold voltage, $V_{\text{c}}^{1/2}$, vs. electron density. Solid lines
correspond to the best linear fits. Upper inset: a typical $I$-$V$ dependence
on the insulating side of the MIT at $T=30$ mK. Lower inset: Arrhenius plots
of the temperature dependence of the resistivity for two electron densities on
the insulating side.

The electron spectrum in silicon-based 2D systems contains two almost
degenerate valleys, which is expected to promote metallicity Punnoose and
Finkel’stein (2001, 2005); Fleury and Waintal (2008). Therefore, the metallic
state may, in principle, survive in these systems in the presence of spin-
polarizing magnetic fields. Below, we show that the metallic state in ultra-
low-disorder SiGe/Si/SiGe quantum wells survives even when the electrons’
spins are fully polarized Melnikov et al. (2020).

In Fig. 2(a) we plot the resistivity $\rho(T)$, measured in a parallel
magnetic field strong enough to polarize the electron spins, for different
electron densities. The magnetic field for complete spin polarization $B^{*}$
is density-dependent and was determined by the saturation of the
$\rho(B_{\parallel})$ dependence, which corresponds to the lifting of the spin
degeneracy Okamoto et al. (1999); Vitkalov et al. (2000); the magnetic fields
used in our experiments fell within the range between approximately 1 and 2 T.
At the lowest temperatures, the resistivity exhibits a strong metallic
temperature dependence ($d\rho/dT>0$) for electron densities above a certain
critical value, $n_{\text{c}}(B^{*})$, and an insulating behavior
($d\rho/dT<0$ with resistivity diverging as $T\rightarrow 0$) for lower
densities. Assuming that the extrapolation of $\rho(T)$ to $T=0$ is valid and
taking into account that the curve separating metallic and insulating regimes
should be tilted Punnoose and Finkel’stein (2005), we identify the critical
density for the metal-insulator transition $n_{\text{c}}(B^{*})=(1.11\pm
0.05)\times 10^{10}$ cm-2 in a way similar to the case of $B=0$ Melnikov et
al. (2019). The $\rho(T)$ dependences on the metallic side of the transition
at $n_{\text{s}}$ just above the critical density are non-monotonic: while at
temperatures exceeding a density-dependent value $T_{\text{max}}$, the
derivative $d\rho/dT$ is negative (“insulating-like”), it changes sign at
temperatures below $T_{\text{max}}$. The measurements were restricted to 0.5 K
that is the highest temperature at which the saturation of the
$\rho(B_{\parallel})$ dependence could still be achieved; the restriction is
likely to reflect the degeneracy condition for the dilute electron system with
low Fermi energy.

On the metallic side of the transition ($n_{\text{s}}>n_{\text{c}}(B^{*})$),
the maximum resistivity drop with decreasing temperature below 0.5 K reaches
almost a factor of 2 (see the inset in Fig. 2(a)), which is weaker compared to
more than an order-of-magnitude drop in this system at $B=0$ Melnikov et al.
(2019). Still, the metallic temperature behavior of spinless electrons in
SiGe/Si/SiGe quantum wells remains strong and similar to that observed in
$p$-type GaAs/AlGaAs heterostructures in zero magnetic field Hanein et al.
(1998); Gao et al. (2006).

Similarly to the way it was done in the previous section, one can also deduce
the critical density for the MIT from two additional criteria not requiring
the extrapolation of the data to $T=0$: namely, vanishing of the activation
energy and nonlinearity of the current-voltage characteristics on the
insulating side of the transition ($n_{\text{s}}<n_{\text{c}}(B^{*})$). The
temperature dependences of the resistivity have an activation character on the
insulating side in the vicinity of the transition (see the lower inset in Fig.
2(b)); the activation energy $E_{\text{a}}$ is plotted in the main panel of
Fig. 2(b) as a function of $n_{\text{s}}$. The dependence is linear, which
corresponds to the constant thermodynamic density of states near the critical
point, and extrapolates to zero at $n_{\text{c}}(B^{*})=(1.07\pm 0.03)\times
10^{10}$ cm-2 which coincides, within the experimental uncertainty, with the
value of $n_{\text{c}}(B^{*})$ determined from the temperature derivative
criterion.

A typical $I$-$V$ curve measured on the insulating side of the MIT
($n_{\text{s}}<n_{\text{c}}(B^{*})$) is shown in the upper inset to Fig. 2(b).
The $V(I)$ dependence obeys Ohm’s law in a very narrow interval of currents
$\left|I\right|\lesssim 1$ pA and almost saturates at higher excitation
currents. The dependence $V_{\text{c}}^{1/2}(n_{\text{s}})$ is linear near the
MIT and extrapolates to zero at the same electron density as the
$E_{\text{a}}(n_{\text{s}})$ dependence.

The fact that the spinless electrons behave differently as compared to those
in Si MOSFETs, where the metallic temperature behavior of the resistance is
completely quenched, can be attributed to different intervalley scattering
rates Melnikov et al. (2020). The level of short-range disorder potential in
our samples is some two orders of magnitude lower than that in the least
disordered Si MOSFETs, hence the intervalley scattering rate should be small
compared to that in Si MOSFETs, corresponding to the case of two distinct
valleys. The critical electron density $n_{\text{c}}(B^{*})$ for the MIT in
the spinless electron system is higher by a factor of about 1.2 compared to
$n_{\text{c}}(0)$ in zero magnetic field, which is consistent with theoretical
calculations Dolgopolov et al. (2017). The fact that the observed metallic
temperature behavior is comparable to that in strongly interacting, spin-
unpolarized single-valley 2D systems in the cleanest $p$-type GaAs/AlGaAs
heterostructures indicates the same role of spins and distinct valleys with
respect to the existence of the metallic state and the MIT.

##  IV Spin effect on the low-temperature resistivity maximum

Early theories of the metallic state in strongly interacting 2D systems
Finkel’stein (1983, 1984); Castellani et al. (1984) were focused on the
interplay between disorder and interactions using renormalization-group
scaling theory. Later, the theory was extended to account for the existence of
multiple valleys in the electron spectrum Punnoose and Finkel’stein (2001,
2005). At temperatures well below the Fermi temperature, the resistivity was
predicted to grow with decreasing temperature, reach a maximum at
$T=T_{\text{max}}$, and then decrease as $T\rightarrow 0$. The maximum in
$\rho(T)$ dependence corresponds to the temperature at which the interaction
effects become strong enough to stabilize the metallic state and overcome the
quantum localization. This theoretical prediction, which is applicable only
within the so-called diffusive regime (roughly, $k_{\text{B}}T\tau/\hbar<1$,
where $\tau$ is the mean-free time), was found to be consistent with the
experimental $\rho(T)$ data in silicon MOSFETs Punnoose and Finkel’stein
(2001); Punnoose et al. (2010); Anissimova et al. (2007), but only in a narrow
range of electron densities near $n_{\text{c}}(0)$ for the resistivities low
compared to $\pi h/e^{2}$. However, strong temperature dependence of the
resistivity has been experimentally observed in a wide range of electron
densities: up to five times the critical density, including the so-called
ballistic regime (roughly, $k_{\text{B}}T\tau/\hbar>1$), where the
renormalization-group scaling theory is not relevant.

An alternative interpretation of the temperature dependence of the resistivity
is based on the so-called Wigner-Mott scenario, which focuses on the role of
strong electron-electron interactions. The simplest theoretical approach to
non-perturbatively tackle the interactions as the main driving force for the
MIT is based on dynamical mean-field theory (DMFT) methods Camjayi et al.
(2008); Radonjić et al. (2012); Dobrosavljević and Tanasković (2017) using the
Hubbard model at half-filling. On the metallic side near the MIT, the
resistivity is predicted to initially increase as the temperature is reduced,
reach a maximum, $\rho_{\text{max}}$, at a temperature $T_{\text{max}}$, and
then decrease as $T\rightarrow 0$ so that the resistivity change
$\rho(T)-\rho(0)$, normalized by its maximum value, is a universal function of
$T/T_{\text{max}}$. According to this theory, $T_{\text{max}}$ corresponds to
the quasiparticle coherence temperature, which is of the order of the Fermi
temperature $T_{\text{F}}$ determined by the effective electron mass
renormalized by interactions: $T_{\text{max}}\sim T_{\text{F}}$. Below this
temperature, the elastic electron-electron scattering corresponds to coherent
transport, while at higher temperatures, the inelastic electron-electron
scattering becomes strong and gives rise to a fully incoherent transport.
Notably, similar functional form of the resistivity $\rho(T)$ can be expected
within the screening theory in its general form (for more on this, see Ref.
Shashkin et al. (2020)). Similar non-monotonic $\rho(T)$ dependence with a
maximum at $T_{\text{max}}\sim T_{\text{F}}$ is predicted by another approach
based on the Pomeranchuk effect expected within a phase coexistence region
between the Wigner crystal and a Fermi liquid Spivak (2003); Spivak and
Kivelson (2004, 2006). The first two approaches allow for a quantitative
comparison with the experiment.

In Fig. 3 we plot the values of $T_{\text{max}}$ as a function of the electron
density in $B=0$ and $B=B^{*}$. The data for $T_{\text{max}}(B^{*})$ lie
significantly lower than those for $T_{\text{max}}(0)$. Interestingly, each
dependence can be described by a linear function that extrapolates to zero at
$n_{\text{s}}$ close to $n_{\text{c}}(0)$ or $n_{\text{c}}(B^{*})$, and the
slopes of both dependences are close to each other. We also plot the
calculated values of renormalized Fermi temperatures $T_{\text{F}}$ for both
cases. In zero magnetic field, the density dependences of the resistivity
maximum temperature $T_{\text{max}}(0)$ and the Fermi temperature
$T_{\text{F}}(0)$ are close to each other in the electron density range where
they overlap. However, there is a qualitative difference between the behavior
of $T_{\text{max}}$ and that of $T_{\text{F}}$ when lifting the spin
degeneracy $g_{\text{s}}$. Rather than increasing along with the Fermi
temperature, the value $T_{\text{max}}$ decreases when polarizing electron
spins Shashkin et al. (2022).

Figure 3: $T_{\text{max}}$ as a function of electron density in $B=0$
(circles) and in $B=B^{*}$ (squares). Solid lines are linear fits. Critical
electron densities for the metal-insulator transition in $B=0$ and $B=B^{*}$
are indicated. Dashed lines show the Fermi temperatures $T_{\text{F}}$ in
$B=0$ and $B=B^{*}$ calculated using the low-temperature value $B^{*}$ and Eq.
(1), see text.

The Fermi temperature $T_{\text{F}}(B^{*})$ has been calculated from the low-
temperature value $B^{*}$ based on the equality of the Fermi energy of
completely spin-polarized electrons and the Zeeman energy in the polarization
field $B^{*}$ Melnikov et al. (2017b):

$k_{\text{B}}T_{\text{F}}(B^{*})=\frac{h^{2}n_{\text{s}}}{2\pi
g_{\text{v}}m}=g_{\text{F}}\mu_{\text{B}}B^{*},$ (1)

where $k_{\text{B}}$ is the Boltzmann constant, $g_{\text{v}}=2$ is the valley
degeneracy, $m$ is the renormalized energy-averaged effective mass that is
determined by the density of states, $g_{\text{F}}\approx g_{0}=2$ is the
$g$-factor at the Fermi level, $g_{0}$ is the $g$-factor in bulk silicon, and
$\mu_{\text{B}}$ is the Bohr magneton. We argue that the Fermi temperature
$T_{\text{F}}(0)$ of spin-unpolarized electrons is approximately half of the
Fermi temperature $T_{\text{F}}(B^{*})$ of completely spin-polarized ones.
Indeed, it was experimentally shown in Ref. Shashkin et al. (2006) that the
electron spin magnetization is proportional to the parallel magnetic field in
the range up to $B=B^{*}$ for the clean, strongly interacting 2D electron
system in Si MOSFETs that is similar to the 2D electron system in SiGe/Si/SiGe
quantum wells. (For strongly disordered Si MOSFETs, the band tail of localized
electrons persists into the metallic regime Prus et al. (2003) in which case
both the nonlinear magnetization as a function of parallel magnetic field and
the shift of the dependence $B^{*}(n_{\text{s}})$ to higher densities are
observed due to the presence of localized electron moments in the band tail
Dolgopolov and Gold (2002); Gold and Dolgopolov (2002); Shashkin (2005); Teneh
et al. (2012); Pudalov et al. (2018).) Taking into account the smallness of
the exchange effects in the 2D electron system in silicon so that the
$g$-factor is approximately constant close to $g_{0}=2$ at low densities
Kravchenko and Sarachik (2004); Shashkin (2005); Melnikov et al. (2017b), this
indicates that the renormalized density of states in a spin subband is
approximately constant below the Fermi level, independent of the magnetic
field. Therefore, the change of $T_{\text{F}}$ when lifting the spin
degeneracy should be controlled by the change of $g_{\text{s}}$. As concerns
the band flattening corresponding to a peak in the density of states at the
Fermi level, observed in the 2D electron system in SiGe/Si/SiGe quantum wells,
the Fermi energy is not particularly sensitive to this flattening, at least,
not too close to the critical point Melnikov et al. (2017b). So, one expects
that the relation $T_{\text{F}}(0)\approx T_{\text{F}}(B^{*})/2$ holds for the
data in question. We stress that its accuracy is not crucial for our
qualitative results.

The dynamical mean-field theory successfully describes the closeness of
$T_{\text{max}}$ and the renormalized Fermi temperature $T_{\text{F}}$ in zero
magnetic field, as well as the resistivity drop at temperatures below
$T_{\text{max}}$ in both spin-unpolarized and fully spin-polarized electron
systems (see the next section). However, the observed decrease of
$T_{\text{max}}$ when lifting the spin degeneracy is opposite to the
predictions of DMFT. At the same time, the reduced value of $T_{\text{max}}$
in spin-polarizing magnetic fields is consistent with the predictions of the
renormalization-group scaling theory, but $T_{\text{max}}$ in zero magnetic
field is in disagreement with this theory. The observed behavior of
$T_{\text{max}}$ cannot be described by existing theories. Nor can it be
explained in terms of the increase of the residual disorder potential and the
reduction of the electron interaction strength due to the reduced spin degrees
of freedom in spin-polarizing magnetic fields, because the relation
$T_{\text{max}}\sim T_{\text{F}}$ still holds for clean Si MOSFETs Radonjić et
al. (2012); Dobrosavljević and Tanasković (2017) and low-mobility Si/SiGe
quantum wells Lu et al. (2011) in zero magnetic field Shashkin et al. (2022).
This indicates the spin-related origin of the effect.

##  V Scaling of the non-monotonic $\rho(T)$

The results of the scaling analysis of the data in $B=0$ in the spirit of the
dynamical mean-field theory Camjayi et al. (2008); Radonjić et al. (2012);
Dobrosavljević and Tanasković (2017) are shown in Fig. 4(a). The data scale
perfectly in a wide range of electron densities and are described well by the
theory in the weak-disorder limit Shashkin et al. (2020); we emphasize that at
some electron densities, the changes of the resistivity with temperature
exceed an order of magnitude. Deviations from the theoretical curve arise in
the high-temperature limit in the transient region and become pronounced for
$T>T_{\text{max}}$ at electron densities within $\sim 10$% of the critical
value, which in these samples is close to $n_{\text{c}}(0)\approx 0.88\times
10^{10}$ cm-2.

Figure 4: (a) The ratio $(\rho(T)-\rho(0))/(\rho_{\text{max}}-\rho(0))$ as a
function of $T/T_{\text{max}}$ in $B=0$. The solid line shows the result of
the dynamical mean-field theory in the weak-disorder limit Camjayi et al.
(2008); Radonjić et al. (2012); Dobrosavljević and Tanasković (2017). The
electron densities are indicated in units of $10^{10}$ cm-2. (b) The ratio
$\rho/\rho_{\text{max}}$ as a function of the product
$\rho_{\text{max}}\ln(T/T_{\text{max}})$. The solid line is the result of the
renormalization-group scaling theory Punnoose and Finkel’stein (2001, 2005).

Figure 5: (a) The ratio $\delta\rho/\delta\rho_{\text{max}}$ plotted as a
function of $T/T_{\text{max}}$ in $B=B^{*}$. The solid line is the result of
DMFT in the weak-disorder limit Camjayi et al. (2008); Radonjić et al. (2012);
Dobrosavljević and Tanasković (2017). The electron densities are indicated in
units of $10^{10}$ cm-2. (b) The analysis based on the scaling form suggested
by the renormalization-group scaling theory Punnoose and Finkel’stein (2001,
2005).

For proper perspective and comparison, we also show the results of a scaling
analysis done in accordance with the renormalization-group scaling theory
Punnoose and Finkel’stein (2001, 2005), according to which the normalized
resistivity $\rho/\rho_{\text{max}}$ should be a universal function of the
product $\rho_{\text{max}}\ln(T/T_{\text{max}})$. The results are plotted in
Fig. 4(b). Only the data obtained at $n_{\text{s}}=1.17\times 10^{10}$ cm-2
coincide nearly perfectly with the theoretical curve, although some deviations
occur at the lowest temperature. Pronounced deviations from the theory are
evident at both higher and lower $n_{\text{s}}$. At lower electron densities,
the scaled experimental curves become wider than the theoretical one, and at
higher densities, they become narrower. A similar shrinkage of the scaled
curves with increasing $n_{\text{s}}$ was reported earlier in Refs. Punnoose
and Finkel’stein (2001); Anissimova et al. (2007); Radonjić et al. (2012). One
should take into account, however, that theory Punnoose and Finkel’stein
(2001, 2005) has been developed for 2D electron systems that, on the one hand,
are in the diffusive regime and, on the other hand, their resistivities are
low compared to $\pi h/e^{2}$: at higher values of $\rho$, higher-order
corrections become important and cause deviations from the universal scaling
curve. As a result, the applicable range of parameters becomes very narrow.

In Fig. 5(a), we plot the ratio
$\delta\rho/\delta\rho_{\text{max}}=(\rho(T)-\rho(0))/(\rho(T_{\text{max}})-\rho(0))$
as a function of $T/T_{\text{max}}$ in $B=B^{*}$ so as to check the
applicability of the DMFT for spinless electrons. The curve for the highest
electron density follows the theoretical dependence in the weak-disorder limit
at all temperatures. Two other curves for lower electron densities also follow
the theoretical dependence at $T\leq T_{\text{max}}$ but deviate from the
theory at higher temperatures, revealing the behavior similar to that observed
at low $n_{\text{s}}$ in zero magnetic field Shashkin et al. (2020). Albeit
the density range of the applicability of DMFT to the completely spin-
polarized system is not as wide as that in $B=0$, the low-temperature
resistivity drop is described by the theory, similar to the case of the spin-
unpolarized electron system. In Fig. 5(b) we plot the ratio
$\rho/\rho_{\text{max}}$ in the fully spin-polarized system as a function of
$\rho_{\text{max}}(e^{2}/\pi h)\,{\text{ln}}(T/T_{\text{max}})$, which is the
scaling form suggested by the renormalization-group scaling theory Punnoose
and Finkel’stein (2001, 2005). The data do not scale in the range of electron
densities studied. Thus, the $\rho(T)$ data are best described by DMFT.

##  VI Conclusions

We have found that the metal-insulator transition occurs in the strongly-
interacting two-valley two-dimensional electron system in ultra-high mobility
SiGe/Si/SiGe quantum wells in zero magnetic field and survives in the spinless
system in spin-polarizing magnetic fields. In both cases, this is accompanied
by the non-monotonic temperature-dependent resistivity on the metallic side
near the transition. In zero magnetic field, the resistivity maximum
temperature is found to be close to the renormalized Fermi temperature.
However, rather than increasing along with the Fermi temperature, the value
$T_{\text{max}}$ decreases appreciably for spinless electrons in spin-
polarizing magnetic fields. The observed behavior of $T_{\text{max}}$ cannot
be described by existing theories. The results indicate the spin-related
origin of the effect. At the same time, the low-temperature resistivity drop
in both spin-unpolarized and spinless electron systems is described
quantitatively by the dynamical mean-field theory.

##  VII Acknowledgments

A.A.S. was supported by RSF Grant No. 22-22-00333. S.V.K. was supported by NSF
Grant No. 1904024.

## References

  * Behnia (2012) K. Behnia, Nat. Nanotechnol. 7, 488 (2012).
  * Zhu et al. (2012) Z. Zhu, A. Collaudin, B. Fauqué, W. Kang, and K. Behnia, Nat. Phys. 8, 89 (2012).
  * Schaibley et al. (2016) J. R. Schaibley, H. Yu, G. Clark, P. Rivera, J. S. Ross, K. L. Seyler, W. Yao, and X. Xu, Nat. Rev. Mater. 1, 16055 (2016).
  * Zhu et al. (2017) Z. Zhu, J. Wang, H. Zuo, B. Fauqué, R. D. McDonald, Y. Fuseya, and K. Behnia, Nat. Commun. 8, 15297 (2017).
  * Lee and Ramakrishnan (1985) P. A. Lee and T. V. Ramakrishnan, Rev. Mod. Phys. 57, 287 (1985).
  * Punnoose and Finkel’stein (2001) A. Punnoose and A. M. Finkel’stein, Phys. Rev. Lett. 88, 016802 (2001).
  * Punnoose and Finkel’stein (2005) A. Punnoose and A. M. Finkel’stein, Science 310, 289 (2005).
  * Fleury and Waintal (2008) G. Fleury and X. Waintal, Phys. Rev. Lett. 101, 226803 (2008).
  * Zavaritskaya and Zavaritskaya (1987) T. N. Zavaritskaya and É. I.. Zavaritskaya, JETP Lett. 45, 609 (1987).
  * Kravchenko et al. (1994) S. V. Kravchenko, G. V. Kravchenko, J. E. Furneaux, V. M. Pudalov, and M. D’Iorio, Phys. Rev. B 50, 8039 (1994).
  * Kravchenko et al. (1995) S. V. Kravchenko, W. E. Mason, G. E. Bowker, J. E. Furneaux, V. M. Pudalov, and M. D’Iorio, Phys. Rev. B 51, 7038 (1995).
  * Popović et al. (1997) D. Popović, A. B. Fowler, and S. Washburn, Phys. Rev. Lett. 79, 1543 (1997).
  * Shashkin and Kravchenko (2019) A. A. Shashkin and S. V. Kravchenko, Appl. Sci. 9, 1169 (2019).
  * Shashkin and Kravchenko (2021) A. A. Shashkin and S. V. Kravchenko, Ann. Phys. 435, 168542 (2021).
  * Melnikov et al. (2015) M. Y. Melnikov, A. A. Shashkin, V. T. Dolgopolov, S.-H. Huang, C. W. Liu, and S. V. Kravchenko, Appl. Phys. Lett. 106, 092102 (2015).
  * Melnikov et al. (2017a) M. Y. Melnikov, V. T. Dolgopolov, A. A. Shashkin, S.-H. Huang, C. W. Liu, and S. V. Kravchenko, J. Appl. Phys. 122, 224301 (2017a).
  * Hanein et al. (1998) Y. Hanein, U. Meirav, D. Shahar, C. C. Li, D. C. Tsui, and H. Shtrikman, Phys. Rev. Lett. 80, 1288 (1998).
  * Brussarski et al. (2018) P. Brussarski, S. Li, S. V. Kravchenko, A. A. Shashkin, and M. P. Sarachik, Nat. Commun. 9, 3803 (2018).
  * Shashkin et al. (2002) A. A. Shashkin, S. V. Kravchenko, V. T. Dolgopolov, and T. M. Klapwijk, Phys. Rev. B 66, 073303 (2002).
  * Mokashi et al. (2012) A. Mokashi, S. Li, B. Wen, S. V. Kravchenko, A. A. Shashkin, V. T. Dolgopolov, and M. P. Sarachik, Phys. Rev. Lett. 109, 096405 (2012).
  * Melnikov et al. (2019) M. Y. Melnikov, A. A. Shashkin, V. T. Dolgopolov, A. Y. X. Zhu, S. V. Kravchenko, S.-H. Huang, and C. W. Liu, Phys. Rev. B 99, 081106(R) (2019).
  * Polyakov and Shklovskii (1993) D. G. Polyakov and B. I. Shklovskii, Phys. Rev. B 48, 11167 (1993).
  * Shashkin et al. (1994) A. A. Shashkin, V. T. Dolgopolov, and G. V. Kravchenko, Phys. Rev. B 49, 14486 (1994).
  * Shashkin et al. (2001) A. A. Shashkin, S. V. Kravchenko, and T. M. Klapwijk, Phys. Rev. Lett. 87, 266402 (2001).
  * Ando et al. (1982) T. Ando, A. B. Fowler, and F. Stern, Rev. Mod. Phys. 54, 437 (1982).
  * Shashkin et al. (2007) A. A. Shashkin, A. A. Kapustin, E. V. Deviatov, V. T. Dolgopolov, and Z. D. Kvon, Phys. Rev. B 76, 241302 (2007).
  * Melnikov et al. (2020) M. Y. Melnikov, A. A. Shashkin, V. T. Dolgopolov, S.-H. Huang, C. W. Liu, A. Y. X. Zhu, and S. V. Kravchenko, Phys. Rev. B 101, 045302 (2020).
  * Okamoto et al. (1999) T. Okamoto, K. Hosoya, S. Kawaji, and A. Yagi, Phys. Rev. Lett. 82, 3875 (1999).
  * Vitkalov et al. (2000) S. A. Vitkalov, H. Zheng, K. M. Mertes, M. P. Sarachik, and T. M. Klapwijk, Phys. Rev. Lett. 85, 2164 (2000).
  * Gao et al. (2006) X. P. A. Gao, G. S. Boebinger, A. P. Mills, Jr., A. P. Ramirez, L. N. Pfeiffer, and K. W. West, Phys. Rev. B 73, 241315(R) (2006).
  * Dolgopolov et al. (2017) V. T. Dolgopolov, A. A. Shashkin, and S. V. Kravchenko, Phys. Rev. B 96, 075307 (2017).
  * Finkel’stein (1983) A. M. Finkel’stein, Sov. Phys. JETP 57, 97 (1983).
  * Finkel’stein (1984) A. M. Finkel’stein, Z. Phys. B 56, 189 (1984).
  * Castellani et al. (1984) C. Castellani, C. Di Castro, P. A. Lee, and M. Ma, Phys. Rev. B 30, 527 (1984).
  * Punnoose et al. (2010) A. Punnoose, A. M. Finkel’stein, A. Mokashi, and S. V. Kravchenko, Phys. Rev. B 82, 201308(R) (2010).
  * Anissimova et al. (2007) S. Anissimova, S. V. Kravchenko, A. Punnoose, A. M. Finkel’stein, and T. M. Klapwijk, Nat. Phys. 3, 707 (2007).
  * Camjayi et al. (2008) A. Camjayi, K. Haule, V. Dobrosavljević, and G. Kotliar, Nat. Phys. 4, 932 (2008).
  * Radonjić et al. (2012) M. M. Radonjić, D. Tanasković, V. Dobrosavljević, K. Haule, and G. Kotliar, Phys. Rev. B 85, 085133 (2012).
  * Dobrosavljević and Tanasković (2017) V. Dobrosavljević and D. Tanasković, in _Strongly Correlated Electrons in Two Dimensions_ , edited by S. V. Kravchenko (Pan Stanford Publishing, 2017), chap. 1, pp. 1–46.
  * Shashkin et al. (2020) A. A. Shashkin, M. Y. Melnikov, V. T. Dolgopolov, M. M. Radonjić, V. Dobrosavljević, S.-H. Huang, C. W. Liu, A. Y. X. Zhu, and S. V. Kravchenko, Phys. Rev. B 102, 081119(R) (2020).
  * Spivak (2003) B. Spivak, Phys. Rev. B 67, 125205 (2003).
  * Spivak and Kivelson (2004) B. Spivak and S. A. Kivelson, Phys. Rev. B 70, 155114 (2004).
  * Spivak and Kivelson (2006) B. Spivak and S. A. Kivelson, Ann. Phys. 321, 2071 (2006).
  * Shashkin et al. (2022) A. A. Shashkin, M. Y. Melnikov, V. T. Dolgopolov, M. M. Radonjić, V. Dobrosavljević, S.-H. Huang, C. W. Liu, A. Y. X. Zhu, and S. V. Kravchenko, Sci. Rep. 12, 5080 (2022).
  * Melnikov et al. (2017b) M. Y. Melnikov, A. A. Shashkin, V. T. Dolgopolov, S.-H. Huang, C. W. Liu, and S. V. Kravchenko, Sci. Rep. 7, 14539 (2017b).
  * Shashkin et al. (2006) A. A. Shashkin, S. Anissimova, M. R. Sakr, S. V. Kravchenko, V. T. Dolgopolov, and T. M. Klapwijk, Phys. Rev. Lett. 96, 036403 (2006).
  * Prus et al. (2003) O. Prus, Y. Yaish, M. Reznikov, U. Sivan, and V. Pudalov, Phys. Rev. B 67, 205407 (2003).
  * Dolgopolov and Gold (2002) V. T. Dolgopolov and A. Gold, Phys. Rev. Lett. 89, 129701 (2002).
  * Gold and Dolgopolov (2002) A. Gold and V. T. Dolgopolov, J. Phys. Condens. Matter 14, 7091 (2002).
  * Shashkin (2005) A. A. Shashkin, Phys. Usp. 48, 129 (2005).
  * Teneh et al. (2012) N. Teneh, A. Y. Kuntsevich, V. M. Pudalov, and M. Reznikov, Phys. Rev. Lett. 109, 226403 (2012).
  * Pudalov et al. (2018) V. M. Pudalov, A. Y. Kuntsevich, M. E. Gershenson, I. S. Burmistrov, and M. Reznikov, Phys. Rev. B 98, 155109 (2018).
  * Kravchenko and Sarachik (2004) S. V. Kravchenko and M. P. Sarachik, Rep. Prog. Phys. 67, 1 (2004).
  * Lu et al. (2011) T. M. Lu, W. Pan, D. C. Tsui, P. C. Liu, Z. Zhang, and Y. H. Xie, Phys. Rev. Lett. 107, 126403 (2011).