# DPDR: Gradient Decomposition and Reconstruction for Differentially Private
Deep Learning

Yixuan Liu
Renmin University of China
<EMAIL_ADDRESS>
&Li Xiong
Emory University
<EMAIL_ADDRESS>
&Yuhan Liu
Renmin University of China
<EMAIL_ADDRESS>
Yujie Gu
Kyushu University
<EMAIL_ADDRESS>
&Ruixuan Liu
Emory University
<EMAIL_ADDRESS>
&Hong Chen
Renmin University of China
<EMAIL_ADDRESS>

###### Abstract

Differentially Private Stochastic Gradients Descent (DP-SGD) is a prominent
paradigm for preserving privacy in deep learning. It ensures privacy by
perturbing gradients with random noise calibrated to their entire norm at each
training step. However, this perturbation suffers from a sub-optimal
performance: it repeatedly wastes privacy budget on the general converging
direction shared among gradients from different batches, which we refer as
common knowledge, yet yields little information gain. Motivated by this, we
propose a differentially private training framework with early gradient
decomposition and reconstruction (DPDR), which enables more efficient use of
the privacy budget. In essence, it boosts model utility by focusing on
incremental information protection and recycling the privatized common
knowledge learned from previous gradients at early training steps. Concretely,
DPDR incorporates three steps. First, it disentangles common knowledge and
incremental information in current gradients by decomposing them based on
previous noisy gradients. Second, most privacy budget is spent on protecting
incremental information for higher information gain. Third, the model is
updated with the gradient reconstructed from recycled common knowledge and
noisy incremental information. Theoretical analysis and extensive experiments
show that DPDR outperforms state-of-the-art baselines on both convergence rate
and accuracy.

##  1 Introduction

Deep learning models achieve great success in various domains, but also pose
privacy risks of the training data. For instance, adversaries are able to
reconstruct original training data from model parameters [9, 33], and infer
the membership of individuals in the training data from model outputs or
gradients [25, 30, 11]. Differential Privacy (DP) [8] is a standard privacy
notion that introduces random noise to the computation, ensuring that the
membership of any single data point remains undetectable from the output,
thereby protecting individual privacy. To achieve DP for a deep learning
model, Differentially Private Stochastic Gradient Descent (DP-SGD) [1] is one
of the most preeminent paradigms, which adds noises to gradients at each
training step. The noise level scales up with the norm of entire gradients,
which can significantly decrease model performance. To reduce noise amount,
DP-SGD and recent variants typically bound the norm by clipping with adaptive
threshold [2, 24] or scaling down gradients by normalization [4].

Improving privacy and utility tradeoff of DP-SGD is a well-recognized
challenge. Existing works still suffer from a sub-optimal performance due to a
common problem that a large amount of privacy budget is wasted on repeatedly
protecting information that has already been learned from previous iterations.
One of the key observations on gradients is that the gradients across
different batches follow a similar direction especially in the early stages
[7] (c.f. Fig. 1(Left)). The coherent direction could be regarded as the
common knowledge shared by gradients over the whole dataset. Repeatedly
collecting and protecting the common knowledge at different training steps
leads to a large privacy budget consumption in return for little information
gain.

Intuitively, by identifying the common knowledge from previous gradients and
recycling it in the subsequent steps, we can significantly save privacy budget
to only protect incremental gradient components complementary to the common
knowledge for higher information gain. A naïve solution is subtracting the
previous noisy gradient from the current one and perturbing only the
difference (c.f. Sec. 4.1). However, the difference may not remove all common
knowledge and may suffer from a norm even larger than the original gradient
norm, leading to more injected noises. Therefore, characterizing the common
knowledge precisely to keep the bounded norm of incremental information as
small as possible is a challenging problem.

To this end, we propose DPDR, a Differentially Private training framework with
gradient Decomposition and Reconstruction at early stage as shown in Fig. 2.
Specifically, it consists of a private Gradient Decomposition and
Reconstruction technique (GDR) and a mixed strategy. For GDR, it first
directionally decomposes gradients into two parts: orthogonal components
$g_{\perp}$ and parallel components $g_{\varparallel}$ based on noisy previous
gradients (c.f. Fig. 1 (Middle)). The extracted incremental information
$g_{\perp}$ is completely independent of common knowledge and achieves a
smaller norm due to Pythagorean Theorem. Then most privacy budget is spent on
perturbing $g_{\perp}$ with bounded norm, and only a small privacy budget is
used on parallel coefficient $\alpha$ for recycling common knowledge. At last,
we recover the whole gradients by summing up noisy incremental information and
common knowledge, which ensures a correct model converging direction and
accelerates the convergence rate.

Furthermore, the mixed strategy applies GDR at the early training steps and
switches to DP-SGD later. As the large proportion of common knowledge that GDR
benefits from mainly appears at early stages (c.f. Fig. 1 (Right)), it is
unnecessary to spend privacy budget on recycling common knowledge when it is
too little at later stages. Switching to DP-SGD allows full use of privacy
budget.

Figure 1: _Left_ : SGD Visualization on linear regression model. Gradient
directions are similar (coherent) at the early training stage, and fluctuate
(stale) later. _Middle_ : In subtraction, incremental information is gradient
difference $\Delta g=g_{t}-\tilde{g}_{t-1}$. In decomposition, incremental
information is orthogonal gradient $g_{\perp}=g_{t}-\alpha\cdot b$, where $b$
is normalized ${g_{t-1}}$, parallel coefficient $\alpha=\langle
g_{t},b\rangle/\|b\|^{2}$. By Pythagorean Theorem, $\Delta g\leq g_{\perp}$.
_Right_ : Norm of gradients on CIFAR10. At early stages, gradient norm
fluctuates while orthogonal norm stays small and stable, which indicates the
portion of common knowledge (green slash) is high compared to incremental
information (orange range).

Our main contributions are summarized as follows:

  * •

We develop a directional-decomposition-based privatization technique for DP-
SGD. It provides a higher information gain with less noise injection by (1)
spending most of the privacy budget on the incremental information in current
gradients, and (2) reusing the common knowledge (a general converging
direction) from historical gradients.

  * •

We design a mixed training framework DPDR based on a universal pattern that
gradients from the early training steps are more alike to each other. It
promises a better performance by making the most of the privacy budget for
obtaining more informative knowledge.

  * •

We theoretically prove that compared to DP-SGD, our proposed methods promise a
faster convergence rate benefiting from the reusing of common knowledge and
less noise injection under the same level of privacy guarantee.

  * •

Our extensive experiments on real-world datasets confirm that DPDR outperforms
DP-SGD and its SOTA variants on both convergence rate and model accuracy.

##  2 Related Work

DP-SGD proposed in [1] develops as a predominant differential private model
training framework in deep learning. Many works have made efforts to improve
the utility from different angles. Different from these works, we focus on the
fundamental operation, the perturbation on common knowledge, which is rarely
noticed. As a result, our method could be regarded as a building block for
most advanced DP-SGD variants.

Clipping strategy in DP-SGD. Clipping is introduced in DP-SGD to bound the
sensitivity of gradients. A larger clipping bound brings to large noise
amount, while smaller bound leads to bias. Therefore, the effects of clipping
is analyzed recently for formal trade-off between them [6, 28]. To reduce
noise amount, several works tunes clip bound by data-dependent strategy[2] ,
or normalize gradients for smaller gradient norm [4, 29]. These work improve
DP-SGD with a better noise scale, while keep the relationship between noise
level and whole gradient norm.

Adaptive Optimization Advanced optimizers such as Adam, Neterov boost the
effectiveness of SGD [20, 12]. However, differential private noises comprise
the performance of these optimizers seriously, as noises accumulate in the
preconditioner along iterations. Averaging and adaptive strategy [17, 31, 28,
16] are applied on preconditioner to decrease the variance of historical
noises. While they sometimes still demonstrate comparable performance with the
original DP-advanced-optimizer [26].

Projection and Heuristic Improvements Recent works attempt to project
gradients into certain space to avoid high dimension curse [32], or predefined
space [3]. An assumption is made on gradient distribution with auxiliary
information or prior knowledge. Another line of works made improvements with
better gradients selection [10]. These works are orthogonal with our method as
none of them consider the internal noise design in one gradient. We notice
that a recent work [19] proposes adding noises to the difference of
consecutive sanitize gradients on the same batch, which is similar with our
strawman approach. It performs well on low dimension datasets, while the
calculation cost doubles for gradient recomputing, and the bias of
reconstructing gradients is not considered.

Figure 2: Framework of DPDR. First, it decompose current gradient $g_{t}$ into
$g_{\perp}$ (incremental information) and $\alpha\cdot b$ by directional
decomposition based on previous normalized noisy gradients $b$ (common
knowledge). The parallel coefficient $\alpha$ and $g_{\perp}$ are perturbed
for further reconstruction with $b$. Model is updated based on reconstructed
gradient

##  3 Preliminaries

In this section, we recap the notions related to differential privacy and the
framework of DP-SGD. First, we introduce the privacy notion, Differential
privacy (DP) [8], a de facto standard that is widely accepted to provide
rigorous privacy for raw data. The formal definition is as follows.

######  Definition 1 (Differential Privacy)

For any $\epsilon,\delta\geq 0$, a randomized algorithm
$M:\mathcal{D}\rightarrow\mathcal{R}$ is $(\epsilon,\delta)$-differential
privacy if for any neighboring datasets $D,D^{\prime}\in\mathcal{D}$ and any
subsets $S\subseteq\mathcal{R}$,

$\Pr[M(D)\in S]\leq e^{\epsilon}\Pr[M(D^{\prime})\in S]+\delta.$

The DP guarantee for the function $f:\mathcal{X}^{n}\rightarrow\mathbb{R}^{d}$
is implemented by adding random noises. The noise scale is determined by
privacy budget $\epsilon$ and sensitivity $\Delta f$.

######  Definition 2 (Sensitivity)

The $l_{s}$-sensitivity of a function
$f:\mathcal{X}^{n}\rightarrow\mathbb{R}^{d}$ is $\Delta
f=\mathop{\max}\limits_{x,x^{\prime}\in\mathcal{X}^{n}}\|f(x)-f(x^{\prime})\|_{s}.$

Sensitivity captures the worst-case changes of outputs when a single input
sample differs. In deep learning, we usually adopt $l_{2}$ norm as metric.
Additionally, following property allows us ensure privacy of arbitrary post
operation on perturbed sensitive data.

######  Lemma 1 (Post-processing)

Let $M:\mathcal{X}^{n}\rightarrow\mathcal{R}$ be a randomized algorithm that
satisfies $(\epsilon,\delta)$-DP,
$f:\mathcal{R}\rightarrow\mathbb{R^{\prime}}$ be an arbitrary function. Then
$f\circ M:\mathcal{X}^{n}\rightarrow\mathbb{R^{\prime}}$ is also
$(\epsilon,\delta)$-DP.

DP-SGD provides a general scheme for private deep learning. Concretely, a
small batch of sample $L_{t}$ is randomly selected from the whole datasets $D$
with probability $\frac{B}{|D|}$, where $B$ is batch size. To protect the
averaged gradient of a batch at each training step, DP-SGD clips per-sample
gradient with pre-defined clipping bound $C$, then adds noises to the sum of
clipped gradients with sensitivity $C$:

$\tilde{g}_{t}=\frac{1}{B}(\sum_{x_{i}\in
L_{t}}g_{t}(x_{i})/\max(1,\frac{\|g_{t}(x_{i})\|_{2}}{C})+N(0,\sigma^{2}C^{2}\boldsymbol{I}))$

Where $\sigma$ is the noise scale determined by privacy budget $\epsilon$. By
clipping and perturbing, model release at each step is protected. Composition
theorem is used for accounting privacy consumption during $T$ epochs, RDP[18]
and Moment Accountant[1] are usually adopted.

##  4 Proposed Methods

In this section, we demonstrate the proposed framework DPDR. We first show the
observation on common knowledge brought by coherent gradients across steps at
the early SGD training process, and introduce a strawman approach to recycle
it for less privacy budget waste in Section 4.1. Then we introduce a
decomposition and reconstruction technique to completely disentangle the
common knowledge from noisy gradients, with a mixed strategy involving DP-SGD
for more effective use of privacy budget in Section 4.2.

###  4.1 Gradient Variation in Vanilla SGD

Figure 3: The norm of difference may be larger than the original gradient.
CIFAR10.

DPDR boosts utility by taking advantage of characteristics of gradients which
are inherently similar to each other at certain training stage. Specifically,
the gradients of stochastic batches of samples are computed at each training
step in SGD, which show two characteristics: coherence and staleness.
Coherence means that gradients overall maintain similar across consecutive
steps to a certain direction due to the similarity of samples, which is also
observed by some works [7, 5]. On the other hand, staleness captures the
difference of gradients across steps, which usually appear at two steps far
from each other or at stages when gradient directions change rapidly. [16].

A key observation in our work is that gradients are coherent at the early
stage of the training process, while easier to stale at the later stage. As
shown in Fig. 1 (Left), gradients first follow similar directions. The general
direction indicates that common knowledge repeatedly appears in each gradient
at early training stage. At later stage, fluctuating gradient direction
suggests less common knowledge preserved. Thus, reusing the common knowledge
shared among recent steps allows us to avoid repeatedly collecting and
perturbing on general direction learned already, and save privacy budget to
protect the incremental components which is the more informative part in
current gradients.

A strawman solution is subtracting the previous noisy direction from current
gradients, then adding noises to the difference and recovering it by adding
the previous noisy direction as Eq.(1).

$\tilde{g}_{t}=\frac{1}{B}\sum_{i=1}^{B}Clip((g_{t}(x_{i})-\tilde{g}_{t-1}),C)+N(0,C^{2}\sigma^{2})+\tilde{g}_{t-1}$
(1)

On the surface, this solution filters out the common knowledge
($\tilde{g_{t-1}}$) from current gradients and exploits all privacy budgets to
protect only the incremental information (difference of gradients). However,
it may not remove all common knowledge completely from current gradient. For
example, cosine similarity between the $\tilde{g}_{t-1}$ and difference
probably is nonzero in Fig1 (Middle). Therefore, when the staleness between
two consecutive gradients grows up, the difference suffers from a large norm
than expected (c.f.Fig. 3), leading to unnecessary noise injecting.

###  4.2 Directional Decomposition and Reconstruction

Figure 4: Distribution of orthogonal components of gradient are more
concentrated compared to difference. CIFAR10.

In this section, we state the whole construction of DPDR (c.f. Fig. 2.), which
consists two parts: (1) the Gradient directional Gecomposition and
Reconstruction technique (GDR) and (2) the mixed strategy, where the
decomposition technique is applied only at the early training steps for a
higher information gain.

Gradient Decomposition and Reconstruction Technique GDR decomposes gradients
based on the previous gradients and extracts an orthogonal component
completely independent with common knowledge. According to the property of
vector decomposition, the orthogonal gradient is smaller than full gradient
(c.f. Fig. 4), hence less noises is injected with smaller sensitivity. To
maintain the full information from original samples, GDR reconstructs noisy
gradients by recycling previous noisy gradients.

As shown in Algorithm 1, immediately after the first step. Gradient
$g_{t}(x_{i})$ is decomposed into orthogonal components
${g_{t}(x_{i})}_{\perp}$ and parallel component $\alpha_{t}(x_{i})\cdot b$
based on the common knowledge vector $b$ computed from the previous steps. The
parallel coefficient $\alpha_{t}(x_{i})$ quantifies the amount of common
knowledge (c.f. Eq.(2)). The whole technique is applied layer by layer. A key
operation is formalized as follows:

$\alpha_{t}(x_{i})=\langle g_{t}(x_{i}),b\rangle/\|b\|^{2}_{2}\text{,}\quad
g_{t}(x_{i})_{\perp}=g_{t}(x_{i})-b\cdot\alpha_{t}(x_{i})\text{,
}\quad\text{where }b=\tilde{g}_{t-1}/\|\tilde{g}_{t-1}\|$ (2)

We then guarantee differential privacy for Algorithm 1. At the first step, the
entire gradient is clipped by $C_{g}$ and perturbed. At the following steps,
we protect both $g_{t}(x_{i})_{\perp}$ and $\alpha_{t}(x_{i})$ by clipping and
perturbing separately. The perturbed gradient components are reconstructed by
adding $b\cdot\tilde{\alpha}_{t}(x_{i})$ back for the following training. At
last, $b$ is also updated with a normalized reconstructed gradient.

Mixed Strategy Furthermore, we propose a mixed strategy by applying the GDR at
the early training steps and maintain DP-SGD later. By spending a small
portion of privacy budget on parallel coefficient $\alpha$, GDR benefits from
the reusing common knowledge with little price, and focuses on incremental
information protection. The model gain drops when the proportion of common
knowledge in current gradients decreases. As mentioned, gradients are coherent
early and stale later, hence the proportion of common knowledge is larger
early and goes down later (Fig. 1 (Right)). Hence at later training stage when
the proportion approaches zero, protecting $\alpha$ and reusing common
knowledge wastes privacy budgets. As a result, we switch to DP-SGD for better
utility and higher efficiency.

Discussion The proposed method, DPDR, is regarded as a fundamental building
block for private deep learning, which achieves smaller noise levels when
gradients are more coherent and less stale at consecutive steps. (1) It is
noticed that $\alpha$ retains magnitude information of $g_{t-1}$, which
measures the amount of common knowledge that needs to recycle. As $\alpha$
only holds constant-level dimensions, the noise amount injected to it is
irrelevant to model parameter dimensions, thereby has little affects on
utility. Hence a large portion of privacy budget is assigned to orthogonal
components ${g_{t}(x_{i})}_{\perp}$ for higher information gain and better
performance. (2) Early stage is a range of training steps. Though the range
varies when the dataset and model change, the model performance is not
sensitive to the number of step under a large range.

Algorithm 1 Decomposition and Reconstruction: $DPDR(\cdot)$

$T$, $s$, $\\{x_{i}\\}_{i\in[n]}$, $L(w)$, $C_{\alpha}$,$C_{\perp}$, $C_{g}$,
$\sigma_{\alpha}$,$\sigma_{\perp}$,$\sigma_{g}$, learning rate $\gamma$, batch
size $B$.

model $w$

Model initializes $w_{0}$, $b=w_{0}/\|w_{0}\|$.

for $t\in[T]$ do

if $t\in[2,...,s]$ then $\triangleright$ Decomposition after first step

Random sample $\\{x_{i}\\}_{i\in L_{t}}$ with sampling ratio $B/|D|$

For each $i\in L_{t}$, compute $g_{t}(x_{i})\leftarrow\nabla L(w_{t},x_{i})$

$\triangleright$ Directional Decomposition

$g_{t}(x_{i})_{\perp},\alpha_{t}(x_{i})\leftarrow\Pi_{b}(g_{t}(x_{i}))$
$\triangleright$ Projection base: noisy grad

$\triangleright$ Clip & Perturbation

$\tilde{g}_{t\perp}\leftarrow\frac{1}{L}(\sum_{i}\frac{g_{t}(x_{i})_{\perp}}{\max(1,\lVert
g_{t}(x_{i})_{\perp}\rVert/C_{\perp})}+\mathcal{N}(0,\sigma_{\perp}^{2}C_{\perp}^{2}\boldsymbol{I}))$

$\tilde{\alpha}_{t}\leftarrow\frac{1}{L}(\sum_{i}\frac{\alpha_{t}(x_{i})}{\max(1,\alpha_{t}(x_{i})/C_{2})}+\mathcal{N}(0,\sigma_{\alpha}^{2}C_{\alpha}^{2}))$
$\triangleright$ Parallel factor

$\triangleright$ Directional Reconstruction

$\tilde{g}_{t}\leftarrow\tilde{\alpha}_{t}\cdot b+\tilde{g}_{t\perp}$

$w_{t+1}\leftarrow w_{t}-\gamma\tilde{g}_{t}$

$\triangleright$ Update Parallel Base

$b\leftarrow\tilde{g}_{t}/\|\tilde{g}_{t}\|$ $\triangleright$ Noisy Base

else

DP-SGD$(w_{t},\sigma_{g},C)$ $\triangleright$ Alternative when gradient
coherence reduce return $w_{T}$

##  5 Privacy Analysis

In this section, we first demonstrate the privacy guarantee for Algorithm 1,
and explain the reason that noises scale introduced in DPDR is smaller than
DP-SGD.

In Algorithm 1, the entire gradient is accessed and protected at the first
step. At the following steps, both $g_{\perp}$ and $\alpha$ are clipped and
perturbed separately. After early decomposition, DP-SGD is adopted at later
stages, which is also differential private.

######  Theorem 1 (Privacy Guarantee of Algorithm 1)

There exists constants $v_{1}$,$v_{2}$,$v_{3}$, batch size $B$, dataset size
$|D|$, clipping bound $C_{\perp}$, $C_{\alpha}$, $C_{g}$ and training steps
$T$ such that for any $\delta>0$, $\epsilon_{\perp}<v_{1}B^{2}/|D|^{2}T$,
$\epsilon_{\alpha}<v_{2}B^{2}/|D|^{2}T$,
$\epsilon_{\perp}+\epsilon_{\alpha}<v_{3}B^{2}/|D|^{2}T$, if noise multipliers
satisfy
$\sigma_{\perp}^{2}\geq\frac{v_{1}|B|^{2}T\ln(1/\delta)}{N^{2}\epsilon_{\perp}^{2}}$,
$\sigma_{\alpha}^{2}\geq\frac{v_{2}^{2}|B|^{2}T\ln(1/\delta)}{N^{2}\epsilon_{\alpha}^{2}}$
and
$\sigma_{g}^{2}\geq\frac{v_{3}^{2}|B|^{2}T\ln(1/\delta)}{N^{2}(\epsilon_{\alpha}+\epsilon_{\perp})^{2}}$,
Algorithm 1 is $(\epsilon_{\perp}+\epsilon_{\alpha},\delta)$-DP.

The noise scale of DPDR is much smaller compared to DP-SGD. Though noise scale
depends on the clipping bound, smaller sensitivity allows lower clipping bound
with same clipping bias. In this section, we prove that the sensitivity is
smaller. The effects of clipping bound selection will be discussed in next
section. Specifically, sensitivity of gradients in DPDR contains two parts:

$\displaystyle\nabla L(w_{t})={\nabla L(w_{t})}_{\perp}+{\nabla
L(w_{t})}_{\parallel}=\underbrace{{\nabla L(w_{t})}-\Pi_{\tilde{\nabla
L}(x_{k-1})}(\nabla L(w_{t}))}_{\Delta
f_{\perp}}+\underbrace{\Pi_{\tilde{\nabla L}(x_{k-1})}(\nabla
L(w_{t}))}_{\Delta f_{\alpha}}$ (3)

Under the assumption that gradient satisfies $\rho$-smoothness (c.f.
Assumption 1), based on triangle properties and Chernoff inequality, we
achieve upper bound of sensitivity separately. The full proof are presented in
Appendix A.2.1.

$\Delta f_{\perp}\leq\min(\|\nabla L(w_{t})\|,2\rho\|w_{t}-w_{t-1}\|)$ (4)
$\Delta f_{\alpha}\leq\|\cos\theta\cdot\nabla L(x_{t})\|\leq\|\nabla
L(w_{t})\|$ (5)

For $\Delta f_{\perp}$, it is noticed that the setting in this work with real-
world datasets is bounded by $2\rho\|w_{t}-w_{t-1}$ which is far less than
$\|\nabla L(w_{t})\|$ with at least $99\%$ probability. Thus, a smaller
clipping bound on $g_{\perp}$ is allowed at very low price of clipping bias
compared to DP-SGD.

For $\Delta f_{\alpha}$, the sensitivity is no more than entire gradient norm.
Though $\Delta f_{\alpha}$ is not as smaller as $\Delta f_{\perp}$, it makes
far less affects on performance as it is not the dominant term on noise
variance. As noises on $\alpha$ scale up with the number of model layers $m$
rather than dimension $d$, where $m\ll d$.

##  6 Convergence Analysis

In this section, we provide convergence analysis of proposed method DPEDR for
non-convex smooth optimization. The effects of noises introduced by DP
guarantee is analyzed, the per-sample clipping strategy on both orthogonal
components and parallel coefficient $\alpha$ is considered as well.

######  Assumption 1 ($\rho$-Smoothness)

The loss function is $\rho$-smooth. for any $w,w^{\prime}\in\mathbb{R}^{d}$
and batch samples $x=[x_{1},x_{2},...,x_{B}]$, we have
$\|\nabla\mathcal{L}(w,x)-\nabla\mathcal{L}(w^{\prime},x)\|\leq\rho\|w-w^{\prime}\|$.

######  Lemma 2 (Convergence without clipping bias)

If the orthogonal components $g_{\perp}$ and parallel coefficient $\alpha$ are
clipped by $C_{\perp}$ and $C_{\alpha}$, sampling ratio as $q=B/|D|$, learning
rate as $\gamma$. over the $T$ iteration, DPDR ensures that for $t=1,2,...,T$,

$\displaystyle\mathbb{E}[\|\nabla\mathcal{L}(x_{t-1})\|^{2}]\leq\frac{1}{\gamma
T}\mathcal{L}(w_{0})+\mathcal{O}(\rho\gamma
dC^{2}_{\perp}\sigma^{2}_{\perp})\vphantom{\frac{1}{\gamma T}}.\par\par$

The utility loss of DP-SGD described in Lemma 2, which is dominated by
perturbation on orthogonal components $g_{\perp}$. Clipping bound $C_{\perp}$
is crucial, which directly enlarge noise scale. While we cannot choose as
small clipping bound as we can, since the bias is introduced into gradients
due to clipping. To analyze the trade-off between DP noise and clipping, we
provide the utility loss for DPDR below.

Then we formalize gradient operation as
$\tilde{g}_{t}=\frac{1}{B}(\sum^{B}_{i=1}Clip(\alpha_{t},C_{\alpha})+N(0,C^{2}_{\alpha}\sigma^{2}_{\alpha}))\cdot
b+\frac{1}{B}(\sum^{B}_{i=1}Clip(g_{t\perp},C_{\perp})+N(0,C^{2}_{\perp}\sigma^{2}_{\perp}))$.
Next we make assumption on sampling noises caused by stochastic gradient
distribution. Along with decomposition, the sampling noises on gradients are
decomposed into the same directions. Hence we have
$\xi_{t}=\xi_{\perp,t}+\xi_{\varparallel,t}$, and
$\|\xi_{t}\|^{2}=\|\xi_{\perp,t}\|^{2}+\|\xi_{\varparallel,t}\|^{2}$. A
minimal assumption on sampling noises is defined in Assumption 2, which is a
generally adopted by recent works [28, 6].

######  Assumption 2 (Bounded Second Moment of Stochastic Gradient)

For and given dataset $D=\\{x_{1},x_{2},...,x_{n}\\}$, loss function
$L(w)=\frac{1}{n}\sum_{i=1}^{n}l(w,x_{i})$ for a random record $x_{i}$
sampling from $D$, the sampling noise is bounded by $\tau^{2}$, after
decomposition ,the sampling noises are bounded by $\tau_{\perp}^{2}$ and
$\tau_{\alpha}^{2}$, i.e., $\mathbb{E}_{x_{i}\in D}[\|\nabla L(w,x)-\nabla
l(w,x)\|^{2}]\leq\tau^{2}$, $\mathbb{E}_{x_{i}\in D}[\|\nabla
L_{\perp}(w,x)-\nabla l_{\perp}(w,x)\|^{2}]\leq\tau_{\perp}^{2}$,
$\mathbb{E}_{x_{i}\in D}[\|\nabla L_{\alpha}(w,x)-\nabla
l_{\alpha}(w,x)\|^{2}]\leq\tau_{\alpha}^{2}$.

######  Theorem 2 (Convergence with clipping threshold)

Set clipping bound on $\alpha$ as $C_{\alpha}$ and orthogonal components as
$C_{\perp}$, and the probability of clipping as
$P_{\perp,t}=\Pr[\xi_{\perp,t}\in
S_{\|\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\perp,t}\|\geq C_{\perp}}]$, as
$P_{\varparallel,t}=\Pr[\xi_{\varparallel,t}\in
S_{\|\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\varparallel,t}\|\geq
C_{\alpha}}]$ separately, sampling ratio as $q=B/|D|$, learning rate as
$\gamma$, $\gamma^{\prime}=B\gamma$. over the $T$ iteration, DPEDR ensures
that for $t=1,2,...,T$,

$\displaystyle\mathbb{E}[(1-P_{\perp,t})\|\nabla\mathcal{L}(x_{t-1})\|^{2}+\|\nabla\mathcal{L}_{\perp}(x_{t-1})\|(\frac{C_{\perp}P_{\perp,t}}{4}-\sqrt{P_{\perp,t}}\tau_{\perp,t})+\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})\|\cdot(\frac{C_{\varparallel}P_{\varparallel,t}}{4}-\sqrt{P_{\varparallel,t}}\tau_{\varparallel,t})]$
$\displaystyle\leq\frac{1}{\gamma
T}\mathcal{L}(w_{0})+\frac{\rho\gamma^{\prime}}{2}(2C_{\perp}^{2}+dC_{\perp}^{2}\sigma_{\perp}^{2}+2C_{\alpha}^{2}+mC_{\alpha}^{2}\sigma_{\alpha}^{2})+\frac{15}{4T}\sum_{t=1}^{T}\mathbb{E}[C_{\perp}\tau_{\perp,t}\sqrt{P_{\perp,t}}+C_{\alpha}\tau_{\alpha,t}\sqrt{P_{\alpha,t}}]$
$\displaystyle\leq\underbrace{\frac{1}{\gamma
T}\mathcal{L}(w_{0})}_{\text{general term of
SGD}}+\underbrace{\mathcal{O}(\rho\gamma
dC^{2}_{\perp}\sigma^{2}_{\perp})\vphantom{\frac{1}{\gamma T}}}_{\text{by DP
noises}}+\underbrace{\frac{15}{4T}\sum_{t=1}^{T}\mathbb{E}[C_{\perp}\tau_{\perp,t}\sqrt{P_{\perp,t}}+C_{\alpha}\tau_{\alpha,t}\sqrt{P_{\alpha,t}}]}_{\text{by
clipping}}\par\par\par$

The utility loss of DPDR with clipping bias is provided in Theorem 2. The
second term _DP noises_ mainly caused by the perturbation on gradients after
decomposition. Though both directions are perturbed, notice that the number of
model layers $m$ is far less than the number of model parameters $d$, the
noises is dominated $\mathcal{O}(dC_{\perp}^{2}\sigma_{\perp}^{2})$ on
orthogonal direction. The last term _clipping_ demonstrates the bias due to
clip on both directions characterized by $\mathcal{O}(C_{\perp}+C_{\alpha})$.

According to Lemma 2, a small bound reduces the noises. However, Theorem 2
indicates that small bound severely slows down the convergence as clipping
bias increases with larger probability of clipping bias $P_{\perp,t}$.
Compared with DP-SGD, DPDR alleviates such degradation by decomposition. (1)
As mentioned, the norm of dominant variable $g_{\perp}$ is smaller than $g$.
As a result, choosing clipping bound the same as DP-SGD leads to same noise
scale, while decreases the probability of clipping bias, thereby achieves
faster convergence rate in practice. In other word, at the same bias level,
DPDR is allowed to select smaller $C_{\perp}$ for less noises. (2) Clipping
bound $C_{\alpha}$ is not in dominant term, hence larger $C_{\alpha}$ is
allowed with low clipping bias probability.

##  7 Experiment Results

In this section, we demonstrate the experiment results to verify the accuracy
enhancement and convergence rate of the proposed method DPDR on public
datasets and classic deep learning models.

Datasets and Models We conduct experiments on datasets MNIST[15], CIFAR-10[14]
and SVHN[21] with model 4-layer CNN, 5-layer CNN, ResNet18 separately, by
group normalization and cross-entropy loss function for all of them. The noise
scale is derived under privacy budget $\epsilon=3$ and $8$ with fixed
$\delta=10^{-5}$. All of the results are presented with the best results after
the parameter tuning. As the hyperparameter tuning process is a common
practice for all private machine learning methods [13, 22], we don’t account
for the privacy loss in this paper. The setting details are presented in
Appendix A.1.

Baseline We compare the DRDP with DP-SGD[1] and its state-of-the-art variants
improved on clipping or adding noises: AutoClip[4], DIFF2[19], DPAdam, DIFF.
AutoClip removes the influence of clipping threshold by normalization, which
represents the line of clipping enhanced work. As mentioned in [26], DPAdam
performs even better on CIFAR-10 and similar datasets than DPAdam-variants,
hence we adopt DPAdam as an important baseline. Additionally, we also
demonstrate the performance of the strawman approach introduced in Section
4.1, and name it as DIFF.

###  7.1 Overall Performance Evaluation

Tab. 1 demonstrates the overall performance of DPDR and baselines. (1)The
proposed method DPDR achieves higher accuracy compared to existing works
across all datasets and privacy budgets. We notice that on larger datasets
CIFAR-10 and SVHN the improvement are more significant, around 2% higher than
baselines. The less promotion on MNIST is reasonable as it has almost
approached non-private accuracy. The enhancement of DPDR benefits from less
noise introduced in training process with smaller norm at the early training
stage, and maintain precision at the following phases. (2) Consistent with our
explanation above, DPAdam and Autoclip achieve similar accuracy with DP-SGD as
they perturb gradient with noises calibrated to the norm of entire gradients,
the ratio between information and noises injecting did not change. (3) An
upward trend in accuracy is seen on DIFF2 and DIFF as the model size
decreases. As a smaller model usually enjoys more coherent gradients, leading
to a much smaller norm. On the contrary, DPDR is more robust to gradient
variation, hence performs stable on different model sizes.

Table 1: Accuracy Comparison on public datasets after paramter tuning. $\epsilon$ |  Method |  MNIST |  CIFAR-10 |  SVHN
---|---|---|---|---
$3$ |  DPDR |  96.42% |  57.14% |  67.44%
|  AutoClip |  96.15% |  55.25% |  65.87%
|  DIFF |  95.65% |  52.02% |  13.60%
|  DIFF2 |  95.91% |  55.86% |  65.84%
|  DPAdam |  96.31% |  55.30% |  65.61%
|  DP-SGD |  96.16% |  55.48% |  65.87%
$8$ |  DPDR |  97.03% |  61.99% |  72.12%
|  AutoClip |  96.85% |  59.81% |  72.03%
|  DIFF |  96.54% |  52.58% |  13.79%
|  DIFF2 |  96.05% |  60.02% |  69.07%
|  DPAdam |  96.53% |  58.69% |  71.15%
|  DP-SGD |  96.62% |  59.85% |  72.05%

###  7.2 Convergence Evaluation

Fig. 5 demonstrates the convergence rate of proposed method DPDR is higher
than all other baselines. (1) Reaching the same accuracy, DPDR requires fewer
steps, which indicates less privacy budgets. For instance, on CIFAR-10, DPDR
consumes 2.59 $\epsilon$ for 55% accuracy, while DP-SGD and its variants need
0.2 higher for same accuracy. (2) Though GDR only applies at the early steps,
DPDR achieves consistently higher accuracy along all training steps. The
promotion of DPDR comes from higher accuracy achieving by faster convergence
rate of GDR at early steps, which provides a better starting point for
following DP-SGD compared to the baseline model at the same step.

Figure 5: Convergence Evaluation on CIFAR-10 with 5-layer CNN.

###  7.3 Effect of Hyperparameters

In Fig. 6, we empirically study the effects of hyperparameters in DPDR,
including batch size $B$, clipping bound of parallel coefficient $\alpha$, and
the steps $s$ for early decomposition. As all datasets show similar properties
over these parameters, we only show the impact for CIFAR-10 due to space.

We evaluate the trade-off between batch size and accuracy in Fig. 6. (1) Both
DP-SGD and proposed methods achieves the highest accuracy at $B=1024$, and
gets lower as $B$ grows. The best accuracy achieved at a medium batch size is
reasonable as larger batch size decreases and privacy amplification effect
from subsampling ratio of batches, while smaller batch size introduce more
sampling bias by stochastic gradient descent. (2) DPDR obtains higher accuracy
across all batch sizes than DP-SGD, especially when batch size is small. It is
reasonable as less samples introduce less incremental information, and common
knowledge reused enjoys less perturbation due to strong privacy amplification
effect. The result suggests the performance DPDR is more stable to smaller
batch size, which is practical in reality considering the limitation of
computational resources.

The impact of clipping bound of parallel coefficient $\alpha$ is demonstrated
in Fig. 6. The accuracy of DPDR is stable when $\alpha$ is over a wide range
of $[0.05,3]$. As smaller bound limits the effect of previous noisy gradients
(common knowledge) and restrains the convergence rate, while reducing the DP
noise amount injecting. While larger bound releashes the power of previous
noisy gradients, but introduce more noises in training process.

Fig. 6 demonstrates the effects of number of steps $s$ for gradient
decomposition and reconstruction technique (GDR) at the beginning of DPDR. (1)
The model achieves the highest accuracy at $s=50$, which is consistent with
what we observed in Fig. 1 (Right), orthogonal components are quite smaller
than the entire gradients at the early training phases. (2) It is noticed that
the accuracy decreases gradually when steps number gets larger. As common
knowledge decreases along training process, the clipping and perturbing on
parallel coefficient $\alpha$ introduce noises and bias in return for little
information gain and still consuming privacy budgets. Hence at later training
stages where gradient direction fluctuates, DP-SGD is more suitable for
finding more elaborate optimal solution.

Figure 6: Effect of Hyperparameters on CIFAR-10 with 5-layer CNN.

##  8 Conclusion

This work proposed DPDR, which focuses on enhancing the performance of DP-SGD
by reducing the amount of noise injection in gradients. We achieved a higher
information gain with a smaller amount of noise by introducing directional
decomposition and reconstruction technique. The model accuracy is further
enhanced by designing and leveraging a mixed strategy which makes the most of
the privacy budget. Comprehensive experiments on real-world datasets and
different models are conducted to confirm the effectiveness of DPDR on
accuracy and convergence. In the future, we plan to extend DPDR to federated
setting, and explore the effectiveness on advanced DP-SGD optimizers.

## References

  * [1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016.
  * [2] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy.  Differentially private learning with adaptive clipping.  Advances in Neural Information Processing Systems, 34:17455–17466, 2021.
  * [3] Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar.  Private adaptive gradient methods for convex optimization.  In International Conference on Machine Learning, pages 383–392. PMLR, 2021.
  * [4] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis.  Automatic clipping: Differentially private deep learning made easier and stronger.  Advances in Neural Information Processing Systems, 36, 2024.
  * [5] Satrajit Chatterjee.  Coherent gradients: An approach to understanding generalization in gradient descent-based optimization.  In ICLR 2020-11th International Conference on Learning Representations, 2020.
  * [6] Xiangyi Chen, Steven Z Wu, and Mingyi Hong.  Understanding gradient clipping in private sgd: A geometric perspective.  Advances in Neural Information Processing Systems, 33:13773–13782, 2020.
  * [7] Wei Dai, Yi Zhou, Nanqing Dong, Hao Zhang, and Eric P Xing.  Toward understanding the impact of staleness in distributed machine learning.  In ICLR 2019-10th International Conference on Learning Representations, 2019.
  * [8] Cynthia Dwork.  Differential privacy.  In International colloquium on automata, languages, and programming, pages 1–12. Springer, 2006.
  * [9] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart.  Model inversion attacks that exploit confidence information and basic countermeasures.  In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322–1333, 2015.
  * [10] Jie Fu, Qingqing Ye, Haibo Hu, Zhili Chen, Lulu Wang, Kuncan Wang, and Ran Xun.  Dpsur: Accelerating differentially private stochastic gradient descent using selective update and release.  arXiv preprint arXiv:2311.14056, 2023.
  * [11] Matthew Jagielski, Milad Nasr, Katherine Lee, Christopher A Choquette-Choo, Nicholas Carlini, and Florian Tramer.  Students parrot their teachers: Membership inference on model distillation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [12] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [13] Antti Koskela and Tejas D Kulkarni.  Practical differentially private hyperparameter tuning with subsampling.  Advances in Neural Information Processing Systems, 36, 2024.
  * [14] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [15] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [16] Tian Li, Manzil Zaheer, Ken Ziyu Liu, Sashank J Reddi, H Brendan McMahan, and Virginia Smith.  Differentially private adaptive optimization with delayed preconditioners.  arXiv preprint arXiv:2212.00309, 2022.
  * [17] Guanbiao Lin, Hu Li, Yingying Zhang, Shiyu Peng, Yufeng Wang, Zhenxin Zhang, and Jin Li.  Dynamic momentum for deep learning with differential privacy.  In International Conference on Machine Learning for Cyber Security, pages 180–190. Springer, 2022.
  * [18] Ilya Mironov.  Rényi differential privacy.  In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263–275. IEEE, 2017.
  * [19] Tomoya Murata and Taiji Suzuki.  Diff2: Differential private optimization via gradient differences for nonconvex distributed learning.  In International Conference on Machine Learning, pages 25523–25548. PMLR, 2023.
  * [20] Yu Nesterov.  Gradient methods for minimizing composite functions.  Mathematical programming, 140(1):125–161, 2013.
  * [21] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al.  Reading digits in natural images with unsupervised feature learning.  In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 7. Granada, Spain, 2011.
  * [22] Nicolas Papernot and Thomas Steinke.  Hyperparameter tuning with renyi differential privacy.  In ICLR 2022-13th International Conference on Learning Representations, 2022.
  * [23] Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson.  Tempered sigmoid activations for deep learning with differential privacy.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9312–9321, 2021.
  * [24] Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar.  Adaclip: Adaptive clipping for private sgd.  arXiv preprint arXiv:1908.07643, 2019.
  * [25] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017.
  * [26] Qiaoyue Tang, Frederick Shpilevskiy, and Mathias Lécuyer.  Dp-adambc: Your dp-adam is actually dp-sgd (unless you apply bias correction).  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15276–15283, 2024.
  * [27] Florian Tramer and Dan Boneh.  Differentially private learning needs better features (or much more data).  arXiv preprint arXiv:2011.11660, 2020.
  * [28] Hanshen Xiao, Zihang Xiang, Di Wang, and Srinivas Devadas.  A theory to instruct differentially-private learning via clipping bias reduction.  In 2023 IEEE Symposium on Security and Privacy (SP), pages 2170–2189. IEEE, 2023.
  * [29] Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu.  Normalized/clipped sgd with perturbation for differentially private non-convex optimization.  arXiv preprint arXiv:2206.13033, 2022.
  * [30] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri.  Enhanced membership inference attacks against machine learning models.  In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093–3106, 2022.
  * [31] Yingxue Zhou, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, and Arindam Banerjee.  Private stochastic non-convex optimization: Adaptive algorithms and tighter generalization bounds.  arXiv preprint arXiv:2006.13501, 2020.
  * [32] Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee.  Bypassing the ambient dimension: Private sgd with gradient subspace identification.  arXiv preprint arXiv:2007.03813, 2020.
  * [33] Ligeng Zhu, Zhijian Liu, and Song Han.  Deep leakage from gradients.  Advances in neural information processing systems, 32, 2019.

## References

  * [1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016.
  * [2] Galen Andrew, Om Thakkar, Brendan McMahan, and Swaroop Ramaswamy.  Differentially private learning with adaptive clipping.  Advances in Neural Information Processing Systems, 34:17455–17466, 2021.
  * [3] Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar.  Private adaptive gradient methods for convex optimization.  In International Conference on Machine Learning, pages 383–392. PMLR, 2021.
  * [4] Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis.  Automatic clipping: Differentially private deep learning made easier and stronger.  Advances in Neural Information Processing Systems, 36, 2024.
  * [5] Satrajit Chatterjee.  Coherent gradients: An approach to understanding generalization in gradient descent-based optimization.  In ICLR 2020-11th International Conference on Learning Representations, 2020.
  * [6] Xiangyi Chen, Steven Z Wu, and Mingyi Hong.  Understanding gradient clipping in private sgd: A geometric perspective.  Advances in Neural Information Processing Systems, 33:13773–13782, 2020.
  * [7] Wei Dai, Yi Zhou, Nanqing Dong, Hao Zhang, and Eric P Xing.  Toward understanding the impact of staleness in distributed machine learning.  In ICLR 2019-10th International Conference on Learning Representations, 2019.
  * [8] Cynthia Dwork.  Differential privacy.  In International colloquium on automata, languages, and programming, pages 1–12. Springer, 2006.
  * [9] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart.  Model inversion attacks that exploit confidence information and basic countermeasures.  In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322–1333, 2015.
  * [10] Jie Fu, Qingqing Ye, Haibo Hu, Zhili Chen, Lulu Wang, Kuncan Wang, and Ran Xun.  Dpsur: Accelerating differentially private stochastic gradient descent using selective update and release.  arXiv preprint arXiv:2311.14056, 2023.
  * [11] Matthew Jagielski, Milad Nasr, Katherine Lee, Christopher A Choquette-Choo, Nicholas Carlini, and Florian Tramer.  Students parrot their teachers: Membership inference on model distillation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [12] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [13] Antti Koskela and Tejas D Kulkarni.  Practical differentially private hyperparameter tuning with subsampling.  Advances in Neural Information Processing Systems, 36, 2024.
  * [14] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [15] Yann LeCun.  The mnist database of handwritten digits.  http://yann. lecun. com/exdb/mnist/, 1998.
  * [16] Tian Li, Manzil Zaheer, Ken Ziyu Liu, Sashank J Reddi, H Brendan McMahan, and Virginia Smith.  Differentially private adaptive optimization with delayed preconditioners.  arXiv preprint arXiv:2212.00309, 2022.
  * [17] Guanbiao Lin, Hu Li, Yingying Zhang, Shiyu Peng, Yufeng Wang, Zhenxin Zhang, and Jin Li.  Dynamic momentum for deep learning with differential privacy.  In International Conference on Machine Learning for Cyber Security, pages 180–190. Springer, 2022.
  * [18] Ilya Mironov.  Rényi differential privacy.  In 2017 IEEE 30th computer security foundations symposium (CSF), pages 263–275. IEEE, 2017.
  * [19] Tomoya Murata and Taiji Suzuki.  Diff2: Differential private optimization via gradient differences for nonconvex distributed learning.  In International Conference on Machine Learning, pages 25523–25548. PMLR, 2023.
  * [20] Yu Nesterov.  Gradient methods for minimizing composite functions.  Mathematical programming, 140(1):125–161, 2013.
  * [21] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al.  Reading digits in natural images with unsupervised feature learning.  In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 7. Granada, Spain, 2011.
  * [22] Nicolas Papernot and Thomas Steinke.  Hyperparameter tuning with renyi differential privacy.  In ICLR 2022-13th International Conference on Learning Representations, 2022.
  * [23] Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Úlfar Erlingsson.  Tempered sigmoid activations for deep learning with differential privacy.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9312–9321, 2021.
  * [24] Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X Yu, Sashank J Reddi, and Sanjiv Kumar.  Adaclip: Adaptive clipping for private sgd.  arXiv preprint arXiv:1908.07643, 2019.
  * [25] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017.
  * [26] Qiaoyue Tang, Frederick Shpilevskiy, and Mathias Lécuyer.  Dp-adambc: Your dp-adam is actually dp-sgd (unless you apply bias correction).  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15276–15283, 2024.
  * [27] Florian Tramer and Dan Boneh.  Differentially private learning needs better features (or much more data).  arXiv preprint arXiv:2011.11660, 2020.
  * [28] Hanshen Xiao, Zihang Xiang, Di Wang, and Srinivas Devadas.  A theory to instruct differentially-private learning via clipping bias reduction.  In 2023 IEEE Symposium on Security and Privacy (SP), pages 2170–2189. IEEE, 2023.
  * [29] Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu.  Normalized/clipped sgd with perturbation for differentially private non-convex optimization.  arXiv preprint arXiv:2206.13033, 2022.
  * [30] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri.  Enhanced membership inference attacks against machine learning models.  In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093–3106, 2022.
  * [31] Yingxue Zhou, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, and Arindam Banerjee.  Private stochastic non-convex optimization: Adaptive algorithms and tighter generalization bounds.  arXiv preprint arXiv:2006.13501, 2020.
  * [32] Yingxue Zhou, Zhiwei Steven Wu, and Arindam Banerjee.  Bypassing the ambient dimension: Private sgd with gradient subspace identification.  arXiv preprint arXiv:2007.03813, 2020.
  * [33] Ligeng Zhu, Zhijian Liu, and Song Han.  Deep leakage from gradients.  Advances in neural information processing systems, 32, 2019.

##  Appendix A Appendix

###  A.1 Experiment Settings

Table 2: Noise Multiplier for DPDR. $\epsilon$ |  Dataset |  $\sigma_{\perp}$ |  $\sigma_{\alpha}$ |  $\sigma_{g}$
---|---|---|---|---
$3$ |  MNIST |  0.81 |  2.0 |  0.803
|  CIFAR10 |  0.84 |  3.0 |  0.835
|  SVHN |  0.695 |  2.0 |  0.696
$8$ |  MNIST |  0.59 |  0.8 |  0.59
|  CIFAR10 |  0.61 |  1.0 |  0.605
|  SVHN |  0.527 |  0.8 |  0.531

Table 2 demonstrates the noise multipliers for DPDR and DPSGD.

We conduct experiments on datasets MNIST[15], CIFAR10[14] with model 4-layer
CNN, 5-layer CNN which follows [23, 27], and set ResNet18 on SVHN[21] for
evaluating DPDR on larger model. All of models adopts group normalization and
cross-entropy loss function.

We set all parameters as follows. The models are trained for 20 epochs under
privacy budget $\epsilon=3$ and $8$ with fixed $\delta=10^{-5}$, batch size
$B=256$. We tune hyperarameters clipping bound for each model and adopts
result based on best parameter combination. The tuning ranges are:
$C_{g}\in[0.05,1]$, $C_{\perp}\in[0.05,1]$, $C_{\alpha}in{0.05,1.5}$, learning
rate $\gamma\in[0.1,2]$. Specifically, for DIFF and DIFF2, the tuning range of
clipping bound $C_{d}\in[0.001,1]$.

In DPDR, perturbing techique switches from GDR to DPSGD. The parameter
selection on DPSGD phase is just the same as original DPSGD. We actually find
out that the best clipping bounds and learning rates on original DPSGD and
DPDR-DPSGD are the same.

###  A.2 Proofs in Theoretical Analysis

####  A.2.1 Upper bound of sensitivity

##### Sensitivity $\Delta f_{\perp}$

We analyze it from two angles. On the one hand, we generate the inequality
based on triangle properties as follows:

$\displaystyle\|\Delta f_{\perp}\|=\|\nabla L(w_{t})\cdot\sin(\nabla
L(w_{t}),\tilde{\nabla L}(w_{t-1})\|\leq\|\nabla L(w_{t})\|$

On the other hand, we could bound the noises with a more elaborate way:

$\displaystyle\|\Delta f_{\perp}\|$ $\displaystyle\leq(\|\nabla
L(w_{t})-\Pi_{\tilde{\nabla L}(x_{k-1})}(\nabla
L(w_{t}))\|^{2}+\|\tilde{\nabla L}(x_{k-1})-\Pi_{\tilde{\nabla
L}(x_{k-1})}(\nabla L(w_{t}))\|^{2})^{\frac{1}{2}}$ $\displaystyle=\|{\nabla
L}(w_{t})-\tilde{\nabla L}(w_{t-1})\|\leq\|\nabla L(w_{t})-\nabla
L(w_{t-1})\|+\|\xi_{t-1}\|$

Based on Chernoff inequality, we have

$\displaystyle\Pr(|\xi_{t-1}-\mathbb{E}[\xi_{t-1}]|<a)=\Pr(|\xi_{t-1}|<a)\geq
1-\frac{\mathbb{D}[\xi_{t-1}]}{a^{2}}\geq
1-\frac{(dC^{2}_{\perp}+mC^{2}_{2})v|B|^{2}T\ln(1/\delta)}{N^{2}\epsilon_{\perp}^{2}a^{2}}$

Considering the fact that real-world datasets usually satisfies that large
dataset size, model size and small batch size, if $C_{\perp}=10^{-1}\|\nabla
L(w_{t})-\nabla L(w_{t-1})\|$, with probability of at least $99\%$ we have
$a<\|\nabla L(w_{t})-\nabla L(w_{t-1})\|$. Based on $a$ and the assumption
that gradient satisfies $\rho$-Lipschitz condition that $\nabla
L(w_{t})-\nabla L(w_{t-1})\leq\rho(w_{t}-w_{t-1})$, $\Delta f_{\perp}$ is
bounded with high probability that $\|\Delta f_{\perp}\|\leq 2\|\nabla
L(w_{t})-\nabla L(w_{t-1})\|\leq 2\rho\|w_{t}-w_{t-1}\|$.

Therefore, we achieve the upper bound for sensitivity of orthogonal
components:

$\displaystyle\|\Delta f_{\perp}\|\leq\min(\|\nabla
L(w_{t})\|,2\rho\|w_{t}-w_{t-1}\|)$ (6)

##### Sensitivity $\Delta f_{\alpha}$

The upper bound of sensitivity for parallel coefficient

$\displaystyle\|\Delta
f_{\alpha}\|=\frac{\|<L(w_{t}),\dot{\tilde{L}}(w_{t-1})>\|}{\|\dot{{\tilde{L}}}(w_{t-1})\|^{2}}=\|\cos\theta\cdot\nabla
L(x_{t})\|\leq\|\nabla L(w_{t})\|$

####  A.2.2 Theorem 2

###### Proof 1

According to the perturbation in DPDR, we have
$\hat{g}_{t}=\frac{1}{B}\sum^{B}_{i=1}{\alpha}_{t}\cdot b+{g}_{\perp t}$ ,
$\tilde{g}_{t}=\frac{1}{B}(\sum^{B}_{i=1}Clip(\alpha_{t},C_{\alpha})+N(0,C^{2}_{\alpha}\sigma^{2}_{\alpha}))\cdot
b+\frac{1}{B}(\sum^{B}_{i=1}Clip(g_{t\perp},C_{\perp})+N(0,C^{2}_{\perp}\sigma^{2}_{\perp}))$,
where $b=\frac{\tilde{g}_{t-1}}{\|\tilde{g}_{t-1}\|}$. Under Assumption 1 we
have

$\displaystyle\mathcal{L}(w_{t+1})$
$\displaystyle\leq\mathcal{L}(w_{t})+<\nabla\mathcal{L}(w_{t}),w_{t+1}-w_{t}>+\frac{\rho}{2}\|w_{t+1}-w_{t}\|^{2}_{2}$
$\displaystyle=\mathcal{L}(w_{t})-\gamma<\nabla
L(w_{t}),\tilde{g}_{t}>+\frac{\rho}{2}\gamma^{2}\|\tilde{g}_{t}\|^{2}_{2}$

Take expectation at both sides,

$\displaystyle\mathbb{E}[\mathcal{L}(w_{t})-\mathcal{L}(w_{t-1})]\leq-\gamma\mathbb{E}[<\nabla\mathcal{L}(w_{t}),\tilde{g}_{t}>]+\frac{\rho}{2}\mathbb{E}[\|\tilde{g}_{t}\|^{2}_{2}]$
(7)

Now we analyze two terms on the right side of Eq.(7) separately.

For first term,

$\displaystyle\mathbb{E}[<\nabla L(w_{t}),\tilde{g_{t}}>]=<\nabla
L(w_{t-1}),\mathbb{E}[Clip(g_{\perp},C_{\perp})]>+<\nabla
L(w_{t-1}),\mathbb{E}[Clip(\alpha,C_{\alpha})\cdot b]>$ (8)

The equation comes from the fact that DP noise $\mathbb{E}[N(0,a^{2})]=0$ for
arbitrary standard deviation $a$. We represent the sampling noise as
$\xi_{t}=\xi_{\perp,t}+\xi_{\varparallel,t}=(g_{\perp,t}-\nabla\mathcal{L}_{\perp}(x_{t-1}))+(g_{\varparallel,t}-\nabla\mathcal{L}_{\varparallel}(x_{t-1}))=g_{t}-\nabla\mathcal{L}(x_{t-1})$,
and the probability of large sampling noise as
$P_{\perp,t}=\Pr[\xi_{\perp,t}\in
S_{\|\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\perp,t}\|<C_{\perp}}]$. Then we
have

$\displaystyle<\nabla L(w_{t-1}),\mathbb{E}[Clip(g_{\perp},C_{\perp})]>$
$\displaystyle=<\nabla
L_{\perp}(w_{t-1}),\mathbb{E}[Clip(g_{\perp},C_{\perp})]>+<\nabla
L_{\varparallel}(w_{t-1}),\mathbb{E}[Clip(g_{\perp},C_{\perp})]>$
$\displaystyle=\mathbb{E}[\boldsymbol{1}_{\|\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\perp,t}\|<C_{\perp}}(<\nabla\mathcal{L}_{\perp}(x_{t-1}),\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\perp,t}>+0)]$
$\displaystyle+C_{\perp}\mathbb{E}[\boldsymbol{1}_{\|\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\perp,t}\|\geq
C_{\perp}}(<\nabla\mathcal{L}_{\perp}(x_{t-1}),\frac{\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\perp,t}}{\|\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\perp,t}\|}>+0)]$
$\displaystyle\geq
P_{\perp,t}\|\nabla\mathcal{L}_{\perp}(x_{t-1})\|^{2}-\|\nabla\mathcal{L}_{\perp}(x_{t-1})\|\cdot\sqrt{(1-P_{\perp,t})\cdot\tau_{\perp,t}^{2}}+\mathbb{E}[\frac{C_{\perp}(1-P_{\perp,t})\|\nabla\mathcal{L}_{\perp}(x_{t-1})\|}{4}-\frac{15C_{\perp}\sqrt{(1-P_{\perp,t})}\tau_{\perp,t}}{4}]$
(9)

The first equation comes from the fact that Similarly, for the parallel part
we demote the probability of large sampling noise as
$P_{\varparallel,t}=\Pr[\xi_{\varparallel,t}\in
S_{\|\nabla\mathcal{L}_{\perp}(x_{t-1})+\xi_{\varparallel,t}\|<C_{\alpha}}]$.

$\displaystyle<\nabla L(w_{t-1}),\mathbb{E}[Clip(\alpha,C_{\alpha})\cdot b]>$
$\displaystyle\geq\mathbb{E}[\boldsymbol{1}_{\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})+\xi_{\varparallel,t}\|<C_{\alpha}}<\nabla\mathcal{L}_{\varparallel}(x_{t-1}),\nabla\mathcal{L}_{\varparallel}(x_{t-1})+\xi_{\varparallel,t}>]$
$\displaystyle+\mathbb{E}[\boldsymbol{1}_{\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})+\xi_{\varparallel,t}\|\geq
C_{\varparallel}}<\nabla\mathcal{L}_{\varparallel}(x_{t-1}),\frac{\nabla\mathcal{L}_{\varparallel}(x_{t-1})+\xi_{\varparallel,t}>}{\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})+\xi_{\varparallel,t}\|}]$
$\displaystyle\geq
P_{\varparallel,t}\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})\|^{2}-\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})\|\cdot\sqrt{(1-P_{\varparallel,t})\cdot\tau_{\varparallel,t}^{2}}+\mathbb{E}[\frac{C_{\alpha}(1-P_{\varparallel,t})\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})\|}{4}-\frac{15C_{\alpha}\sqrt{1-P_{\varparallel,t}}\tau_{\varparallel,t}}{4}]$
(10)

the first inequatility results from when clipping happens,
$\frac{\alpha}{\|\alpha\|}\cdot b\geq\frac{\alpha\cdot b}{\|\alpha\cdot
b\|}\geq\frac{\nabla\mathcal{L}_{\varparallel}(x_{t-1})+\xi_{\varparallel,t}}{\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})+\xi_{\varparallel,t}\|}$
Take Eq. (1) and (1) back into Eq. (8), we obtain the first term of Eq. (7)

For second term,

$\displaystyle\mathbb{E}[\|\tilde{g}_{t}\|^{2}_{2}]=\mathbb{E}[\|(Clip(g_{\perp},C_{\perp})]+\mathbb{E}[\|\eta_{\perp}\|^{2}]+\mathbb{E}[\|Clip(\alpha,C_{\alpha})\cdot
b\|^{2}]+\mathbb{E}[\|\eta_{\alpha}\cdot b\|^{2}]$
$\displaystyle\leq\frac{1}{n^{2}q^{2}}(C_{\perp}(n(n-1)q^{2}+qn)+dC^{2}_{\perp}\sigma^{2}_{\perp}+C_{\alpha}(n(n-1)q^{2}+qn)+mC^{2}_{\alpha}\sigma^{2}_{\alpha})$

Overall,

$\displaystyle\mathbb{E}[<\nabla\mathcal{L}(w_{t}),\tilde{g}_{t}>]\leq\frac{1}{\gamma}\mathbb{E}[\mathcal{L}(w_{t})-\mathcal{L}(w_{t-1})]+\frac{\rho\gamma}{2}\mathbb{E}[\|\tilde{g}_{t}\|^{2}$
(11)

$\displaystyle
P_{\perp,t}\|\nabla\mathcal{L}(x_{t-1})\|^{2}+\|\nabla\mathcal{L}_{\perp}(x_{t-1})\|\cdot(\frac{C_{\perp}(1-P_{\perp,t})}{4}-\sqrt{1-P_{\perp,t}}\tau_{\perp,t})$
$\displaystyle+\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})\|\cdot(\frac{C_{\varparallel}(1-P_{\varparallel,t})}{4}-\sqrt{1-P_{\varparallel,t}}\tau_{\varparallel,t})$
$\displaystyle\leq\frac{1}{\gamma}\mathbb{E}[\mathcal{L}(w_{t})-\mathcal{L}(w_{t-1})]+\frac{\rho\gamma}{2}\mathbb{E}[\|\tilde{g}_{t}\|^{2}+\mathbb{E}[\frac{15C_{\perp}\sqrt{(1-P_{\perp,t})}\tau_{\perp,t}}{4}+\frac{15C_{\alpha}\sqrt{(1-P_{\alpha,t})}\tau_{\alpha,t}}{4}]$

Considering T steps,

$\displaystyle\mathbb{E}[P_{\perp,t}\|\nabla\mathcal{L}(x_{t-1})\|^{2}+\|\nabla\mathcal{L}_{\perp}(x_{t-1})\|\cdot(\frac{C_{\perp}(1-P_{\perp,t})}{4}-\sqrt{1-P_{\perp,t}}\tau_{\perp,t})$
$\displaystyle+\|\nabla\mathcal{L}_{\varparallel}(x_{t-1})\|\cdot(\frac{C_{\varparallel}(1-P_{\varparallel,t})}{4}-\sqrt{1-P_{\varparallel,t}}\tau_{\varparallel,t})]$
$\displaystyle\leq\frac{1}{\gamma
T}\mathcal{L}(w_{0})+\frac{\rho\gamma^{\prime}}{2}(2C_{\perp}^{2}+dC_{\perp}^{2}\sigma_{\perp}^{2}+2C_{\alpha}^{2}+mC_{\alpha}^{2}\sigma_{\alpha}^{2})$
$\displaystyle+\frac{15}{4T}\sum_{t=1}^{T}\mathbb{E}[C_{\perp}\tau_{\perp,t}\sqrt{(1-P_{\perp,t})}+C_{\alpha}\tau_{\alpha,t}\sqrt{(1-P_{\alpha,t})}]$
$\displaystyle=\leq\frac{1}{\gamma T}\mathcal{L}(w_{0})+\mathcal{O}(\rho\gamma
dC^{2}_{\perp}\sigma^{2}_{\perp}+mC^{2}_{\alpha}\sigma^{2}_{\alpha})+\frac{15}{4T}\sum_{t=1}^{T}\mathbb{E}[C_{\perp}\tau_{\perp,t}\sqrt{(1-P_{\perp,t})}+C_{\alpha}\tau_{\alpha,t}\sqrt{(1-P_{\alpha,t})}]$


arxiv-pic-prez/

11institutetext: National Research University Higher School of Economics,
Moscow
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
22institutetext: Babyblog LTD, Moscow

# Transformer-based classification of user queries for medical consultancy
with respect to expert specialization

Dmitry Lyutkin 11 Andrey Soloviev 22 Dmitry Zhukov 22 Denis Pozdnyakov 11
Muhammad Shahid Iqbal Malik 11 Dmitry I. Ignatov 11

###### Abstract

The need for skilled medical support is growing in the era of digital
healthcare. This research presents an innovative strategy, utilizing the
RuBERT model, for categorizing user inquiries in the field of medical
consultation with a focus on expert specialization. By harnessing the
capabilities of transformers, we fine-tuned the pre-trained RuBERT model on a
varied dataset, which facilitates precise correspondence between queries and
particular medical specialisms. Using a comprehensive dataset, we have
demonstrated our approach’s superior performance with an F1-score of over 92%,
calculated through both cross-validation and the traditional split of test and
train datasets. Our approach has shown excellent generalization across medical
domains such as cardiology, neurology and dermatology. This methodology
provides practical benefits by directing users to appropriate specialists for
prompt and targeted medical advice. It also enhances healthcare system
efficiency, reduces practitioner burden, and improves patient care quality. In
summary, our suggested strategy facilitates the attainment of specific medical
knowledge, offering prompt and precise advice within the digital healthcare
field.

###### Keywords:

Transformers, Query Matching, Medical Texts, Many-class Learning

##  1 Introduction

The demand for qualified medical assistance has never been more significant,
especially in the digital era. As online platforms increasingly serve as
crucial sources of medical information and support [1], ensuring the provision
of accurate and specialized advice becomes imperative. One such platform that
has garnered attention is Babyblog.ru [2], which uniquely leverages user-
generated content as a gateway and contextual backdrop for medical
professionals’ knowledge dissemination.

However, the abundance of user-generated content poses challenges regarding
the scientific credibility and reliability of the information shared [3].
Consequently, there is a pressing need to implement mechanisms that ensure the
verification and enrichment of user-generated content through the input and
recommendations of diverse professionals, including doctors, psychologists,
speech therapists, and educators. This collaborative approach allows
professionals to review user posts, comments, and discussions, thereby
providing expert insights, correcting non-specialist advice, and ensuring the
delivery of accurate and reliable medical information.

Given the substantial volume of user-generated content across various
platforms, encompassing a wide array of topics including medical, quasi-
medical, and non-medical domains, the challenge of identifying content
requiring medical or professional verification becomes increasingly
significant. Furthermore, the importance of classifying this diverse content
based on thematic specialization emerges as a critical factor in directing
relevant user queries to the appropriate professionals for verification
purposes.

To address these challenges, the research and development team embarked on the
development of an automatic classifier for medical texts. This classifier aims
to determine the likelihood of associating a given text with a specific
medical specialization. The envisioned implementation involves integrating the
classifier into the platform, wherein it identifies medical content and
assigns corresponding medical specializations. Subsequently, professionals in
the respective specializations are notified to verify the content and provide
appropriate responses.

The successful development of the classification system offers multiple
benefits, including streamlining the verification process by reducing
irrelevant information presented to medical professionals, alleviating the
workload involved in content verification, and accelerating the provision of
professional responses to users. Moreover, the proposed system serves as a
valuable tool in improving the quality, completeness, and reliability of
medical information related to conception, pregnancy, and motherhood on the
platform.

This study aims to explore the efficacy of a transformer-based system in
classifying user-generated medical content within the context of Babyblog.ru.
By leveraging advanced Natural Language Processing (NLP) techniques, this
research endeavors to revolutionize the ways users access specialized medical
expertise, ensuring the delivery of timely and accurate guidance while
upholding scientific rigor and reliability.

##  2 Related works

In the realm of medical text classification, the research paper titled
”Automatic Medical Specialty Classification Based on Patients’ Description of
Their Symptoms” [4] presents a significant contribution to the field. The
study proposes a pioneering Hybrid Model (HyM) that combines multiple deep
learning techniques, including LSTM, TEXT-CNN, BERT, and TF-IDF, along with an
attention mechanism to address the critical challenge of accurately directing
patients to the appropriate medical specialty based on their symptom
descriptions.

The article ”Text Classification Using Improved Bidirectional Transformer” [5]
presents a significant contribution to the field of text processing,
particularly in the context of handling large amounts of text data generated
daily. The authors highlight the necessity for automation in text data
handling and discuss recent developments in text processing, including
attention mechanisms and transformers, as promising methods to address this
need.

In their study, the authors introduce a novel bidirectional transformer
(BiTransformer) model, constructed using two transformer encoder blocks that
utilize bidirectional position encoding. By considering both forward and
backward position information of the text data, the proposed BiTransformer
aims to capture more comprehensive contextual dependencies, enhancing the
model’s ability to handle complex text data.

To evaluate the effectiveness of attention mechanisms in the classification
process, the authors compare four models, namely long short-term memory
(LSTM), attention, transformer, and their proposed BiTransformer. Experiments
are conducted on a large Turkish text dataset comprising 30 categories,
allowing for a comprehensive assessment of the models’ performance.

One of the notable findings of the study is the promising results obtained
from the classification models that employ transformer and attention
mechanisms compared to classical deep learning models. This demonstrates the
potential of attention mechanisms and transformers in text classification
tasks, showcasing their ability to capture meaningful patterns and context in
textual data.

The authors also investigate the impact of using pretrained embeddings on the
models’ performance. Pretrained embeddings, which capture semantic
representations of words based on large corpora, have been a popular approach
to improve model performance in various NLP tasks. The study sheds light on
how pretrained embeddings can further enhance the efficiency and accuracy of
text classification models.

Perhaps the most significant result of the study is the superior performance
of their proposed BiTransformer in text classification. By effectively
incorporating bidirectional position encoding and leveraging transformer-based
architecture, the BiTransformer outperforms other models in accurately
categorizing the text data.

”Text Classification Using Improved Bidirectional Transformer” provides
insights into the potential of attention mechanisms and transformers in text
processing. The introduction of the BiTransformer and its superior performance
in text classification open up new avenues for future research and application
of transformer-based models in NLP tasks. The study’s findings have important
implications for automating text data handling, sentiment analysis,
information retrieval, and other text-related applications. As the demand for
efficient and accurate text processing techniques continues to grow, this
research makes a significant contribution to the advancement of the field and
serves as a valuable reference for researchers and practitioners in the domain
of natural language processing.

##  3 Data Collection: Building a Comprehensive Dataset for Medical Text
Classification

In this section, we describe the process of building the dataset. It includes
developing data parsers and normalizers to create a normalized dataset for the
experimental setup.

###  3.1 Data parsing

To obtain a suitable training sample, we extensively explored various Russian-
language websites that provide public access to medical questions posed by
users to healthcare professionals. We employed specific criteria to select our
data sources, including: 1) presence of openly accessible sections containing
medical questions, 2) availability of pre-annotated questions based on medical
specialization, and 3) responses provided by healthcare professionals, which
verified the appropriateness of the assigned medical specialization to the
responding doctor.

Based on our analysis, we selected the following sources for data acquisition:
sprosivracha.com [6], doctu.ru [7], 03online.com [8] and health.mail.ru [9].
We developed software for parsing these data sources, allowing for the
asynchronous, multi-threaded retrieval of information from public data
sources. The software was designed to extract relevant information from the
HTML structure and store them for further processing.

Table 1: Comparison of Medical Question Platforms Website |  Number of Questions | Percentage of Total
---|---|---
sprosivracha.com |  550,000 | 23.2
doctu.ru |  83,000 | 3.5
03online.com |  1,148,000 | 48.4
health.mail.ru |  590,000 | 24.9

In the subsequent step, the algorithm asynchronously processes each row of the
obtained table and retrieves the HTML code of the page containing the question
posed to the doctor. From each HTML code, the algorithm extracts the question
text and the doctor’s specialization using predefined tags and classes that
enclose the relevant text.

The extracted data (question text and doctor’s specialization) are then added
to the same table, complementing the rows with the data source (URL as the
data source identifier).

Once the parsing process and table population are complete, all the acquired
data are exported to a CSV file for further processing.

###  3.2 Data Augmentations

After analyzing the acquired dataset, we noticed that the distribution of data
units across medical specializations followed a pattern similar to a Pareto
distribution. This observation can be attributed to the fact that certain
medical specializations are in higher demand compared to others, resulting in
a significant number of data units belonging to those specific classes.
However, to ensure the stability and resilience of our classifier, it was
crucial to address the class imbalance issue [10].

To tackle class imbalance and enhance the model’s generalization ability, we
employed data augmentation methods facilitated by the Albumentations library
[11]. This versatile tool enabled us to create new textual data by rearranging
word positions within sentences, preserving the overall context and meaning.
This diversification of input data aimed to produce a more balanced and
comprehensive dataset. Specifically, our data augmentation techniques involved
shuffling words and reordering sentence components.

Through the augmentation process, we were able to generate additional data
points for the minority classes, effectively reducing the class imbalance and
achieving a more uniform distribution across all medical specializations. This
augmentation strategy not only helped to improve the classifier’s performance
for underrepresented classes but also enhanced its ability to handle unseen
data during the testing phase.

After applying all the necessary transformations and augmentations, we
successfully obtained a dataset with a more uniform distribution of classes
and expanded the original dataset to 5 million texts, where there are
approximately 50000 exemplars per class. This balanced dataset formed the
basis for training and evaluating our proposed framework.

Figure 1: Class Distribution after transformation and augmentation (first 70
classes).

The development of the proposed dataset arises from the recognition of a
crucial need in the field. While there exist analogous datasets, they exhibit
certain limitations in adequately covering a comprehensive spectrum of
diseases and medical conditions. Additionally, these existing datasets suffer
from a paucity of records, which impedes their capacity to comprehensively
represent the diverse range of health concerns. A further challenge lies in
the nature of the content within these datasets; predominantly composed in
technical language, they lack congruence with the narratives of individuals
detailing their ailments. This discrepancy hampers the efficacy of these
datasets in capturing the nuanced descriptions of health issues as articulated
by individuals themselves. In light of these deficiencies, the development of
the proposed dataset emerges as a pivotal endeavor, with the intent to address
these gaps and furnish a resource that aligns more closely with the authentic
narratives of people regarding their health conditions. Through the proposed
dataset, an avenue is created to elicit novel insights that may have remained
obscured within the confines of the existing datasets, fostering a more
holistic understanding of individuals’ health experiences.

##  4 Proposed Methodology

This section provides details of proposed methodology. We explore various
methods of transformers and their training. The pipeline is presented in
Figure 2.

Figure 2: Processing pipeline.

###  4.1 Transformer Models

Typically, neural networks are trained using the backpropagation algorithm
[12], which optimizes model parameters by computing gradients to improve
generalization performance through error minimization and/or enhancing metrics
on the validation set. However, this method heavily relies on the choice of
optimization algorithm [13], as there is a risk of getting stuck in local
minima during gradient computation, leading to the model’s inability to learn
and improve prediction/recognition quality (vanishing gradients). To address
this, we employed the AdamW [14] optimizer, one of the state-of-the-art
methods, which leverages information about the learning rate history to
approximate the direction of the anti gradient while incorporating momentum to
expedite the convergence of our function. This optimizer significantly
improves model training; however, it is sensitive to the choice of the
learning rate. Hence, we employed a learning rate scheduler that suits our
task best - the cosine scheduler [15]. This scheduler adjusts the learning
rate for each batch of data, allowing transformers to adaptively change the
learning rate. We opted for the cross-entropy loss function as our choice for
the loss function, as it measures how well the model is trained for
classification tasks. The utilization of the AdamW optimizer and the linear
scheduler with a warm-up for training text classifiers based on BERT has
proven effective for several reasons:

AdamW Optimizer: AdamW is a variant of the Adam optimizer that has been shown
to work well for fine-tuning pre-trained models such as Transformer [16]. It
addresses the weight decay issue in Adam, helping to prevent overfitting [17].

Cosine Scheduler: The cosine scheduler modifies learning rate starting with a
lower learning rate and gradually increases it over a specified number of
training steps. This warm-up period allows the model to converge faster,
preventing instability or fluctuations in the loss function during training
[18].

It is worth noting that the optimal training methods may vary depending on the
specific task and the data being used, requiring experimentation and fine-
tuning.

Furthermore, there are several reasons why transformer-based models have
emerged as the preferred choice for medical text classification compared to
classical machine learning methods [19]:

Pretraining: Transformers are pre-trained on large corpora of texts, which
provides them with a strong knowledge base and an understanding of language
patterns and word relationships. This pretraining allows models like
Transformer to perform well across various NLP tasks with limited fine-tuning.

Contextual Representation: Transformers employ bidirectional attention
mechanisms to create contextual word representations, enabling them to capture
the context and meaning of words within a sentence. This is particularly
crucial for text classification, where understanding sentence context is key
to assigning the correct label.

Transfer Learning: The pretraining and fine-tuning process of BERT allows for
transfer learning, where a pre-trained model on a related domain can be
accurately fine-tuned for specific tasks with a limited amount of labeled
data. This is a significant advantage for text classification tasks, which
often have a limited number of annotated data.

Superior Performance: Transformers have shown to outperform traditional
machine learning methods in various NLP tasks, including text classification.
This can be attributed to their ability to capture contextual representations
and word relationships, which are crucial for understanding sentence
semantics.

Pretraining on Russian Texts: Models pre-trained on large Russian corpora
exhibit improved performance and quality in Russian text processing tasks
compared to training from scratch. Raw textual data provides models with a
natural foundation for building language contextual representations. The size
of the raw text corpus is crucial during the pretraining phase.

It is important to note that traditional machine learning methods are still
widely used and can yield good results for specific NLP tasks. However, the
possibilities offered by pretraining, contextual representation, transfer
learning, availability of Russian language models, and the use of raw texts
for training make transformers a powerful tool for medical text
classification.

###  4.2 Training Process

The training algorithm makes use of the architecture and pre-trained weights
of a transformer model, obtained from the transformers [20] package, and
cached for subsequent utilization. During this phase, the model initialization
is executed, which includes the initiation of the tokenizer via the
AutoTokenizer module from the transformers library. Additionally, the output
layer of the model is modified to suit the specific task at hand.

Subsequently, an optimal batch size is determined by generating an artificial
dataset and conducting a grid search to identify the batch size that optimally
balances computational efficiency and resource utilization. This strategic
step is essential to ensure the model’s efficiency during computations on the
server.

During the course of training, a significant aspect involves the aggregation
of energy following the application of the softmax activation function [21].
This process offers insights into the model’s confidence levels for each
distinct class. The resulting energy accumulation, presented in the form of
probability scores, functions as an indicator of the model’s assurance in
assigning input data to specific classes. This measure of confidence holds a
central role in the model’s final predictions, contributing to its ability to
make well-informed decisions about the designated classes. It’s important to
mention that the target labels are numerical class identifiers, previously
encoded using the LabelEncoder, while the input data comprises natural
language questions with descriptive explanations of medical issues.

##  5 Experimental Setup

In this section, we describe the detail of experimental setup including the
hardware setup used for training the models. Furthermore the training time
required for each transformer model is also discussed.

###  5.1 Hardware Setup

For the model training, we utilized a powerful hardware setup consisting of
two NVIDIA V100 GPUs with 32GB of memory each. The GPUs were complemented by
250GB of RAM, ensuring efficient processing and storage of the large-scale
dataset. The training process was conducted on the high-performance computing
system cHARISMa [22], which provided the necessary computational resources for
training deep learning models.

###  5.2 Training Time

The training time for each transformer model varies depending on its
architecture and complexity. Following are the training times observed for
each model:

  * •

SBERT [23] : The SBERT model required approximately 54 hours to complete the
training process. The extensive training time can be attributed to its deep
architecture and complex attention mechanisms.

  * •

LaBSE [24]: The LaBSE model demonstrated faster training times, with the
training process taking approximately 12 hours. The model’s efficient
architecture and advanced pre-training techniques contribute to its reduced
training time.

  * •

RuBERT [25] : Training the RuBERT model took around 13 hours. The model’s
architecture, specifically designed for the Russian language, required
additional time for fine-tuning and convergence.

  * •

BERT [26] : Similar to LaBSE, the BERT model also completed training in
approximately 12 hours. Its widely adopted architecture and availability of
pre-trained weights contribute to the faster training time.

  * •

BART [27]: The BART model, known for its transformer-based sequence-to-
sequence architecture, required a longer training time of 55 hours. The
complexity of the model and the additional training required for the encoder-
decoder structure contributed to the extended duration.

The complete training and evaluation cycle, including cross-validation with a
k-fold value of 3, ranged from 3 days to 12 days, depending on the specific
model. This timeframe accounted for multiple iterations of training,
hyperparameter tuning, and performance evaluation.

The significant training times for certain models underscore the computational
resources and time investment required for training large-scale transformer
models. However, the improved performance achieved by these models justifies
the efforts put into training and fine-tuning them.

In the next section, we present the results of our experiments and evaluate
the performance of each model.

##  6 Experimental Results and Performance Analysis

In this section, we present the analysis of learning outcomes obtained by
several experiments. We trained several models using cross-validation
techniques and evaluated their performance using the F1-score metric.

As depicted on Figure 3, the plot presents the learning curves of the LaBSE,
SBERT, BERT, and BART models. It is evident from the graph that LaBSE
demonstrates remarkable performance superiority compared to the other models.
The learning curve of LaBSE displays significantly higher accuracy and faster
convergence, indicating its exceptional capability to learn from the provided
dataset. However, for the Russian text specifically, the RuBERT model achieves
the highest quality due to its pre-training on a Russian corpus of texts.

Figure 3: Training Curve of Various Models across Folds.

Conversely, the learning curves of SBERT, BERT, and BART models exhibit
relatively lower accuracy and slower convergence, suggesting their relatively
inferior performance in this specific task. The notable contrast in
performance between LaBSE and RuBERT underscores their effectiveness and
underscores their potential as robust models for the given classification
problem.

This can be explained by the fact that LaBSE is good at distinguishing between
entities, this can be seen in the Umap image, which converts sentence
embeddings into a two-dimensional representation. Also, Umap shows that RuBERT
is very similar to other pictures which show rather poor quality, but
considering that this model is well adapted for Russian, after fine-tuning it
starts to show much better quality.

((a)) RuBERT

((b)) LaBSE

((c)) BERT

((a)) BART

((b)) SBERT

Figure 5: Umap image of different models on our dataset

The following models were trained and their corresponding F1-scores are
reported.

Table 2: Performance Comparison of Transformer Models Model | K-fold (F1-score, k = 3) | Split (F1-score, train = 90%)
---|---|---
|  not augmented |  augmented |  not augmented |  augmented
BART |  0.798 |  0.891 |  0.794 |  0.896
BERT |  0.743 |  0.894 |  0.760 |  0.903
LaBSE |  0.824 |  0.911 |  0.833 |  0.913
LogRegression |  0.457 |  0.552 |  0.531 |  0.564
Random Forest |  0.521 |  0.579 |  0.596 |  0.603
RuBERT |  0.839 |  0.918 |  0.852 |  0.918
SBERT |  0.782 |  0.905 |  0.761 |  0.895
SVM |  0.525 |  0.565 |  0.534 |  0.598

Results in table 2 provide insights into the performance of different models
in our experimental setup. The high F1-scores obtained by RuBERT and LaBSE
suggests that they effectively captured the semantic representations and
contextual information in the text data. SBERT and BERT also demonstrated
competitive performance, although slightly lower than RuBERT and LaBSE. BART
exhibited a slightly lower F1-score, indicating that its performance may be
influenced by the specific task and dataset. Overall, the analysis of learning
outcomes highlights the effectiveness of various models in our experiments,
with RuBERT and LaBSE demonstrating particularly promising results. These
findings contribute to our understanding of the strengths and limitations of
different models and can guide future research and practical applications in
the field of NLP.

Table 3: Performance Evaluation of RuBERT for Medical Specialties Classification Category | Precision | Recall | F1-Score | Support
---|---|---|---|---
ENT | 0.7555 | 0.7432 | 0.7493 | 15276
Ophthalmologist | 0.9403 | 0.9210 | 0.9305 | 14936
Pediatric Surgeon | 0.8405 | 0.8782 | 0.8589 | 14847
Gynecologist | 0.7834 | 0.7459 | 0.7642 | 14844
Dentist | 0.8815 | 0.8893 | 0.8854 | 14861
Sexologist-Andrologist | 0.7904 | 0.6955 | 0.7399 | 15148
Therapist | 0.5066 | 0.3738 | 0.4302 | 15080
Surgeon | 0.6705 | 0.5818 | 0.6230 | 14929
Cardiologist | 0.8646 | 0.8567 | 0.8606 | 14836
Psychologist | 0.7759 | 0.7215 | 0.7477 | 15020
Orthopedic Traumatologist | 0.7981 | 0.7683 | 0.7829 | 15081
Pediatrician | 0.6482 | 0.5712 | 0.6073 | 15087
Dermatologist | 0.7111 | 0.6569 | 0.6829 | 14941
Neurosurgeon | 0.8797 | 0.9025 | 0.8910 | 14898
Endocrinologist | 0.8478 | 0.8072 | 0.8270 | 15011
Venereologist | 0.7763 | 0.8112 | 0.7934 | 15140
Urologist | 0.6445 | 0.6240 | 0.6341 | 15110
Neuropathologist | 0.6633 | 0.5834 | 0.6206 | 15058
Medical Doctor | 0.8667 | 0.8824 | 0.8745 | 14959
Infectious Disease Specialist | 0.8409 | 0.7986 | 0.8192 | 14924
Oncologist | 0.8796 | 0.8742 | 0.8769 | 14957
Gastroenterologist | 0.7574 | 0.7339 | 0.7455 | 14839
. . . |  |  |  |
accuracy | 0.9111 | 0.9111 | 0.9111 | 0.9031
macro avg | 0.9177 | 0.9205 | 0.9189 | 1470000
weighted avg | 0.9178 | 0.9201 | 0.9189 | 1470000

Table 3 presents the performance evaluation results of a classification model
for various categories of medical specialists. It includes the metrics
precision ($P$), recall ($R$), F1-score ($F1$), and support for each category.
These metrics provide insights into the model’s ability to correctly classify
instances belonging to different medical specialties.

These evaluation metrics help assess the effectiveness of the classification
model in distinguishing between different medical specialties. The values in
the table represent the performance of the model for each category, allowing
for a comparison of its accuracy and effectiveness across various medical
specialties.

Figure 6: Confusion Matrix

The Confusion Matrix allowed for a detailed exploration of classification
outcomes, delineating true positives, true negatives, false positives, and
false negatives. This analysis unveiled a notable trend: the majority of
errors observed stemmed from the disparities present in the real-world data’s
structure and semantics. This observation can be attributed to the inherent
diversity and complexity of genuine medical texts, where nuances in language
and context can lead to intricate classification challenges.

Interestingly, when the model was evaluated using synthetic data, the
Confusion Matrix demonstrated a contrasting pattern. Synthetic data, crafted
to adhere to specific structures and semantics, presented fewer challenges for
the model’s classification accuracy. This stark difference suggests that the
model might encounter difficulties when confronted with the heterogeneity
inherent in genuine medical text, compared to the more controlled environment
of synthetic data.

The primary limitations observed encompass various aspects. First, the text
length constraint, set at 128 words, significantly affects the model’s ability
to capture intricate nuances present in longer textual data. When exceeding
this threshold, the data becomes represented as sparse vectors, potentially
leading to information loss and diminished performance.

Secondly, the distinctive writing style encountered in the test data, which
differs from that seen in the training data, poses a challenge. The model’s
training on a particular style limits its adaptability to new, previously
unseen writing patterns. This mismatch between training and test data styles
can result in reduced accuracy and nuanced misclassifications.

Moreover, the presence of questions addressing topics that were not covered
extensively in the training data presents another constraint. Models struggle
when faced with questions that delve into unfamiliar territories, as they lack
the contextual familiarity to provide accurate predictions.

In conclusion, the discussed limitations, including text length constraints,
writing style divergence, and unfamiliar thematic areas, collectively
highlight the challenges faced when applying classifiers to such datasets.
These limitations underscore the need for robust pre-training, data
augmentation, and model fine-tuning strategies to enhance the model’s
performance and mitigate the observed shortcomings.

##  7 Conclusion

In this study, we collected a comprehensive dataset for text classification,
augmented it with various techniques, and conducted experiments using five
state-of-the-art transformer models: SBERT, BERT, LaBSE, BART, and RuBERT. We
observed that RuBERT achieved the best performance with f1-score of 91.9%,
outperforming the other models. Based on these findings, we conclude that
transformer models, particularly RuBERT, are highly effective for text
classification tasks. The ability of transformers to capture contextual
information and learn complex patterns in textual data contributes to their
superior performance compared to classical machine learning methods.

Further research can be conducted to explore the performance of these
transformer models on smaller datasets or specific domain-related datasets.
Additionally, there is potential for developing new transformer architectures
tailored specifically for text classification tasks. These architectures can
incorporate domain-specific knowledge and enhance the model’s ability to
extract meaningful features from text, further improving classification
accuracy.

Investigating transfer learning techniques, fine-tuning strategies, and
hyperparameter optimization for these transformer models can also be valuable
directions for future work. The exploration of different augmentation
techniques and their impact on model performance can provide insights into
improving the robustness and generalization capabilities of text
classification models. Overall, there is ample opportunity for advancing the
field of text classification using transformer models, and these future works
can contribute to the development of more accurate and efficient models for
various applications.

## References

  * [1] Song, H., Omori, K., Kim, J., Tenzek, K.E., Hawkins, J., Lin, W., Kim, Y., Jung, J.:  Trusting social media as a source of health information: Online surveys comparing the united states, korea, and hong kong.  Journal of Medical Internet Research 18(3) (2016) e25
  * [2] :  Babyblog - otvety na lyubye voprosy o beremennosti, detyakh i semeynoy zhizni.  https://www.babyblog.ru/ Accessed: December 19 , 2022.
  * [3] Keshavarz, H.:  Evaluating credibility of social media information: current challenges, research directions and practical criteria.  Information Discovery and Delivery 49(4) (2021) 269–279
  * [4] Mao, C., Zhu, Q., Chen, R., Su, W.:  Automatic medical specialty classification based on patients’ description of their symptoms.  BMC Medical Informatics and Decision Making 23 (01 2023)
  * [5] Tezgider, M., Yildiz, B., Aydin, G.:  Text classification using improved bidirectional transformer.  Concurrency and Computation: Practice and Experience 34(9) (2022) e6486
  * [6] :  Sprosivracha: Zadai vopros vrachu online i poluchi otvet mgnovenno.  https://sprosivracha.com/ Accessed: February 17, 2023.
  * [7] :  Doctu - poisk luchshikh vrachey i klinik v rossii.  https://doctu.ru/ Accessed: February 17, 2023.
  * [8] :  03 online — meditsinskie konsultatsii v rezhime online.  https://03online.com/ Accessed: February 17, 2023.
  * [9] :  health.mail.ru - poisk po boleznyam, lekarstvam i otvetam vrachey.  https://health.mail.ru/ Accessed: February 17, 2023.
  * [10] Johnson, J.M., Khoshgoftaar, T.M.:  Survey on deep learning with class imbalance.  Journal of Big Data 6(1) (2019) 27
  * [11] Buslaev, A., Parinov, A., Khvedchenya, E., Iglovikov, V.I., Kalinin, A.A.:  Albumentations: fast and flexible image augmentations.  ArXiv e-prints (2018)
  * [12] HECHT-NIELSEN, R.:  Iii.3 - theory of the backpropagation neural network**based on “nonindent” by robert hecht-nielsen, which appeared in proceedings of the international joint conference on neural networks 1, 593–611, june 1989. © 1989 ieee.  In Wechsler, H., ed.: Neural Networks for Perception.  Academic Press (1992) 65–93
  * [13] Shaheen, Z., Wohlgenannt, G., Filtz, E.:  Large scale legal text classification using transformer models (2020)
  * [14] Zhuang, Z., Liu, M., Cutkosky, A., Orabona, F.:  Understanding adamw through proximal methods and scale-freeness (2022)
  * [15] Kim, C., Kim, S., Kim, J., Lee, D., Kim, S.:  Automated learning rate scheduler for large-batch training (2021)
  * [16] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.:  Attention is all you need (2017)
  * [17] You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., Hsieh, C.J.:  Large batch optimization for deep learning: Training bert in 76 minutes (2020)
  * [18] Bai, Y., Mei, J., Yuille, A.L., Xie, C.:  Are transformers more robust than cnns?  In: Advances in Neural Information Processing Systems. Volume 34., Curran Associates, Inc. (2021) 26831–26843
  * [19] Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P.S., He, L.:  A survey on text classification: From shallow to deep learning (2021)
  * [20] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.:  Transformers: State-of-the-art natural language processing.  In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, Association for Computational Linguistics (October 2020) 38–45
  * [21] Maida, A.:  Cognitive computing and neural networks: Reverse engineering the brain.  In: Handbook of Statistics. Volume 35.  Elsevier (2016) 39–78
  * [22] Kostenetskiy, P.S., Chulkevich, R.A., Kozyrev, V.I.:  Hpc resources of the higher school of economics.  Journal of Physics: Conference Series 1740(1) (jan 2021) 012050
  * [23] Reimers, N., Gurevych, I.:  Sentence-bert: Sentence embeddings using siamese bert-networks (2019)
  * [24] Feng, F., Yang, Y., Cer, D., Arivazhagan, N., Wang, W.:  Language-agnostic bert sentence embedding (2022)
  * [25] Kuratov, Y., Arkhipov, M.:  Adaptation of deep bidirectional multilingual transformers for russian language (2019)
  * [26] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.:  Bert: Pre-training of deep bidirectional transformers for language understanding (2019)
  * [27] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.:  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension (2019)