# Scaling transition from momentum stochastic gradient descent to plain
stochastic gradient descent

Kun Zeng<EMAIL_ADDRESS>Jinlan Liu<EMAIL_ADDRESS>Zhixia Jiang  Dongpo Xu
<EMAIL_ADDRESS>Department of Mathematics, Changchun University of Science
and Technology, Changchun 130022, China  School of Mathematics and Statistics,
Northeast Normal University, Changchun 130024, China

###### Abstract

The plain stochastic gradient descent and momentum stochastic gradient descent
have extremely wide applications in deep learning due to their simple settings
and low computational complexity. The momentum stochastic gradient descent
uses the accumulated gradient as the updated direction of the current
parameters, which has a faster training speed. Because the direction of the
plain stochastic gradient descent has not been corrected by the accumulated
gradient. For the parameters that currently need to be updated, it is the
optimal direction, and its update is more accurate. We combine the advantages
of the momentum stochastic gradient descent with fast training speed and the
plain stochastic gradient descent with high accuracy, and propose a scaling
transition from momentum stochastic gradient descent to plain stochastic
gradient descent(TSGD) method. At the same time, a learning rate that
decreases linearly with the iterations is used instead of a constant learning
rate. The TSGD algorithm has a larger step size in the early stage to speed up
the training, and training with a smaller step size in the later stage can
steadily converge. Our experimental results show that the TSGD algorithm has
faster training speed, higher accuracy and better stability. Our
implementation is available at: https://github.com/kunzeng/TSGD.

###### keywords:

gradient descent, scaling transition, artificial neural network, image
classification

††journal:

##  1 Introduction

In recent years, with the development of deep learning technologies, neural
networks have been extremely widely used[1, 2, 3]. Gradient descent is an
important algorithm for training neural networks. As an unconstrained
optimization method, the stochastic gradient descent is applied in many
fields, especially in neural networks. However, the stochastic gradient
descent has some disadvantages, such as slow training speed and unstable
training. Researchers have proposed many adaptive gradient descent methods and
their variants, such as AdaGrad[4], Adam[5], AmsGrad[6] and so on. However,
the adaptive gradient descent needs to calculate the second moment, which has
high computational complexity. In some aspects, the generalization ability of
adaptive gradient descent is often not as good as stochastic gradient
descent[7]. In practical applications, most neural networks still use
stochastic gradient descent for training, such as ResNet[8], DensNet[9] and so
on.

Plain stochastic gradient descent[10] uses the negative direction of the
gradient of loss function as the direction of parameters update. The
hyperparameter setting is relatively simple and has low computational
complexity. It has higher efficiency for large-scale data training. Polyak et
al.[11] improved the plain stochastic gradient descent and proposed a heavy-
ball momentum stochastic gradient descent. The heavy-ball momentum stochastic
gradient descent uses the accumulated historical gradient as momentum. Then it
uses momentum to update the current parameters. So as to speed up training and
has better stability. Nesterov’s accelerated gradient (NAG)[12] is a
modification of the momentum-based update which uses a look-ahead step to
improve the momentum term[13]. These two momentum gradient descent methods
have a wide range of applications due to their simple settings and good
generalization ability. Subsequently, researchers improved the momentum
stochastic gradient descent, and proposed synthesized Nesterov variants[14],
PID control[15], AccSGD[16], SGDP[17] and so on.

In 2019, Jerry Ma et al.[18] proposed the quasi-hyperbolic momentum algorithm.
For the direction of parameters update, QHM algorithm mainly considers a
linear combination of the last momentum and current gradient, rather than just
using momentum to update the parameters. The more important thing of this
algorithm is a general structure of the momentum stochastic gradient descent.
Through the adjustment of the parameters $\displaystyle v$ and
$\displaystyle\beta$, it can be transformed into the Nesterov’s accelerated
gradient[12], PID control[15], synthesized Nesterov variant[14], least-squares
acceleration of SGD algorithm[16].

We can see that the historical gradient is used in the momentum stochastic
gradient descent to make a correction to the current gradient direction,
thereby speeding up the training. In fact, we can empirically see that
momentum is an exponentially weighted combination of historical gradient and
current gradient. Since the past gradient is not computed based on the most
updated stochastic gradient descent state, it can introduce a deviation to the
new gradient computation, negatively impacting its rate of convergence[19].
Its update direction is no longer the optimal direction for the current
parameters, which affects the accuracy. For the update direction of plain
stochastic gradient descent each time, it is the optimal direction of the
current parameters. Although it may be easy to fall into the local optimum,
the training speed is slow, etc. But for the current parameters, the plain
stochastic gradient descent has a more accurate direction. Therefore, for the
gradient descent, how to get fast training speed and high accuracy is very
important.

In this paper, we combine the fast training speed of the heavy-ball momentum
stochastic gradient descent and the high accuracy of the plain stochastic
gradient descent. By scaling, we achieve a smooth and stable transition from
momentum stochastic gradient descent to plain stochastic gradient descent. In
the early stage of training, we use momentum stochastic gradient descent, and
in the later stage we use plain stochastic gradient descent for training. It
has a faster training speed in the early stage, and a higher accuracy in the
later stage of training. And this transition process is automatic, smooth and
stable. The experimental results show that our proposed algorithm has a faster
training speed and better accuracy in the neural network model.

##  2 Preliminaries

Optimization problem. We consider the following unconstrained convex
optimization problem[20]:

$\displaystyle\mathop{\min}\limits_{\theta\in X}{\rm{}}F\left(\theta\right).$

Where $\displaystyle F$ is a continuous differentiable convex function on
$\displaystyle X$. We use the gradient descent to solve the above problem

Plain stochastic gradient descent. In the plain stochastic gradient descent,
we randomly select the sample and use the current gradient to update the
parameters[21]:

$\displaystyle{\theta_{t+1}}={\theta_{t}}-{\eta}{g_{t}}.$

Where $\displaystyle\theta$ is the parameter that needs to be optimized, and
$\displaystyle{g_{t}}$ is the gradient of the current loss function.
$\displaystyle{\eta}$ is the step size.

Momentum stochastic gradient descent. The momentum stochastic gradient descent
considers the historical gradient information, and uses it to make a
correction to the direction of the current gradient, which speeds up the
training. The update formula of the heavy-ball momentum stochastic gradient
descent is[22]:

$\displaystyle\begin{array}[]{l}m_{t}=\beta m_{t-1}+(1-dampening)g_{t},\\\
\theta_{t+1}=\theta_{t}-\eta_{t}\hat{m}_{t}.\end{array}$

Where $\displaystyle{m_{t}}$ is momentum, $\displaystyle\beta$ is momentum
factor, and $\displaystyle dampening$ is the dampening for
momentum$\displaystyle(dampening=0)$.

Momentum and gradient. In the heavy-ball momentum stochastic gradient descent,
the accumulation of historical gradient is used as the momentum. It uses the
momentum direction to replace the current gradient direction for parameters
update. Although the momentum stochastic gradient descent reduces the
variance, it has an acceleration effect. But from the composition of the
momentum term, we can empirically see that the negative direction of the
gradient of the loss function
$\displaystyle\left({-\nabla{F_{t}}\left({{\theta_{t}}}\right)}\right)$ is the
optimal direction for the current parameters update. If the correction is made
to the direction of the current gradient by using the momentum, the momentum
direction is no longer the optimal direction for the current parameters.

(a) The accuracy on test set

(b) The loss on training set

Figure 1: We use ResNet18 to train cifar10 data set, the first 75 epochs use
momentum stochastic gradient descent(SGDM), and the remaining epochs use plain
stochastic gradient descent(plain SGD) for training

We use the ResNet18 to train the cifar10 data set, and use the SGDM algorithm
training for the first 75 iterations (Hyperparameters settings:
weight_decay=$\displaystyle 5e-4$, $\displaystyle lr$=$\displaystyle 0.1$,
$\displaystyle\beta$=$\displaystyle 0.9$, dampening=0). After the 75th
iteration, we change the momentum to the current gradient (if epoch
$\displaystyle<$ 76: $\displaystyle{m_{t}}$ else: $\displaystyle{g_{t}}$, the
learning rate has not been changed, always $\displaystyle lr$=$\displaystyle
0.1$). Picture (a) of Figure 1 shows the accuracy of the test set during the
training process. It can be seen that after the accuracy increases rapidly
from starting, the SGDM algorithm enters the plateau after about
75-$\displaystyle th$ iteration, and the accuracy no longer increases. After
75-$\displaystyle th$ iteration, the current gradient direction is used to
update the parameters, and it can be seen that the accuracy has been
significantly improved. This also shows from the side that the current
negative gradient direction is more accurate. Picture (b) of Figure 1 shows
the loss value of the training set. It can be seen that the loss of the plain
SGD method is faster and smaller. But for deep learning, a faster training
speed is often very important. Therefore, a good idea is to combine fast speed
and high accuracy for the gradient descent method.

Assumption A. The following conditions hold for $\displaystyle F$ and the
stochastic gradient oracle[23]:

1\. $\displaystyle F$ is differentiable and $\displaystyle\nabla F$ is
Lipschitz continuous, i.e., there is a constant $\displaystyle L$ such that

$\displaystyle\left\|{\nabla F\left(x\right)-\nabla
F\left(y\right)}\right\|\leq L\left\|{x-y}\right\|,x,y\in{R^{n}}.$

2\. $\displaystyle F$ is bounded below and $\displaystyle\left\|{\nabla
F\left(x\right)}\right\|$ is bounded above, i.e., there exist
$\displaystyle{F_{*}}$ and $\displaystyle G$ such that

$\displaystyle F\left(x\right)\geq{F_{*}},\left\|{\nabla
F\left(x\right)}\right\|\leq G,x\in{R^{n}}.$

3\. For $\displaystyle k=0,1,2,\ldots,$ the stochastic gradient
$\displaystyle{g^{k}}=\nabla F\left({{x^{k}}}\right)+{\xi^{k}}$ , where the
random noise $\displaystyle{\xi^{k}}$ satisfies

$\displaystyle{E_{k}}\left[{{\xi^{k}}}\right]=0,{E_{k}}\left[{{{\left\|{{\xi^{k}}}\right\|}^{2}}}\right]\leq
C.$

where $\displaystyle{E_{k}}\left[\cdot\right]$ denotes expectation conditioned
on
$\displaystyle\left\\{{{x^{0}},{g^{0}},\ldots,{x^{k-1}},{g^{k-1}},{x^{k}}}\right\\}$,
and $\displaystyle C$ is a constant.

Lemma 1. Let $\displaystyle F$ satisfy Assumption A. Additionally, assume
$\displaystyle 0\leq{v_{k}}\leq 1$ and the sequences
$\displaystyle\left\\{{{a_{k}}}\right\\}$ and
$\displaystyle\left\\{{{\beta_{k}}}\right\\}$ satisfy the following
conditions[23]:

$\displaystyle\sum\limits_{k=0}^{\infty}{{\alpha_{k}}}=\infty,\sum\limits_{k=0}^{\infty}{\alpha_{k}^{2}}=\infty,\mathop{\lim}\limits_{k\to\infty}{\beta_{k}}=0,\bar{\beta}\buildrel\Delta\over{=}\mathop{\sup}\limits_{k}{\beta_{k}}<1.$

Then the sequence $\displaystyle\left\\{{{x_{k}}}\right\\}$ generated by the
QHM in literature[23]. Moreover, we have:

$\displaystyle\limsup_{k\rightarrow\infty}F\left(x^{k}\right)=\limsup_{k\rightarrow\infty,\left\|\nabla
F\left(x^{k}\right)\right\|\rightarrow 0}F\left(x^{k}\right).$

##  3 TSGD Algorithm

Based on the above analysis and the basis of the QHM algorithm, we combine the
advantages of the momentum stochastic gradient descent with fast training
speed and the plain stochastic gradient descent with high accuracy. The
scaling transition from momentum stochastic gradient descent to plain
stochastic gradient descent method is proposed. At the same time, a learning
rate that decreases linearly with the number of iterations is used instead of
a constant learning rate. It allows the momentum stochastic gradient descent
with a larger step size to be used in the initial stage to speed up the
training, and the plain stochastic gradient descent with a smaller step size
for training in the later stage can steadily converge and avoid oscillation.
The process of this transformation is realized with the scaling by the
iterations, which is automatic, smooth and stable. TSGD algorithm is described
as Algorithm 1.

Algorithm 1: Scaling Transition from Momentum Stochastic Gradient descent to
Plain
---
Stochastic Gradient Descent(TSGD).
Input: $\displaystyle x\in\mathcal{F}$, $\displaystyle UR:$ upper learning
rate, $\displaystyle LR:$ lower learning rate, $\displaystyle\gamma$: scaling
factor, $\displaystyle{\beta}$:
momentum factor, $\displaystyle{\eta_{t}}:$ learning rate, $\displaystyle T:$
epochs, $\displaystyle dampening$: dampening for momentum.
Initialize: $\displaystyle m_{0}=0,t=0$
for $\displaystyle{t}$ to $\displaystyle{T-1}$ do
$\displaystyle t=t+1$
$\displaystyle{g_{t}}=\nabla{f_{t}}({\theta_{t}})$ Update the gradient of
objective function
$\displaystyle{m_{t}}={\beta}{m_{t-1}}+(1-dampening){g_{t}}$ Update first
moment momentum
$\displaystyle\hat{m}_{t}=\left(m_{t}-g_{t}\right)\frac{1}{\gamma^{t}}+g_{t}$
Scaling the momentum and gradient
$\displaystyle\eta_{t}=(UR-LR)(1-t/T)+LR$ Decreasing learning rate
$\displaystyle\theta_{t+1}=\theta_{t}-\eta_{t}\hat{m}_{t}$ Update parameters
end for

Oracle recommends an empirical value of: $\displaystyle\beta=0.9$,
$\displaystyle UR=0.1$, $\displaystyle LR=0.005$$\displaystyle\left({UR\geq
LR}\right)$, $\displaystyle\gamma$$\displaystyle\left({\gamma\geq 1}\right)$.
Please see section 6 for details on the value of $\displaystyle UL$,
$\displaystyle LR$, $\displaystyle\gamma$. $\displaystyle{g_{t}}$: gradient of
loss function $\displaystyle f$ in $\displaystyle t$-th iteration.
$\displaystyle{m_{t}}$: momentum. $\displaystyle{\hat{m}_{t}}$: scaled
momentum. $\displaystyle{\theta_{t}}$: optimized parameters.

In Algorithm 1, we use a linearly decreasing learning rate with the number of
iterations instead of a constant learning rate, give as:

$\displaystyle{\eta_{t}}=\left({UR-LR}\right)(1-t/T)+LR.$

From the above formula, the TSGD algorithm can have a larger step size in the
early stage, which can speed up the training. In the later stage of training,
having a small step size can avoid skipping the optimal solution or
oscillating, and makes training more stable.

For the transition process, we implemente the following formula:

$\displaystyle{\hat{m}_{t}}=\left({{m_{t}}-{g_{t}}}\right)\frac{1}{{{\gamma^{t}}}}+{g_{t}}.$

We rearrange the formula of parameters updated as follows:

$\displaystyle{\theta_{t+1}}={\theta_{t}}-{\eta_{t}}\left[{\frac{\beta}{{{\gamma^{t}}}}{m_{t-1}}+\left({1-\frac{dampening}{{{\gamma^{t}}}}}\right){g_{t}}}\right].$

It can be seen that the direction of the parameters update of the TSGD
algorithm is a linear combination of the last momentum and the current
gradient. But the weight coefficient here decreases as the iteration
increasing. When $\displaystyle t$ is smaller, $\displaystyle 1/{\gamma^{t}}$
is close to 1, the TSGD algorithm is momentum stochastic gradient descent.
When $\displaystyle t$ is larger, $\displaystyle 1/{\gamma^{t}}$ is close to
0, and the TSGD algorithm is plain stochastic gradient descent. As the
iteration $\displaystyle t$ increasing, $\displaystyle 1/{\gamma^{t}}$
achieves a smooth and stable transition from the momentum stochastic gradient
descent to the plain stochastic gradient descent.

The comparison of TSGD and QHM algorithm[18] shows that QHM algorithm is a
special case of TSGD algorithm. In the TSGD algorithm, when
$\displaystyle{\gamma^{t}}$ is $\displaystyle\gamma$, the TSGD algorithm is
the QHM algorithm. In the literature[18], the author analyzes the relationship
between QHM algorithm and Nesterov’s accelerated gradient, PID control[15],
synthesized Nesterov varian $\displaystyle\left(SNV\right)$[14], AccSGD[16]
algorithms in detail. At the same time, in the deterministic (full-batch)
case, QHM can recover the global linear convergence rate of $\displaystyle
1-1/\sqrt{k}$ for strongly convex, smooth loss functions when QHM recovers
Triple Momentum. In the stochastic (minibatch) case, QHM’s recovery of AccSGD
gives QHM the same convergence results as in the literature[16]’s least-
squares regression setting, of $\displaystyle{\rm
O}\left({\sqrt{\kappa}\cdot\log\kappa\cdot\log\frac{1}{\epsilon}}\right)$
iterations for $\displaystyle\epsilon$-approximation of the minimal loss. The
QHM algorithm is a special case of the TSGD algorithm. Obviously, the
relationship and convergence between different algorithms and QHM in the
literature[18] have the same results as the TSGD algorithm. In particular, we
take $\displaystyle\beta_{k}=\beta=dampening$,
$\displaystyle{v_{k}}=1/{\gamma^{k}}$, under the Assumption A, we can get the
same Lemma 1 of TSGD algorithm.

In fact, the idea of combining last momentum stochastic gradient and plain
stochastic gradient is not the first time to appear in this paper. For
example, the QHM[18] algorithm uses the moving weighted average last momentum
and current gradient as the current update direction. The main formula is:

$\displaystyle\begin{array}[]{l}g_{t+1}\leftarrow\beta\cdot
g_{t}+(1-\beta)\cdot\nabla\hat{L}_{t}\left(\theta_{t}\right),\\\
\theta_{t+1}\leftarrow\theta_{t}-\alpha\left[(1-\nu)\cdot\nabla\hat{L}_{t}\left(\theta_{t}\right)+\nu\cdot
g_{t+1}\right].\end{array}$

The author gives an empirical value of $\displaystyle v=0.7$ in the paper. In
the literature[24], the author transforms gradients computed at previous
iterates into gradients evaluated at the current iterate without using the
Hessian explicitly. The momentum factor is taken as a function with the
iterations, and its main formula is:

$\displaystyle\begin{array}[]{l}\gamma_{t}\leftarrow\frac{t}{t+1},\\\
v_{t}\leftarrow\gamma_{t}v_{t-1}+\left(1-\gamma_{t}\right)g\left(\theta_{t}+\frac{\gamma_{t}}{1-\gamma_{t}}\left(\theta_{t}-\theta_{t-1}\right),x_{t}\right).\end{array}$

From the above, we can see that although many researchers have explored the
combination of momentum stochastic gradient descent and plain stochastic
gradient descent, and many achievements have been made. But according to the
best of our knowledge, the transition from momentum stochastic gradient
descent to plain stochastic gradient descent by scaling with the iterations is
proposed in this paper for the first time. In our proposed TSGD algorithm, we
use a linearly decreasing learning rate instead of the constant learning rate,
so that the algorithm has a larger step size in the early stage to speed up,
and a smaller step size in the later stage of training can be stable
convergence. At the same time, we achieved a smooth and stable transition from
momentum stochastic gradient descent to stochastic gradient descent through
scaling. It combines the advantages of the momentum stochastic gradient
descent with fast training speed and the plain stochastic gradient descent
with high accuracy. Making the TSGD algorithm has a faster convergence speed
and also a higher accuracy.

##  4 TAdam Algorithm

Plain stochastic gradient descent, due to the slow convergence speed and
unstable training. It often needs to manually adjust the learning rate through
the interval learning rate (StepLR). In 2016, Duchi et al.[11] proposed the
adaptive gradient descent AdaGrad using the second moment as the adaptive
learning rate. The adaptive gradient descent has a faster training speed and
the training process is more stable. It is more suitable for different
scenarios, and does not need or very few parameters need to be manually
adjusted. Then researchers improved it, and Adam[5] algorithm has good
performance as an excellent representative of adaptive gradient descent. It
also has a wide range of applications in the field of deep learning. We use
the scaled momentum term in TSGD to replace the momentum term in Adam’s
algorithm. As a result, the descent direction in Adam’s algorithm transform
from the momentum stochastic gradient descent direction to the plain
stochastic gradient descent direction with the iterations. We name it TAdam
algorithm, such as Algorithm 2.

Algorithm 2: Adam with Scaling Transition from Momentum stochastic gradient
---
descent to Plain Stochastic Gradient Descent(TAdam).
Input: $\displaystyle x\in\mathcal{F}$, $\displaystyle\gamma$: scaling
factor($\displaystyle\gamma\geq 1$,see section 6 for details),
$\displaystyle{\beta_{1}},{\beta_{2}}:$ constants, $\displaystyle{\eta}:$
global learning rate, $\displaystyle T:$ epochs, $\displaystyle\varepsilon$:
regularization constant.
Initialize: $\displaystyle m_{0}=0,v_{0}=0,t=0$
for $\displaystyle{t}$ to $\displaystyle{T-1}$ do
$\displaystyle t=t+1$
$\displaystyle{g_{t}}=\nabla{f_{t}}({\theta_{t}})$ Update the gradient of
objective function
$\displaystyle{m_{t}}={\beta_{1}}{m_{t-1}}+(1-{\beta_{1}}){g_{t}}$ Update
first moment
$\displaystyle\hat{m}_{t}=\left(m_{t}-g_{t}\right)\frac{1}{\gamma^{t}}+g_{t}$
Scaling the momentum and gradient
$\displaystyle{v_{t}}={\beta_{2}}{v_{t-1}}+(1-{\beta_{2}})g_{t}^{2}$ Update
second moment
$\displaystyle{\theta_{t+1}}={\theta_{t}}-\eta\frac{{{{\hat{m}}_{t}}}}{{\sqrt{{v_{t}}}+\varepsilon}}$
Update parameters
end for

The authors of proposed Adam algorithm gave some empirical values
$\displaystyle{\beta_{1}}=0.9$, $\displaystyle{\beta_{2}}=0.999$,
$\displaystyle\eta=0.001$. $\displaystyle{f_{t}}\left({{\theta_{t}}}\right)$:
loss function, $\displaystyle{g_{t}}$: gradient of loss function,
$\displaystyle{m_{t}}$: momentum, $\displaystyle{\hat{m}_{t}}$: scaled
momentum, $\displaystyle{v_{t}}$: adaptive learning rate,
$\displaystyle{\theta_{t}}$: optimized parameters. From Algorithm 2, it can be
seen that when $\displaystyle\gamma=1$, Algorithm 2 is Adam algorithm, when
$\displaystyle\gamma>1$, Algorithm 2 is TAdam algorithm.

##  5 Experiments

In this section, in order to verify the performance of the TSGD algorithm, we
select different data sets, different neural network models and different
training algorithms for comparison. The experimental environment is: CPU:
Intel(R) Core(TM) i7-6500U CPU @ 2.5GHz 2.60GHz, 16 cores; memory: 32GB;
operating system: Centos8.3; programming language: Python3.7 (Aanaconda3);
deep learning framework: Pytorch1.7.0; graphics card: Quadro P600, CUDA 11.0,
cuDNN v8.0.4; graphics card memory: 2GB. In order to ensure fairness, we set
the same random number seed through pytorch, so that it is not affected by
other factors except the training algorithm.

MNIST and FCNN. MNIST[25] is a classic data set for testing model performance
in the field of deep learning. This data set consists of 0-9 handwritten
digits, with a total of 70,000 pictures, of which 60,000 are the training
samples and 10,000 are the test samples. We select the classic fully connected
neural network(FCNN) for training. The structure of FCNN is: 2 hidden layers,
each hidden layer has 512 neurons. The loss function is the cross-entropy
function. When the batch is divisible by 100, we record a result of the
experiment. The hyperparameters of the TSGD algorithm is set to:
($\displaystyle UR=0.1$, $\displaystyle LR=0.005$, $\displaystyle\beta=0.9$,
$\displaystyle\gamma=1.000767$, epoch $\displaystyle=20$, dampening =
$\displaystyle 0$, batchSize $\displaystyle=100$, weight-decay =
5$\displaystyle e$-4, seed = 1).

We chose SGDM, AdaGrad, Adam, AdaBound algorithms to compare with the proposed
TSGD in this paper. Figure 2 shows the accuracy on the test set. It can be
seen that the TSGD algorithm has a better performance on the test set, more
stable, higher accuracy, faster speed. Figure 3 shows the loss on the training
set. It can be seen that the loss of each algorithm has larger oscillation,
but compared with SGDM, it can be seen that the TSGD algorithm has smaller
loss, and the oscillation is also relatively small. We use the optimal value
of the training process as the result of this experiment, and the results are
shown in Table 1.

Figure 2: The accuracy of MNIST-FCNN on test set

Figure 3: The loss of MNIST-FCNN on training set

Table 1: The results of FCNN training MNIST without StepLR Optimizer/Result | NSGD | Adam | AmsGrad | AdaBound | TSGD
---|---|---|---|---|---
Test Accuracy | $\displaystyle 0.9813$ | $\displaystyle 0.9813$ | $\displaystyle 0.9811$ | $\displaystyle 0.9797$ | $\displaystyle 0.9829$
Training Accuracy | $\displaystyle 1.0000$ | $\displaystyle 1.0000$ | $\displaystyle 1.0000$ | $\displaystyle 1.0000$ | $\displaystyle 1.0000$
Training Loss | $\displaystyle 0.0105$ | $\displaystyle 0.0144$ | $\displaystyle 0.0129$ | $\displaystyle 0.0126$ | $\displaystyle 0.0098$

Cifar10 and ResNet18. The Cifar10 data set[26] was collected by Hinton et al.
It contains 60,000 color images in 10 categories, such as airplanes, cars, and
birds. There are 50,000 training samples and 10,000 test samples . We use a
more complex ResNet18 network. In this experiment, for the TSGD algorithm, we
do not use the decreasing learning rate, and the learning rate is taken as a
constant 0.05. For the hyperparameters of other algorithms, we use default
values or empirical values, and we also do not use the interval of learning
rate adjustment strategy(StepLR). The hyperparameter setting of the TSGD
algorithm is: ($\displaystyle UR=0.05$, $\displaystyle LR=0.05$,
$\displaystyle\beta=0.9$, $\displaystyle\gamma=1.000153$, epoch
$\displaystyle=200$, dampening = $\displaystyle 0$, batchSize
$\displaystyle=128$, weight-decay = 5$\displaystyle e$-4, seed = 1).

Figure 4 shows the accuracy of different algorithms on the test set. It can be
seen from Figure 4, compared to the plain stochastic gradient descent, the
accuracy, convergence speed, and stability of the TSGD algorithm have greatly
improved. Compared with the adaptive gradient descent, the TSGD algorithm has
a faster speed in the early stage of training, and even exceeds the
convergence speed of the adaptive gradient descent. In the later stage of
training, the accuracy has been greatly improved compared with other
algorithms. Figure 5 shows the accuracy of different algorithms on the
training set. The TSGD algorithm has the fastest convergence and the best
accuracy. Figure 6 shows the loss of different algorithms on the training set.
It can be seen that the loss of TSGD algorithm drops faster and the final loss
is the smallest. We use the optimal value in the iterative process as the
result of this experiments. The results are shown in Table 2.

Figure 4: The accuracy of Cifar10-ResNet18 on test set without StepLR

Figure 5: The accuracy of Cifar10-ResNet18 on training set without StepLR

Table 2: The results of ResNet18 training Cifar 10 without StepLR Optimizer/Result | NSGD | Adam | AmsGrad | AdaBound | TSGD
---|---|---|---|---|---
Test Accuracy | $\displaystyle 0.8802$ | $\displaystyle 0.9039$ | $\displaystyle 0.9070$ | $\displaystyle 0.9215$ | $\displaystyle 0.9277$
Training Accuracy | $\displaystyle 0.9004$ | $\displaystyle 0.9392$ | $\displaystyle 0.9476$ | $\displaystyle 0.9856$ | $\displaystyle 0.9980$
Training Loss | $\displaystyle 0.2961$ | $\displaystyle 0.1776$ | $\displaystyle 0.1534$ | $\displaystyle 0.0447$ | $\displaystyle 0.0218$

Under the above experimental conditions, we use a decreasing learning rate for
the TSGD algorithm. For other algorithms, we use the interval of learning rate
adjustment strategy(StepLR), and divide the learning rate by 10 after the
150-$\displaystyle th$ iteration. Figure 7 shows the accuracy of different
algorithms on the test set. It can be seen that when the learning rate is
divide by 10 , the accuracy of the other algorithms has risen sharply. The
Adabound algorithm has better result than other algorithms. However, the TSGD
algorithm has better performance, and its convergence speed even exceeds the
adaptive gradient descent in the early stage of training. The highest accuracy
is achieved in the later stage of training. We use the optimal value of the
training process as the result of this experiment, and the results are shown
in Table 3.

Figure 6: The loss of Cifar10-ResNet18 on training set without StepLR

Figure 7: The accuracy of Cifar10-ResNet18 on test set with StepLR

Table 3: The results of ResNet18 training Cifar 10 with StepLR Optimizer/Result | NSGD | Adam | AmsGrad | AdaBound | TSGD
---|---|---|---|---|---
Test Accuracy | $\displaystyle 0.9456$ | $\displaystyle 0.9311$ | $\displaystyle 0.9384$ | $\displaystyle 0.9466$ | $\displaystyle 0.9514$
Training Accuracy | $\displaystyle 0.9916$ | $\displaystyle 0.9936$ | $\displaystyle 0.9976$ | $\displaystyle 0.9999$ | $\displaystyle 1.0000$
Training Loss | $\displaystyle 0.0270$ | $\displaystyle 0.0206$ | $\displaystyle 0.0116$ | $\displaystyle 0.0014$ | $\displaystyle 0.0015$

Similarly, we do the experiment on the cifar100 data set to compare different
algorithms. Cifar100 is similar to Cifar10, but Cifar100 has 10 categories.
Each category has 10 subcategories and total of 100 categories. Each category
contains 600 pictures. The 500 pictures are used to train and 100 pictures are
used to test. The neural network model we still use ResNet18. In this
experiment, for the TSGD algorithm, we do not use the decreasing learning
rate, and the learning rate is taken as a constant 0.05. For other algorithms,
we also do not use the interval of learning rate adjustment strategy(StepLR).
And the hyperparameters, we use default values or empirical values. The
hyperparameter setting of the TSGD algorithm is: ($\displaystyle UR=0.05$,
$\displaystyle LR=0.05$, $\displaystyle\beta=0.9$,
$\displaystyle\gamma=1.000153$, epoch $\displaystyle=200$, dampening =
$\displaystyle 0$, batchSize $\displaystyle=128$, weight-decay =
5$\displaystyle e$-4, seed = 1).

Figure 8 shows the accuracy of different algorithms on the test set. It can be
seen that TSGD algorithm has faster speed in the early stage of training.
There are two sudden changes in accuracy curve, which may fall into the local
optimum. But with learning rate decreasing, the accuracy gradually rises, and
finally stabilizes. Figure 9 shows the accuracy of the training set. It can be
seen that the TSGD algorithm on the training set has fast speed and high
accuracy. Figure 10 shows the loss on the training set. It can be seen that
the loss of the TSGD algorithm drops faster, and finally has a smaller loss.
We use the optimal value of the training process as the result of this
experiment, and the results are shown in Table 4.

Figure 8: The loss of Cifar100-ResNet18 on training set without StepLR

Figure 9: The accuracy of Cifar100-ResNet18 on test set without StepLR

Table 4: The results of ResNet18 training Cifar100 witout StepLR Optimizer/Result | NSGD | Adam | AmsGrad | AdaBound | TSGD
---|---|---|---|---|---
Test Accuracy | $\displaystyle 0.6286$ | $\displaystyle 0.6762$ | $\displaystyle 0.6862$ | $\displaystyle 0.6995$ | $\displaystyle 0.7604$
Training Accuracy | $\displaystyle 0.7283$ | $\displaystyle 0.8203$ | $\displaystyle 0.8412$ | $\displaystyle 0.9790$ | $\displaystyle 0.9995$
Training Loss | $\displaystyle 0.9218$ | $\displaystyle 0.5840$ | $\displaystyle 0.5243$ | $\displaystyle 0.0817$ | $\displaystyle 0.0075$

Under the above experimental conditions, we use a decreasing learning rate for
the TSGD algorithm, and for other algorithms we use the interval of learning
rate adjustment strategy(StepLR). we divide the learning rate by 10 after the
150-$\displaystyle th$ iteration. Figure 11 shows the accuracy of different
algorithms on the test set. It can be seen that the final accuracy of the TSGD
algorithm that uses a decreasing learning rate is highest. We use the optimal
value of the training process as the result of this experiment, and the
results are shown in Table 5.

Figure 10: The loss of Cifar100-ResNet18 on training set without StepLR

Figure 11: The accuracy of Cifar100-ResNet18 on test set with StepLR

Table 5: The results of ResNet18 training Cifar100 with StepLR Optimizer/Result | NSGD | Adam | AmsGrad | AdaBound | TSGD
---|---|---|---|---|---
Test Accuracy | $\displaystyle 0.7669$ | $\displaystyle 0.7333$ | $\displaystyle 0.7428$ | $\displaystyle 0.7548$ | $\displaystyle 0.7763$
Training Accuracy | $\displaystyle 0.9942$ | $\displaystyle 0.9833$ | $\displaystyle 0.9906$ | $\displaystyle 0.9998$ | $\displaystyle 0.9998$
Training Loss | $\displaystyle 0.0391$ | $\displaystyle 0.0743$ | $\displaystyle 0.0199$ | $\displaystyle 0.0052$ | $\displaystyle 0.0064$

In this section, we use experiments to verify the performance of the TSGD
algorithm proposed in this paper. From the experiments, it can be seen that
TSGD achieves the best results in terms of convergence speed and accuracy.
Compared with the plain stochastic gradient descent, the convergence speed,
stability, and accuracy of TSGD algorithm have been greatly improved. Compared
with the adaptive gradient descent, TSGD does not need to calculate the second
moment, reducing the computational complexity. In particular, the convergence
speed even exceeds the convergence speed of the adaptive gradient descent. In
general, the TSGD algorithm has better performance.

##  6 Extensions

Sensitivity of hyperparameters. In this section, we verify the sensitivity of
the hyperparameters UR and LR of the TSGD algorithm. We select different upper
and lower learning rate for testing. The data set is cifar10, and the neural
network model is ResNet18. The hyperparameter setting of the TSGD algorithm
is: ($\displaystyle UR=0.1$, $\displaystyle LR=0.005$,
$\displaystyle\beta=0.9$, $\displaystyle\gamma=1.000153$, epoch
$\displaystyle=200$, dampening = $\displaystyle 0$, batchSize
$\displaystyle=128$, weight-decay = 5$\displaystyle e$-4, seed = 1).

When testing different upper learning rate $\displaystyle UR$, we fix the
lower learning rate $\displaystyle LR=0.005$. When testing the lower learning
rate $\displaystyle LR$, we fix the upper learning rate $\displaystyle
UR=0.1$. Figure 12 shows the accuracy of different upper learning rate on the
test set. Figure 13 shows the accuracy of different lower learning rate on the
test set. It can be seen that the TSGD algorithm does not have a very strong
dependence on the upper and lower learning rate. The final accuracy also has a
smaller gap. For this training model, the upper learning rate $\displaystyle
UR=0.1$ and lower learning rate $\displaystyle LR=0.005$ of the TSGD algorithm
has a better convergence result, which are two very good recommendations.

Figure 12: The accuracy of Cifar10-ResNet18 on test set for different
$\displaystyle UR$

Figure 13: The accuracy of Cifar10-ResNet18 on test set for different
$\displaystyle LR$

Selection of hyperparameter $\displaystyle\gamma$. In the TSGD algorithm,
compared to the heavy-ball momentum gradient descent algorithm, we introduce
more hyperparameters $\displaystyle UR$, $\displaystyle LR$,
$\displaystyle\gamma$. In the above experiment, we can see that the TSGD
algorithm does not have a very strong dependence on the upper and lower
learning rate. Therefore, here we only discuss the choice of
$\displaystyle\gamma$. $\displaystyle\gamma$ as a conversion factor, it
controls the moment of transition from the momentum gradient descent to the
plain stochastic gradient descent. The main formula for the transition is:
$\displaystyle{\hat{m}_{t}}=\left({{m_{t}}-{g_{t}}}\right)1/{\gamma^{t}}+{g_{t}}$.
When the magnitude of $\displaystyle 1/{\gamma^{t}}$ decreases by
$\displaystyle k={10^{-2}}$( the magnitude of the difference between
$\displaystyle{m_{t}}$ and $\displaystyle{g_{t}}$ is $\displaystyle k$), we
think that the transition has been achieved. During the experiment,
$\displaystyle T$ denotes the total number of iterations, $\displaystyle
sampleSize$ denotes the total number of training samples, $\displaystyle
batchSize$ denotes the size of each batch. According to the above variables,
we can calculate the specific value of $\displaystyle\gamma$ at a certain
transition point. For example: we choose the first 3/8 of the total number of
iterations $\displaystyle T$ for momentum gradient descent training, and the
last 5/8 for plain stochastic gradient descent training. The total number of
samples is $\displaystyle sampleSize=50000$, the batch size is $\displaystyle
batchSize=128$, the epochs is $\displaystyle epoch=200$, and the magnitude is
$\displaystyle k=0.01$ (in fact, through the solving process of
$\displaystyle\gamma$, we can know that the value of $\displaystyle k$ has
little effect on $\displaystyle\gamma$). Then we have $\displaystyle
T=sampleSize/batchSize*epoch$, and take $\displaystyle T=80000$. Since when
the magnitude of $\displaystyle 1/{\gamma^{t}}$ is reduced by $\displaystyle
k=0.01$, we think that a transition has been achieved. so there is
$\displaystyle k=1/{\gamma^{T3/8}}$, and $\displaystyle\gamma=1.000153$.
Obviously, the selection of hyperparameters $\displaystyle\gamma$ is expected
and controllable for the experimental results, and does not require empirical
selection or repeated experimental selection.

Figure 14: The accuracy of Cifar10-ResNet18 on test set for different
$\displaystyle\gamma$

Figure 15: The accuracy of Cifar10-ResNet18 on training set for different
$\displaystyle\gamma$

Figure 16: The loss of Cifar10-ResNet18 on training set for different
$\displaystyle\gamma$

Figure 17: The accuracy of Cifar100-ResNet18 on test set for different
$\displaystyle\gamma$

Table 6: The values of $\displaystyle\gamma$ with respect to different transition points Name/No | 1 | 2 | 3 | 4 | 5 | 6 |
---|---|---|---|---|---|---|---
$\displaystyle\mathrm{t}$ | 10000 | 20000 | 30000 | 40000 | 50000 | 60000 |
$\displaystyle\mathrm{t}/\mathrm{T}$ | $\displaystyle 1/8$ | $\displaystyle 1/4$ | $\displaystyle 3/8$ | $\displaystyle 1/2$ | $\displaystyle 5/8$ | $\displaystyle 3/4$ |
$\displaystyle\gamma$ | $\displaystyle 1.000460$ | $\displaystyle 1.000230$ | $\displaystyle 1.000153$ | $\displaystyle 1.000115$ | $\displaystyle 1.000092$ | $\displaystyle 1.000076$ |

We use ResNet18 to train the Cifar10 data set. In Figures 14, 15, and 16, it
shows the accuracy of the test set and training set, and the loss value of the
training set for different values of $\displaystyle\gamma$. We use ResNet18 to
train the Cifar100 data set. In Figure 17, it shows the accuracy of different
values of $\displaystyle\gamma$ on the test set. For larger values of
$\displaystyle\gamma$, the convergence effect of TSGD is not so ideal. It can
be seen from Figures 14 and 17, that the accuracy of the SGDM algorithm is no
longer increased when the SGDM algorithm is applied to the plateau about 50
epochs. Therefore, empirical selection of transition points after the plateau
usually has a better result. For the remaining values of
$\displaystyle\gamma$, the TSGD algorithm can converge very well, and the
final gap is also small. For the accuracy and loss value of training set, the
final gap is relatively small, and the final curves almost overlap. In Table
6, the value of $\displaystyle\gamma$ corresponding to different transition
points are given.

##  7 Conclusion

In this paper, we combine the advantages of the momentum stochastic gradient
descent with fast training speed and the plain stochastic gradient descent
with high accuracy, and propose the scaling transition from momentum
stochastic gradient descent to plain stochastic gradient descent method. At
the same time, we use a decreasing learning rate, eliminating the need for
complicated procedures of the interval of learning rate adjustment(StepLR).
They make TSGD algorithm has faster convergence speed and higher accuracy. The
experimental results show that the TSGD algorithm has good performance in
terms of convergence speed and generalization ability.

## Acknowledgements

This work was funded in part by National Key R&D Program of China
(No.2020YFA0714102), and in part by the Fundamental Research Funds for the
Central Universities of China (No.2412020F Z024).

## References

  * Ma and Zhang [2021] L. Ma, Y. Zhang,  Research on vehicle license plate recognition technology based on deep convolutional neural networks,  Microprocessors and Microsystems 82 (2021) 103932.
  * Wang et al. [2021] T. Wang, Y. Lei, Y. Fu, J. F. Wynne, W. J. Curran, T. Liu, X. Yang,  A review on medical imaging synthesis using deep learning and its clinical applications,  Journal of Applied Clinical Medical Physics 22 (2021) 11–36.
  * Farooq et al. [2021] U. Farooq, M. S. M. Rahim, N. Sabir, A. Hussain, A. Abid,  Advances in machine translation for sign language: approaches, limitations, and challenges,  Neural Computing and Applications (2021) 1–43.
  * Duchi et al. [2011] J. Duchi, E. Hazan, Y. Singer,  Adaptive subgradient methods for online learning and stochastic optimization.,  Journal of machine learning research 12 (2011).
  * Kingma and Ba [2014] D. P. Kingma, J. Ba,  Adam: A method for stochastic optimization,  arXiv preprint arXiv:1412.6980 (2014).
  * Reddi et al. [2019] S. J. Reddi, S. Kale, S. Kumar,  On the convergence of adam and beyond,  arXiv preprint arXiv:1904.09237 (2019).
  * Wilson et al. [2017] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, B. Recht,  The marginal value of adaptive gradient methods in machine learning,  arXiv preprint arXiv:1705.08292 (2017).
  * He et al. [2016] K. He, X. Zhang, S. Ren, J. Sun,  Deep residual learning for image recognition,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.
  * Huang et al. [2017] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger,  Densely connected convolutional networks,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708.
  * Robbins and Monro [1951] H. Robbins, S. Monro,  A stochastic approximation method,  The annals of mathematical statistics (1951) 400–407.
  * Polyak [1964] B. T. Polyak,  Some methods of speeding up the convergence of iteration methods,  Ussr computational mathematics and mathematical physics 4 (1964) 1–17.
  * Arnold et al. [2019] S. Arnold, P. A. Manzagol, R. Babanezhad, I. Mitliagkas, N. L. Roux,  Reducing the variance in online optimization by transporting past gradients (2019).
  * Cun et al. [1990] Y. L. Cun, B. Boser, J. S. Denker, D. Henderson, L. D. Jackel,  Handwritten digit recognition with a back-propagation network,  Advances in neural information processing systems 2 (1990) 396–404.
  * Lessard et al. [2016] L. Lessard, B. Recht, A. Packard,  Analysis and design of optimization algorithms via integral quadratic constraints,  SIAM Journal on Optimization 26 (2016) 57–95.
  * Recht [2018] B. Recht, The best things in life are model free,argmin (personal blog), [EB/OL], 2018\. http://www.argmin.net/2018/04/19/pid/. Accessed June 6, 2021.
  * Kidambi et al. [2018] R. Kidambi, P. Netrapalli, P. Jain, S. Kakade,  On the insufficiency of existing momentum schemes for stochastic optimization,  in: 2018 Information Theory and Applications Workshop (ITA), IEEE, pp. 1–9.
  * Heo et al. [2021] B. Heo, S. Chun, S. J. Oh, D. Han, S. Yun, G. Kim, Y. Uh, J.-W. Ha,  Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights,  in: Proceedings of the International Conference on Learning Representations (ICLR), Online, pp. 3–7.
  * Ma and Yarats [2018] J. Ma, D. Yarats,  Quasi-hyperbolic momentum and adam for deep learning,  arXiv preprint arXiv:1810.06801 (2018).
  * Song et al. [2021] C. Song, A. Pons, K. Yen,  Ag-sgd: Angle-based stochastic gradient descent,  IEEE Access 9 (2021) 23007–23024.
  * Savarese [2019] P. Savarese,  On the convergence of adabound and its connection to sgd,  arXiv preprint arXiv:1908.04457 (2019).
  * Loizou and Richtárik [2020] N. Loizou, P. Richtárik,  Momentum and stochastic momentum for stochastic gradient, newton, proximal point and subspace descent methods,  Computational Optimization and Applications 77 (2020) 653–710.
  * Ghadimi et al. [2015] E. Ghadimi, H. R. Feyzmahdavian, M. Johansson,  Global convergence of the heavy-ball method for convex optimization,  in: 2015 European control conference (ECC), IEEE, pp. 310–315.
  * Gitman et al. [2019] I. Gitman, H. Lang, P. Zhang, L. Xiao,  Understanding the role of momentum in stochastic gradient methods,  arXiv preprint arXiv:1910.13962 (2019).
  * Arnold et al. [2019] S. M. Arnold, P.-A. Manzagol, R. Babanezhad, I. Mitliagkas, N. L. Roux,  Reducing the variance in online optimization by transporting past gradients,  arXiv preprint arXiv:1906.03532 (2019).
  * LeCun et al. [1998] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner,  Gradient-based learning applied to document recognition,  Proceedings of the IEEE 86 (1998) 2278–2324.
  * Krizhevsky et al. [2009] A. Krizhevsky, G. Hinton, et al.,  Learning multiple layers of features from tiny images (2009).