# A Two-Speed Actuator for Robotics with Fast Seamless Gear Shifting

Alexandre Girard1 and H. Harry Asada2 This work was supported by The Boeing
Company. 1 A. Girard is with the Department of Mechanical Engineering,
Universite de Sherbrooke, Qc, Canada.2 H. H. Asada is with Department of
Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA,
USA.3 ©IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.
DOI:10.1109/IROS.2015.7354047

###### Abstract

This paper present a novel dual-speed actuator adapted to robotics. In many
applications, robots have to bear large loads while moving slowly and also
have to move quickly through the air with almost no load. This lead to
conflicting requirements for their actuators. Multiple gear ratios address
this issue by allowing an effective use of power over a wide range of torque-
speed load conditions. Furthermore, very different gear ratios also lead to
drastic changes of the intrinsic impedance, enabling a non-back-drivable mode
for stiff position control and a back-drivable mode for force control. The
proposed actuator consists of two electric motors coupled to a differential;
one has a large gear ratio while the other is almost direct-drive and equipped
with a brake. During the high-force mode the brake is locked, only one motor
is used, and the actuator behaves like a regular highly-geared servo-motor.
During the high-speed mode the brake is open, both motor are used at the same
time, and the actuator behaves like a direct drive motor. A dynamic model is
developed and novel controllers are proposed for synergic use of both motors.
The redundancy of motors is exploited for maintaining full control of the
output during mode transitions, allowing for fast and seamless switching even
when interacting with unknown environments. Results are demonstrated with a
proof-of-concept linear actuator.

##  I INTRODUCTION

In many robotic systems, including legged robots and wearable robots,
actuators are often required to operate in distinctively different torque-
speed load conditions. A legged robot, for example, has to move its leg
forward quickly through the air and, once touching the ground, it has to bear
a large load [1]. These two operating conditions, high speed at low torque vs.
high torque at low speed, are often an order of magnitude different, while the
required output power is similarly low. When the torque-speed load conditions
do not vary significantly, a single gear ratio can be picked so that the
actuator always operate under nearly optimal conditions. For distinctively
different torque-speed conditions, the actuator will be far for its optimal
operating conditions with a gear ratio picked from the middle ground. As
illustrated on Fig. 1 for a typical electromagnetic (EM) actuator, extremum
torque-speed conditions are not optimal in term of efficiency and power
output. This often leads to the use of oversized and inefficient actuators,
which is inhibitory particularly for mobile robots.

Figure 1: Limitations of EM motors for extremum torque-speed operations

Automobiles with internal combustion (IC) engines use transmissions with
multiple gear ratios to match torque-speed conditions. IC engines have a very
narrow speed range in which they can effectively deliver power; a transmission
with multiple gear ratios is a necessity for the engine to work effectively
for a wide range of output speed. EM motors are more flexible than IC engines,
but still far from ideal sources. EM motors cannot output high power at low
speed because of thermal dissipation and magnetic flux limits related to
material properties; are limited in speed by the supply tension and others;
and are very inefficient when producing large forces at low speed [2]. In
robotic, since it is often the extremums, i.e. maximum torque and speed, that
determine the actuator design instead of the power requirement, much can be
gained with multiple gear ratios.

It will be a significant breakthrough if a type of multiple speed transmission
can be used effectively in robotics. The use of multiple reduction ratios
would enable actuators to meet diverse speed-torque load conditions. Even a
small, lightweight actuator can generate large torques and move at high speed
if equipped with both large and small gear ratio. Moreover, multiple speed
transmission can allow an actuator to work closer to its optimal operating
conditions, improving overall efficiency significantly. Furthermore, gear
shifting significantly changes the intrinsic impedance of an actuator. Since
the impedance is proportional to the square of gear ratio, the impedance may
vary in a range that is several order-of-magnitude different. The actuator may
be made back-drivable while using its small reduction ratio, an important
property in many applications where the robot physically interacts with the
environment [3]. Also the same actuator may be made non-back-drivable while
using its large reduction ratio, allowing the actuator to support loads
without consuming energy and enabling high-stiffness position control.

Despite the desirable features, gear shifting is more technically challenging
in robotics applications than in vehicle applications. For powertrains, the
load is mostly a large inertia, while for robots, the loads may exhibit a rich
range of dynamics including spring-like and damper-like loads. Hence, unlike
vehicle applications, leaving the load free momentarily during transitions
(from one gear ratio to another) is not acceptable in the context of robotics;
the actuator must be engaged with the load at all time to maintain control of
the output. Hence, an effective gear shifting methodology is necessary for
seamless transitions under diverse load conditions.

This paper presents a novel two-speed actuator that meets the requirements of
the robotics context. The system consist of a highly geared EM motor and a
direct drive EM motor equipped with a locking brake, both engaged with a
3-port gearbox. In high-force mode the actuator is non-back-drivable and can
produce large torque at low speed, while in high-speed mode the actuator is
back-drivable and capable of force control and high-speed motion. Moreover,
the two motors are coordinated to fully control the output load, while
performing fast and seamless mode transitions. In the following, related works
are briefly reviewed (section II), then the principle of the proposed two-
speed actuator is presented (section III). Next, a dynamic model is derived
(section IV) and control algorithms for fast seamless transition and
synergistic coordination of the dual motors will be discussed (section V). The
results are demonstrated experimentally with a proof-of-concept linear
actuator (section VI).

##  II Related Works

Traditional robots generally use actuators that behave as displacement-sources
because of their high intrinsic impedance. These include geared EM motors and
hydraulics cylinders. Using a force sensor, it is possible to control the
output force using those actuators, but the bandwidth is rather limited. To
guarantee the stability of the force-feedback scheme only half the intrinsic
inertia can be canceled [3]. Since 70’s, roboticists have been attempting to
build actuators that can behave naturally as a force-source such as series-
elastic actuators, pneumatic cylinder and air-muscles [4][5]. However, because
of the physical limitation of compliant transmission materials, the achievable
bandwidth is limited and precise position control is hardly achievable.
Another interesting approach is using force-controllable clutches to decouple
a geared EM motor with high-impedance [6]. However, those actuators using
friction-drive or electro/magnetorehologic fluids clutches have poor
efficiency and limited bandwidth and fidelity due to physical limitation and
difficulties for accurate control. Direct drive EM actuators are the best
force-source actuators with high fidelity, high bandwidth. However, the very
low force density and low efficiency at low speeds make them impractical for
mobile robot applications [2]. On the other hand, actuators with non-back-
drivable mechanisms have the advantage for pure position control tasks and
they can bear very large load without any power consumption.

Since both small and large intrinsic impedances are advantageous in different
scenario, several group have developed variable intrinsic impedance actuators.
Novel design concepts, such as those using a variable stiffness spring [7] or
antagonist non-linear devices [8], can alter the stiffness continually, yet
the variable range is rather limited. A promising approach using a series-
compliance that can be locked with a brake vary the impedance in a broader
range [9]. Another concept is a dual actuator systems using EM motors in a
serial configuration, one controlling the position and the other the impedance
[10]. Furthermore, so-called macro-micro actuators, can improve the bandwidth
of force-source type of actuators by exploiting the high-bandwidth of a small
actuator in parallel, allowing for wider-range impedance control and improved
position control[11].

While the actuator work in robotics have been focused on impedance and
bandwidth issues, in the powertrain field the torque-speed matching issue is
predominant, since power density and efficiency are critical for mobile
systems. The idea of using multiple gear ratios with electric motors has been
explored occasionally, to improve efficiency and power density [12]. A twin
motor configuration has been proposed for smooth gear shifting, where each
motor shifts at a different timing [13]. Also, a dual motor configuration
using a planetary coupling and non-back-drivable worm-gears was proposed for a
mobile robot powertrain [14]. Multiple speed powertrains provide effective
solutions for torque-speed matching, however robots require more distinct gear
ratios and a gear shifting methodology adapted to the wide dynamic behaviors
of the load.

Actuator | Force | Stiff position | Ideal op. | Holding
---|---|---|---|---
technology | control | control | speed11footnotemark: 1 | efficiency
Geared EM | Poor | Good | Low | Good
Direct drive EM | Good | Fair | High | Poor
Series elastic | Fair | Poor | Low22footnotemark: 2 | Good
Micro-Macro | Good | Fair | Low22footnotemark: 2 | Good
Serial Dual Motors | Good | Fair | High | Poor
Clutch actuators | Fair | Poor | Low22footnotemark: 2 | Poor
Pneumatic | Fair | Poor | High | Good
Hydraulic | Poor | Good | Low | Good
1 Operating speed at which the actuator is powerful and efficient.
2 Assuming the primary actuator is a geared EM motor.
TABLE I: Actuator technologies for robotics

Table I summarizes the comparison of the actuators discussed above. Note that
the actuators developed for good force control performance have lost many
advantages of traditional high impedance actuators. The proposed dual-speed
actuator with two distinctively different gear ratios is an alternative
approach that combines the advantages of both types of actuators. Compared to
the existing variable impedance actuators, the new design can achieve order-
of-magnitude different impedance instead of continuous small variations.
Moreover, the proposed dual-speed design aims to improve available power and
efficiency over a wide range of operating speed, which has rarely been
addressed in the robotics literature. All actuators listed in Table I are
powerful and efficient for either low or high output speed alone. Furthermore,
unlike the powertrains of automobiles, the proposed concept has distinctively
different reduction ratio and the output load is always fully under control
even during the transitions.

##  III Dual-speed dual-motor actuator principle

The new design concept, referred to as a Dual-Speed Dual-Motor (DSDM)
actuator, consists of a direct drive motor (M1) equipped with a locking brake
and an geared EM motor (M2) with a large reduction ratio coupled to the same
output through a differential, see Fig. 2. The differential can be viewed as a
series-type junction where the speeds add up and the force is shared.

Figure 2: DSDM actuator concept

The envisioned implementation of the DSDM concept is to embed all the
components into a single compact unit, as illustrated by Fig. 3. A lot of
weight and space could be saved by combining the reduction and the
differential gearing and having all the components inside a single housing.

Figure 3: Possible architecture of an integrated DSDM concept

The DSDM can be used in two modes, high-force mode when the brake is closed
and high-speed mode when the brake is open. The result is like having two very
different reduction ratio you can choose from during operation. Fig. 4
conceptually illustrates the principle with a leverage analogy, M1 acts like a
force source connected almost directly to the output and M2 acts like a
displacement source with a large lever arm relative to the output.

Figure 4: Dual input system

During the high-force mode, see Fig. 5a, the brake is closed and M2 drives the
output with a large mechanical advantage. The result is a low-speed
displacement-source type of actuation like a geared EM motor. During the high-
speed mode, see Fig. 5b, M1 drive the output almost directly, creating a high-
speed force-source actuator like a direct drive EM motor. Additionally, both
motors can be used simultaneously to drive the output even faster.

(a) High force mode

(b) High speed mode

Figure 5: Two modes of operation

Fig. 6 illustrates the operating range of the DSDM actuator plotted on the
standard torque-speed plane. The high-force mode region is determined by the
performance of M2 alone, since M1 is locked. The high-speed mode region can
exceed the performance of M1 alone, as M2 can be used simultaneously to
increase the output speed. The fail safe zone indicate the guaranteed
performance of the DSDM actuator in case of failure in either motor.

Figure 6: DSDM actuator operation region, with a difference between M1 and M2
gearing ratio of only 4 for illustration purposes

###  III-A Weight advantage

A DSDM actuator will be lighter than a single motor for applications with a
wide range of operating speed. Suppose that an actuator must generate 10 W
output power at two operating points: 0.5 Nm of torque at a speed of 20
rad/sec and 0.1 Nm at 100 rad/sec. A single EM motor that satisfies these
requirements at both operating points tends to be oversized in terms of power,
to reach both operating points, see Fig. 7a. A DSDM actuator can reach the
same operating points using two smaller motors with appropriate gear ratios,
see Fig. 7b. On the other hand the DSDM actuator uses more components: two
motors instead of one, more gearing and an additional brake. The DSDM concept
pays-off when the difference in speed between two required operating points
becomes larger. Fig. 8 shows the estimated weight of actuators in relation to
the ratio of operating speeds ($\lambda=\frac{w_{1}}{w_{2}}$), while the
required power output is kept at 10 W. The actuator weight is computed
assuming that the mass of each component is proportional to its maximum output
torque, with values taken from commercially available components in the 10 -
100 watts range: 2 kg/Nm for motors, 0.1 kg/Nm for gearboxes and differentials
and 0.2 kg/Nm for brakes [15]. As shown in Fig. 8, the DSDM concept becomes
advantageous when there is a large speed difference between the operating
points. This is because only the gearbox and brake need to be scaled up for
the DSDM actuator to meet the high torque requirement of the low-speed
operating point, while the motor size must be increased for the single motor
solution.

(a) One motor solution

(b) DSDM solution

Figure 7: Case study of two actuator solution for two 10 W operating points
Figure 8: Weight of a single motor compared to the DSDM concept for two 10 W
operating points at different speeds $w_{1}=100$ rad/sec,
$w_{2}=w_{1}/\lambda$

###  III-B Efficiency advantage

Since EM motor are not efficient when producing large torque at low speeds,
the DSDM concept can be very advantageous in term of efficiency. The power
loss $P_{loss}$ in an EM motor is proportional to the square of the motor
output torque $\tau_{m}$. Hence, for a given output torque $\tau_{out}$, the
motor with the larger gear ratio $R$ will be more efficient:

$\displaystyle
P_{loss}=rI^{2}=\frac{r}{k_{m}^{2}}\tau_{m}^{2}=\frac{r\tau_{out}^{2}}{k_{m}^{2}R^{2}}$
(1)

where $r$ is the winding resistance and $k_{m}$ is the motor torque constant.
Taking the same torque-speed requirements used for analyzing the weight
advantage (see Fig. 8), the single motor would have power loss $\lambda^{2}$
times greater than the DSDM at the low speed operating point.

##  IV Modeling

###  IV-A 3-ports planetary gear junction

If a planetary gear box is used to implement the serial junction with the
planet carrier connected to the output, M1 to the sun gear and M2 to the ring
gear, the kinematic relation of the system is given by

$\displaystyle
w_{o}=\underbrace{\left[\frac{1}{N+1}\right]}_{1/R_{1}}w_{1}+\underbrace{\left[\frac{N}{r_{2}(N+1)}\right]}_{1/R_{2}}w_{2}$
(2)

where $r_{2}$ is the additional reduction of M2, $N$ is the ratio of gear
teeth of the ring gear over the sun gear, and $w_{o}$, $w_{1}$ and $w_{2}$ are
angular velocities of the output shaft (port $o$), M1 input shaft (port $1$)
and M2 input shaft (port $2$). Assuming the planet gears are massless, the
torque relation of the system is given by:

$\displaystyle-\tau_{o}=\underbrace{\left[N+1\right]}_{R_{1}}\tau_{1}=\underbrace{\left[\frac{r_{2}(N+1)}{N}\right]}_{R_{2}}\tau_{2}$
(3)

Hence, the 3-ports planetary coupling can be interpreted as a 0-junction, in
the bond graph terminology, with different mechanical advantages ($R_{1}$ and
$R_{2}$) on each input ports.

###  IV-B Dynamics

Fig. 9 shows a lumped-parameter dynamic model of a DSDM when the brake is open
(high-speed mode). $J_{i}$ and $b_{i}$ are the inertia and damping of the
respective i-th ports, $I_{1}$, $I_{2}$, $k_{1}$, $k_{2}$ are the currents and
the torque constants of M1 and M2. Applying Newton’s law on each ports yields
the following equations of motions:

$\displaystyle\tau_{ext}-\tau_{o}$ $\displaystyle=Z_{o}(s)w_{o}$ (4)
$\displaystyle k_{1}I_{1}-\tau_{1}$ $\displaystyle=Z_{1}(s)w_{1}$ (5)
$\displaystyle k_{2}I_{2}-\tau_{2}$ $\displaystyle=Z_{2}(s)w_{2}$ (6)

where $Z_{i}(s)=J_{i}s+b_{i}$ represents the mechanical impedance of the i-th
ports. Note that the system is coupled due to the constraint given by eq. (2)
and (3), and that there is only two degrees of freedom among the three ports.
Fig. 10, illustrate the coupled equations motion in block diagram form.

Figure 9: Lumped-parameter dynamic model of a DSDM Figure 10: Dynamics of a
DSDM

It is then possible to eliminate one variable and express the dynamic as the
following system of two equations:

$\displaystyle\left[\begin{array}[]{l}k_{1}I_{1}\scriptstyle+\textstyle\frac{\tau_{ext}}{R_{1}}\\\
k_{2}I_{2}\scriptstyle+\textstyle\frac{\tau_{ext}}{R_{2}}\\\
\end{array}\right]=\left[\begin{array}[]{c
c}\frac{Z_{o}(s)}{R_{1}}&\scriptstyle Z_{1}(s)\\\
\frac{Z_{o}(s)}{R_{2}}\scriptstyle+R_{2}Z_{2}(s)&\scriptstyle-\textstyle\frac{R_{2}}{R_{1}}\scriptstyle
Z_{2}(s)\\\ \end{array}\right]\left[\begin{array}[]{l}w_{o}\\\ w_{1}\\\
\end{array}\right]$ (13)

The equations can be converted to state space form:

$\displaystyle\left[\begin{array}[]{c}\dot{w_{o}}\\\
\dot{w_{1}}\end{array}\right]$
$\displaystyle=\frac{-1}{J_{T}}\left[\begin{array}[]{c c}b_{T}&0\\\
R_{1}b_{o}&0\\\ \end{array}\right]\left[\begin{array}[]{c}w_{o}\\\
w_{1}\end{array}\right]+\underline{B}\left[\begin{array}[]{c}I_{1}\\\
I_{2}\end{array}\right]$ (22) $\displaystyle\text{with}\quad\underline{B}$
$\displaystyle=\frac{1}{J_{T}}\left[\begin{array}[]{c
c}k_{1}R_{1}&k_{2}R_{1}\frac{R_{1}J_{1}}{R_{2}J_{2}}\\\
k_{1}(R_{1}^{2}+\frac{R_{1}^{2}J_{o}}{R_{2}^{2}J_{2}})&-k_{2}\frac{R_{1}J_{o}}{R_{2}J_{2}}\\\
\end{array}\right]$ (25) $\displaystyle J_{T}$
$\displaystyle=\scriptstyle\left[J_{o}+R_{1}^{2}J_{1}+\left(\frac{R_{1}}{R_{2}}\right)^{2}\frac{J_{1}}{J_{2}}J_{o}\right]$
(26) $\displaystyle b_{T}$
$\displaystyle=\scriptstyle\left[b_{o}+\left(\frac{R_{1}}{R_{2}}\right)^{2}\frac{J_{1}}{J_{2}}b_{o}\right]$
(27)

where the external torque and the damping at each input port are eliminated
for brevity. From eq.(22) it can be seen that if $R_{1}<<R_{2}$ then M1
current $I_{1}$ dominates the output responses during high-speed mode.

##  V Control

###  V-A Output nullspace

From the kinematic input-output view point, the DSDM actuator has one
redundant degree of freedom. In other words, there is an infinite number of
combinations of $w_{1}$ and $w_{2}$ producing the same output speed $w_{0}$,
from eq.(2):

$\displaystyle\left[w_{o}\right]=\left[\begin{array}[]{c
c}\frac{1}{R_{1}}&\frac{1}{R_{2}}\end{array}\right]\left[\begin{array}[]{c}w_{1}\\\
w_{2}\\\ \end{array}\right]$ (31)

A vector perpendicular to the above coefficient vector forms the null space of
the DSDM actuator system. Any input combination in this direction produces
zero output speed:

$\displaystyle\left[\begin{array}[]{c}w_{1}\\\ w_{2}\\\
\end{array}\right]=\underbrace{\left[\begin{array}[]{c}1\\\ -R_{2}/R_{1}\\\
\end{array}\right]}_{\text{Nullspace
Projection}}u\;\rightarrow\;w_{0}=0\quad\forall u\in\Re$ (36)

Interestingly, a similar expression can be obtained for the dynamics of the
output in response to current inputs at the two motors, from eq.(22):

$\displaystyle J_{T}\dot{w}_{o}+b_{T}w_{o}=$
$\displaystyle\left[\begin{array}[]{c
c}k_{1}R_{1}&k_{2}R_{1}\frac{R_{1}J_{1}}{R_{2}J_{2}}\end{array}\right]\left[\begin{array}[]{c}I_{1}\\\
I_{2}\end{array}\right]$ (40)

Hence, there is a one degree of freedom space of inputs $I_{1}$ and $I_{2}$
that do not affect the output:

$\displaystyle\left[\begin{array}[]{c}I_{1}\\\
I_{2}\end{array}\right]=\underbrace{\left[\begin{array}[]{c}\frac{J_{1}}{k_{1}}\\\
-\frac{R_{2}J_{2}}{R_{1}k_{2}}\end{array}\right]}_{\text{Nullspace
Projection}}u\;\rightarrow\;J_{T}\dot{w}_{o}+b_{T}w_{o}=0\quad\forall u\in\Re$
(45)

###  V-B High-force mode

During high-force mode, with M1 locked by the brake, the actuator behaves like
a regular geared EM motor and standard speed or position control scheme can be
used.

###  V-C High-speed mode

Figure 11: High-speed mode control loop

With a small reduction ratio $R_{1}$ the DSDM actuator can be controlled like
a direct drive motor: the output torque can be controlled directly by
controlling M1 current, see inner feedback loop in Fig. 11. The output loop
can be used for various objectives. With the measurement of output position
and velocity, impedance control, friction compensation and feedback
linearization can be implemented. Moreover, exploiting the nullspace, a
secondary objective can be brought into the system without influencing the
output. In case the first motor is overloaded, for example, the second motor
can reduce the load by projecting inputs through the null space, producing no
effect upon the output, but changing the proportion of the two input commands.
Note that the nullspace projection vector, see eq.(45), depends only on
parameters associated with the motors. Therefore, it is possible to project
the secondary controller inputs on the output nullspace even if the output
dynamic parameters that include the load inertia and damping are unknowns. The
secondary controller can be used not only for load balancing but also for
minimizing power consumption, avoiding speed saturation of M1, etc.

###  V-D Seamless mode transitions

Unlike powertrains, where the load is mostly a large inertia and can be
disengaged momentarily during gear shiftings, robotic actuators face a wide
range of dynamic load, which must be controlled continuously even during a
transition. For instance, if it drives a spring-like load, even a very short
disengagement from the load might lead to large undesirable output motion.
This section addresses transition control between the two modes.

####  V-D1 From high-force mode to high-speed mode

This transition control is simple. The locking brake can be released anytime,
M1 is then instantaneously freed and the controller can immediately switch to
the high-speed control mode.

####  V-D2 From high-speed mode to high-force mode

For this transition, M1 speed $w_{1}$ must be brought to zero so that the
locking brake can be engaged. Two algorithms, a kinematic and a dynamic
approach, can be considered. The former assumes local high gain velocity
feedback controls for the individual motors. Hence velocities $w_{1}$ and
$w_{2}$ can be treated as control inputs. Using the nullspace projection
vector from eq.(36), the kinematic control law can be written as

$\displaystyle\left[\begin{array}[]{c}w_{1}\\\ w_{2}\\\
\end{array}\right]=\underbrace{\left[\begin{array}[]{c}1\\\ -R_{2}/R_{1}\\\
\end{array}\right]}_{\text{Nullspace
Projection}}u_{1}+\left[\begin{array}[]{c}0\\\ R_{2}\\\
\end{array}\right]u_{2}$ (52)

leading to

$\displaystyle w_{o}=u_{2}\quad\text{with}\quad w_{1}=u_{1}$ (53)

Therefore during transitions, using $u_{1}$ velocity $w_{1}$ can be driven to
zero, while fully controlling the output velocity using $u_{2}$. The kinematic
control law is valid only when high fidelity velocity controls are available.
Alternatively the dynamic control algorithm does not require this. While
running the high-speed control scheme of Fig. 11, a braking law for $w_{1}$
can be used in parallel as the secondary controller projected on the output
nullspace:

$\displaystyle\left[\begin{array}[]{c}I_{1}\\\
I_{2}\end{array}\right]=\underbrace{\left[\begin{array}[]{c}\frac{J_{1}}{k_{1}}\\\
-\frac{R_{2}J_{2}}{R_{1}k_{2}}\end{array}\right]}_{\text{Nullspace
Projection}}\underbrace{-Cw_{1}}_{\text{Braking
Law}}+\left[\begin{array}[]{c}\frac{1}{k_{1}R_{1}}\\\
0\end{array}\right]\tau_{d}$ (60)

leads to:

$\displaystyle J_{T}\dot{w}_{o}+b_{T}w_{o}=$
$\displaystyle\,\tau_{d}\quad\text{and}\quad\dot{w}_{1}=-Cw_{1}+f(w_{o},\tau_{d})$
(61)

Hence, the output is not influenced by the braking law due to orthogonality,
and is still controlled using the desired torque $\tau_{d}$ determined by the
high-speed mode controller. On the other hand, $w_{1}$ is directly influenced
by the braking law but also by the output speed and the desired output torque
$\tau_{d}$. Mathematically, it would be possible to also fully uncouple
$\dot{w}_{1}$ equation, but the control law would not be practical in the
scenario of $R_{1}<<R_{2}$ when considering torque and speed saturations.
Increasing the gain $C$ will lead to faster braking of $w_{1}$, however
$I_{2}$ will saturate if the gain is too large. A large $C$ can still be used
for faster braking at a cost of deviation from the desired torque $\tau_{d}$.
There is a trade-off, passed the $I_{2}$ saturation point, between fast
braking of $w_{1}$ for fast transition and high-fidelity output torque
control.

###  V-E Automatic mode selection

To simplify the programming of a DSDM actuator by an end-user, it is
interesting to integrate an automatic mode selection in the control scheme. As
shown in Fig. 12, the mode selection could be made based the on the load
conditions and a user-provided desired impedance for each task. When a large
impedance is desired, the actuator automatically select the mode that allow
the fastest convergence to the target given the load conditions. Hence, the
actuator will use high-speed mode, except when encountering resistance it will
automatically ”down shift” to high-force mode. If a low impedance or torque
limitation is desired, then the DSDM actuator must stay in high-speed mode to
enforce those specifications.

Figure 12: State machine for automatic mode selection

##  VI EXPERIMENTAL VALIDATIONS

###  VI-A Test bench design

A test bench was developed to demonstrate the functionality of the DSDM
concept, see Fig. 13. Discrete components (motors, brake, gearhead and
differential) are used for the ease of implementation and future
modifications. Two 20 Watts DC motor (Maxon RE 25) are used for both M1 and M2
positions. M1 is equipped with an electromechanical brake (Maxon AB 28) and M2
with a gearhead reductor of 18:1 (Maxon GP 32). The differential is a custom
made 3-ports planetary gear, see Fig. 14, with an additional reduction of 3:1
on the ring gear input. The planetary gearing is designed with a ratio $N$ of
3 (see eq.(2)), hence $R_{1}$= 4 and $R_{2}$= 72. The output is connected to a
20 mm lead ballscrew, leading to specifications given by Table II.

Figure 13: Proof-of-concept DSDM linear actuator Mode | Max Force | Max Speed | Apparent Output Mass
---|---|---|---
High-force | 600 N | 40 mm/sec | 750 kg
High-speed | 30 N | 0.7 m/sec | 2 kg
TABLE II: Prototype specifications using two 20 W motors

(a) Prototype

(b) CAD section view

Figure 14: Custom 3-ports gearbox

###  VI-B Controller Implementations

The controller was implemented using a NI Compact Rio platform and two NI 9505
DC motor drives. All inner current control loops are implemented on the FPGA
at a 20 kHz sampling rate and the outer control loops are implemented in the
real time OS at a 500 Hz sampling rate. Two encoders provide the position at
both motor shafts and the output position is computed from those values using
the kinematic relation of eq.(2). The current is sensed with a Hall effect
sensor (ACS712) and the speed data is computed using filtered discrete
differentiation of the position data.

###  VI-C High-force mode

For the high-force mode experiment, M2 is controlled using an experimentally
tuned PI with an anti-windup scheme. Fig. 15 shows the response to a 200 mm
step of the reference position, with a perturbation at $t\,$= 5 sec (human
applying about 50 lbs), see experiment #1 in the video. The results show that
the actuator is speed limited (maximum supply voltage reached) for most of the
course, but very strong as illustrated by a small speed loss during the
perturbation. The actuator precision is also very good as the final
positioning error is less than a micron (neglecting the backlash as the
position sensor is upstream of the gearbox). Hence, the actuator acts like a
strong and precise displacement source as desired in this mode.

Figure 15: Output motion using high-force mode for a 200 mm target step

###  VI-D High-speed mode

For the high-speed mode experiment, the inner current controllers are pure
integrator leading to closed-loop bandwidth well over 500 Hz. Part of the
prototype coulomb friction (mainly in the ballscrew) is canceled using the
feedback linearization loop, see Fig. 11, and the prescribed impedance is a
pure stiffness, relying on the natural friction of the system to provide the
damping. Fig. 16 shows the results of, as before, the response to a 200 mm
reference position step, see experiment #2 in the video. The result with the
high-speed mode is much more dynamic than the high-force mode response, and a
maximum speed of 0.3 m/sec was reach during the motion. The actuator overshoot
a little bit before stabilizing on the target. Also, when the desired
impedance is set to zero, the actuator is easy to back-drive by hand, as
illustrated by experiment #3 in the video. Moreover, the actuator can limit
its output force and handle collision without stability issues as demonstrated
in experiment #5 in the video. An inherent advantage of not relying on force
feedback to control the impedance.

Figure 16: Output motion using high-speed mode for a 200 mm target step

###  VI-E Mode transitions

Fig. 17 shows the results from an experiment were the actuator first control
the output speed using high-force mode, then switch to high-speed mode (M2
speed is then brought to zero for demonstration purposes) and then switch back
to high-force mode, see experiment #4 in the video. The results show that the
transitions are seamless and that the output is always under control, here
maintaining a constant speed. Since the actuator is fighting a dissipative
load (the friction in the ballscrew), a constant force must be applied at all
time, if not the output would suddenly stop. Note, the noisy behavior when
using high-speed mode at low speed is due to a limited angular velocity
resolution on $w_{1}$.

Figure 17: Mode transitions with a constant output speed target

Fig. 18 shows the results from an experiment were the actuator is controlled
in position (target at 300 mm) and automatically select the mode as proposed
in section V-E. Here the actuator first accelerate quickly using high-speed
mode since there is no resistance and switch back to high-force mode after
impacting a ball to continue its course (experiment #6 in the video). Hence
the transition is done while in contact with a spring-like load. The results
show that the DSDM is still able to switch mode quickly even during this
dynamic scenario. This experiment also illustrate how a single task can take
advantage of the two operating modes of the actuator; fast reaching and large
force as soon as the contact is made.

Figure 18: Automatic down shift during contact with a spring-like load

##  VII CONCLUSION AND OUTREACH

In this paper, the use of distinctively different gear ratios with EM motors
in robotics is proposed. This allow for an effective use of power over a wide
range of torque-speed conditions, enabling smaller and more efficient actuator
systems. Furthermore, very different gear ratios lead to drastic intrinsic
impedance change, enabling a displacement-source type of behavior with the
large non-back-drivable reduction ratio and good force control capability with
the small reduction ratio. Additionally, a novel dual-motor implementation of
dual-speed is proposed to address the problematics of gear shifting in the
context of robotics. The proposed DSDM actuator can keep full control of the
output during gear shifting, with a controller exploiting the redundancy of
inputs. The DSDM actuator is advantageous for many actual robotic
applications, but also enabled new applications that are impossible with
regular actuators because the requirements are too conflicting to meet with a
single motor of usable size.

Also, many aspect of using multiple gear ratios in a robotic context have yet
to be explored. Gear shifting is a very non-linear process and it would be
interesting to explore with greater depth how it should be integrated in a
robot routine. Regarding the DSDM concept, in this first iteration two
identical motors were used. However, it could be more effective to use
asymmetric motors optimized for their particular roles. For instance M1 could
be optimized for force bandwidth and M2 for power density and precision.
Generalizing, it would even be possible to use different technologies for the
force source and the displacement source, recall Fig. 4.

## ACKNOWLEDGMENTS

Special thanks to Lluis Penalver-Aguila and James Torres for the help with the
detailed drawing and the manufacturing of the gearbox.

## References

  * [1] S. Hirose, “A study of design and control of a quadruped walking vehicle,” _The International Journal of Robotics Research_ , vol. 3, no. 2, pp. 113–133, Jun. 1984.
  * [2] J. Hollerbach, I. Hunter, and J. Ballantyne, “A comparative analysis of actuator technologies for robotics,” in _The Robotics Review_ , MIT press ed., 1992, vol. 2, pp. 299–342.
  * [3] N. Hogan and S. Buerger, “Impedance and interaction control,” in _Robotics and Automation Handbook_. CRC Press, 2004.
  * [4] H. Hanafusa and H. Asada, “Stable prehension by a robot hand with elastic fingers,” _Proc. 7th Int. Symp. Industrial Robots_ , pp. 361–368, 1977.
  * [5] G. Pratt and M. Williamson, “Series elastic actuators,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , vol. 1, Aug. 1995, pp. 399–406 vol.1.
  * [6] P. Fauteux, M. Lauria, B. Heintz, and F. Michaud, “Dual-differential rheological actuator for high-performance physical robotic interaction,” _IEEE Transactions on Robotics_ , vol. 26, no. 4, pp. 607 –618, Aug. 2010.
  * [7] G. Tonietti, R. Schiavi, and A. Bicchi, “Design and control of a variable stiffness actuator for safe and fast physical human/robot interaction,” in _IEEE International Conference on Robotics and Automation_ , Apr. 2005, pp. 526–531.
  * [8] K. Koganezawa, T. Nakazawa, and T. Inaba, “Antagonistic control of multi-DOF joint by using the actuator with non-linear elasticity,” in _IEEE International Conference on Robotics and Automation_ , May 2006, pp. 2201–2207.
  * [9] D. Leach, F. Gunther, N. Maheshwari, and F. Iida, “Linear multi-modal actuation through discrete coupling,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Oct. 2012, pp. 2437–2442.
  * [10] B.-S. Kim, J.-B. Song, and J.-J. Park, “A serial-type dual actuator unit with planetary gear train: Basic design and applications,” _IEEE/ASME Transactions on Mechatronics_ , vol. 15, no. 1, pp. 108–116, 2010.
  * [11] J. B. Morrell and J. K. Salisbury, “Parallel-coupled micro-macro actuators,” _The International Journal of Robotics Research_ , vol. 17, no. 7, pp. 773–791, Jul. 1998.
  * [12] N. McKeegan, “Antonov’s 3-speed transmission for electric vehicles boosts efficiency by 15 percent,” Jul. 2011. [Online]. Available: http://www.gizmag.com/antonov-3-speed-transmission-ev/19088/
  * [13] S. Bologna, “Electric propulsion system for vehicles,” U.S. Patent US8 739 655 B2, Jun., 2014.
  * [14] H. Lee and Y. Choi, “A new actuator system using dual-motors and a planetary gear,” _IEEE/ASME Transactions on Mechatronics_ , vol. 17, no. 1, pp. 192–197, 2012.
  * [15] “Maxon motor: Small dc motors and drive systems.” [Online]. Available: www.maxonmotorusa.com


# Universality in High Energy Collisions of small and large systems

P. Castorina1,2, A. Iorio2, D. Lanteri1,3, H. Satz4 and M. Spousta2

###### Abstract

Strangeness enhancement and collective flow are considered signatures of the
quark gluon plasma formation. These phenomena have been detected not only in
relativistic heavy ion collisions but also in high energy, high multiplicity
events of proton-proton and proton-nucleus (“small systems”) scatterings. A
universal behavior emerges by considering the parton density in the transverse
plane as the dynamical quantity to specify the initial condition of the
collisions, which in $e^{+}e^{-}$ annihilation at the available energies is
too low to expect collective effects.

##  1 Introduction

Recent experimental results in proton-proton ($pp$) and proton-nucleus ($pA$)
collisions [1, 2, 3, 4, 5, 6, 7, 8, 9] support the conclusion that the system
created in high energy, high multiplicity collisions with these “small”
initial settings is essentially the same as that one produced with “large”
initial nucleus-nucleus ($AA$) configurations.

The ALICE collaboration reported [1] an enhanced production of multi-strange
hadrons, previously observed in $PbPb$ collisions [10], in high energy, high
multiplicity $pp$ events, shown in fig. 1.(left).

Moreover, the energy loss in $AA$ collisions was shown to scale in small and
large systems [11] and another important similarity among $pp$, $pA$, and $AA$
collisions was identified in several measurements of long-range di-hadron
azimuthal correlations [12, 4, 6, 3], indicating universality in flow-like
patterns.

In this contribution, based on ref. [13], the universal behavior will be
discussed by considering a dynamical variable, previously introduced to
predict the strangeness enhancement in $pp$ [14, 15, 16], which corresponds to
the initial entropy density of the collisions and takes into account the
transverse size (and its fluctuations) of the initial configuration in high
multiplicity events.

The next sections are devoted to the strangeness enhancement, mean transverse
momentum and elliptic flow. The final section contains our comments,
clarifying why collective effects cannot arise in $e^{+}e^{-}$ annihilation at
LEP (or lower) energies.

##  2 Universality in strangeness production and in mean transverse momentum

In fig.1.(left) the multi-strange hadron production data are plotted versus
the charge multiplicity at mid-rapidity. A stronger evidence of the universal
behavior in $pp,pA,AA$ can be discoverd by considering the initial entropy
density $s_{0}$, given in the one-dimensional hydrodynamic formulation [17] by
the form

$s_{0}\tau_{0}\simeq\frac{1.5}{A_{T}}\;\frac{dN^{x}_{ch}}{dy}=\frac{1.5}{A_{T}}\;\frac{N_{part}^{x}}{2}\left.\frac{dN^{x}_{ch}}{dy}\right|_{y=0}\;,$
(1)

with $x\simeq pp$, $pA$, $AA$. Here $A_{T}$ is the transverse area,
$(dN^{x}_{ch}/dy)_{y=0}$ denotes the number of produced charged secondaries,
normalized to half the number of participants $N_{part}^{x}$, in reaction $x$,
and $\tau_{0}$ is the formation time. The quantity $s_{0}\tau_{0}$ is directly
related to the number of partons per unit of transverse area and therefore,
due to the large fluctuations in high multiplicity events, a reliable
evaluation of the transverse area for different collisions as a function of
the multiplicity is required.

In studying the strangeness enhancement and the average $p_{t}$, we use
results from Glauber Monte Carlo (MC) [18] to obtain $A_{T}$ and the
multiplicity for $AA$ and for $pPb$ collisions. For $pp$ scaterings the
effective transverse area is sensitive to the fluctuations of the gluon field
configurations and therefore we apply the CGC parameterization of the
transverse size as a function of $N_{ch}$ [19]. The universal trend in
elliptic flow needs a more refined analysis (see sec. 3).

Alice data are plotted versus $s_{0}\tau_{0}$ in fig. 1.(right), showing a
complete smooth trend of the strangeness production among $pp,pA,AA$ data.

Figure 1: The strangeness production quantified in terms of the ratio of
yields of K, $\Lambda$, $\Xi$, and $\Omega$ hadrons to pions evaluated as a
function of the multiplicity (left panel) and of the initial entropy density
(right panel). Data from Ref. [1] (and refs. therein).

This universal behavior is confirmed by a similar analysis based on the
Statistical Hadronization model (SHM) [20], where strangeness production is
reduced with respect to the predicted rates by one further parameter,
$0<\gamma_{s}\leq 1$. The predicted rate for a hadron species containing
$\nu=1$, $2$, $3$ strange quarks is suppressed by the factor
$\gamma_{s}^{\nu}$.

The energy dependence of $\gamma_{s}$ is reported in fig. 2.(left), suggesting
a different behavior in strangeness production in $pp$ versus $AA$ collisions.
On the other hand, by plotting $\gamma_{s}$ versus the parton density in the
transverse plane ( see fig. 2.(right)), a smooth behavior for small and large
setting emerges, with a enhancement for $pp$ in high energy, high multiplicity
events. The universal trend shows that $\gamma_{s}$ increases with the parton
density in the transverse plane, up to the fixed point $\gamma_{s}=1$, where
any suppression disappears.

Figure 2: The strangeness suppression factor $\gamma_{s}$ as a function of the
energy (left) and of the initial entropy density (right) evaluated for data
from Refs. [21, 22, 23]. The Phobos parameterization [24] for the relation
between charge multiplicity, energy and the number of participants is applied
for RHIC data.

An analogous study can be done for the scaling of the average $p_{t}$, as
discussed in [19]. We analyze the average $p_{t}$ in the low transverse
momentum region where the soft, non-perturbative, effects in the particle
production are more important than in the higher $p_{t}$ range. The behavior
of the average $p_{t}$ is evaluated in the region $0.15<p_{t}<1.15$ GeV (fig.
3.(left)) and $0.15<p_{t}<2$ GeV (fig. 3-(right)) for different colliding
systems as a function of the previous dynamical variable. One can see that the
average $p_{t}$ for soft particle production follows the same slowly
increasing trend for all the collisional systems.

Figure 3: Average $p_{t}$ as a function of initial entropy density evaluated
in the interval of $0.15<p_{t}<1.15$ GeV (left) and $0.15<p_{t}<2$ GeV ([13]
for refs. to experimental data).

##  3 Elliptic flow and partecipants eccentricity

In non-central collisions, the beam direction and the impact parameter vector
define a reaction plane for each event. If the nucleon density within the
nuclei is continuous, the initial nuclear overlap region has an “almond-like”
shape and the impact parameter determines uniquely the initial geometry of the
collision. In a more realistic description, where the position of the
individual nucleons that participate in inelastic interactions is considered,
the overlap region has a more irregular shape and the event-by-event
orientation of the almond fluctuates around the reaction plane. Therefore, in
the analysis of the elliptic flow where the fluctuations are important, the
geometrical eccentricity is replaced by the participant eccentricity,
$\epsilon_{part}$, defined using the actual distribution of participants. The
size of the fluctuation in $\epsilon_{part}$ and its correlated transverse
area $S$ (different from the geometrical one, $A_{T}$) are evaluated by
Glauber MC.

The scaling of $v_{2}/\epsilon_{part}$ versus the initial entropy density is
depicted in fig. 4.(left) for $AA$ and $pp$. One can see that the $pp$ trend,
at lower values, is smoothly followed by the data-points from $AA$ collisions.

To clarify the different role of $A_{T}$ versus $S$, fig. 4.(right) shows that
the scaling in $v_{2}/\epsilon_{part}$ is not observed if one considers
$A_{T}$ rather than $S$ in evaluating the initial entropy density.

Figure 4: The $v_{2}/\epsilon_{part}$ values for $pp$, $PbPb$, $AuAu$, and
$CuCu$ evaluated as a function of entropy density (left) (for data references
see [13]). The $v_{2}/\epsilon_{part}$ values for $pp$ and $PbPb$ evaluated as
a function of entropy density when the geometrical transverse area $A_{T}$,
rather tha $S$, is used in the evaluation of the initial energy density for
data (right).

##  4 Comments and Conclusions

By the scaling variable $s_{0}\tau_{0}$, one can evaluate at which
multiplicity the same behavior in high-multiplicity $pp$ and $PbPb$ collisions
is expected, by solving the equation
$(dN/d\eta)_{AA}/A_{T}^{AA}=x/A_{T}^{pp}(x)$ for $x$ being the multiplicity in
$pp$. The result is shown in Tab. 2 of Ref. [13].

A final comment concerns the possible detection of collective effects in
$e^{+}e^{-}$ annihilation. Indeed, more recently, LEP data have been
reconsidered [25] to check if a flow-like behavior is generated with this
initial, small, non hadronic, setting. The answer is negative and confirmed at
lower energy by the BELLE collaboration [26]. Moreover, there is no
strangeness enhancement in $e^{+}e^{-}$ annihilation [27] as a function of the
available energy.

According to the universality point of view, strangeness enhancement and
collective flow are both indications of the formation of an initial system
with high parton number density in the transverse plane. Therefore, as
confirmed in Ref. [28], in $e^{+}e^{-}$ annihilation at LEP or lower energies
there is no chance of observing the enhancement of the strangeness production
or flow-like effects because the parton density in the transverse plane is too
small. Acknowledgements

P.C. is partially supported by UNCE/SCI/013.

## References

  * [1] Jaroslav Adam et al., Nature Phys., 13:535–539, 2017.
  * [2] Betty Bezverkhny Abelev et al.,Phys. Lett., B728:25–38, 2014.
  * [3] Vardan Khachatryan et al., Phys. Lett., B765:193–220, 2017.
  * [4] Georges Aad et al,Phys. Rev. Lett., 116(17):172301, 2016.
  * [5] Vardan Khachatryan et al., JHEP, 09:091, 2010.
  * [6] Vardan Chachatryan et al.Phys. Rev. Lett., 116(17):172302, 2016.
  * [7] C. Aidala et al. Nature Phys., 15(3):214–220, 2019.
  * [8] Edward K. G. Sarkisyan, Aditya Nath Mishra, Raghunath Sahoo, and Alexander S. Sakharov, Phys. Rev., D94(1):011501, 2016.
  * [9] Constantin Loizides,Nucl. Phys., A956:200–207, 2016.
  * [10] Betty Bezverkhny Abelev et al, Phys. Lett., B728:216–227, 2014.[Erratum: Phys. Lett.B734,409(2014)].
  * [11] A. Adare et al. Phys. Rev., C93(2):024911, 2016.
  * [12] Georges Aad et al., Phys. Rev. Lett., 110(18):182302, 2013.
  * [13] P.Castorina, A.Iorio, D.Lanteri, M.Spousta and H.Satz, Phys. Rev. C 101, 054902 (2020).
  * [14] P. Castorina and Helmut Satz. Eur. Phys. J., A52(7):200, 2016.
  * [15] P. Castorina, S. Plumari, and H. Satz.Int. J. Mod. Phys., E25(08):1650058, 2016.
  * [16] P.Castorina, Salvatore Plumari, and Helmut Satz. Int. J. Mod. Phys., E26(12):1750081, 2017.
  * [17] J.D. Bjorken, Lecture Notes in Physics (Springer) 56 (1976) 93.
  * Loizides et al. [2018] Constantin Loizides, Jason Kamin, and David d’Enterria.
  * [19] L. McLerran and P. Tribedy.Nucl. Phys., A945:216–225, 2016 and references therein.
  * [20] F. Becattini, review paper International School on Quark-Gluon Plasma and Heavy Ion Collisions: past, present, future, 1 2009.
  * [21] F. Becattini, 33rd Workshop of the INFN Eloisatron Project, Erice, Italy, October 19-25, 1996, pages 74–104, 1996.
  * [22] J. Takahashi and R. Derradi de Souza,_24th Winter Workshop on Nuclear Dynamics (WWND 2008) South Padre Island, Texas, April 5-12, 2008_ , 2008.
  * STAR Collaboration [2017] STAR Collaboration, _Phys. Rev._ , C96(4):044904, 2017.
  * PHOBOS Collaboration [2011] PHOBOS Collaboration, _Phys. Rev._ , C83:024913, 2011.
  * [25] A.Badea et. al, Phys. Rev. Lett. 123,212002 (2020).
  * [26] A.Abdesselam et al. , Measurement of two-particle correlation in hadronic $e^{+}e^{-}$ collisions at Belle, arXiv:2008.04187
  * [27] F. Becattini, P. Castorina, J. Manninen and H. Satz, Eur.Phys.J.C56:493-510,2008.
  * [28] P.Castorina, D.Lanteri and H.Satz, Strangeness enhancement and flow-like effects in $e^{+}e^{-}$ annihilation at high parton density, arXiv:2011.06966.
  * [29]


# Training With Data Dependent Dynamic
Learning Rates

Shreyas Saxena
Apple
<EMAIL_ADDRESS>Nidhi Vyas
Apple
<EMAIL_ADDRESS>Dennis DeCoste
Apple
<EMAIL_ADDRESS>
###### Abstract

Recently many first and second order variants of SGD have been proposed to
facilitate training of Deep Neural Networks (DNNs). A common limitation of
these works stem from the fact that they use the same learning rate across all
instances present in the dataset. This setting is widely adopted under the
assumption that loss-functions for each instance are similar in nature, and
hence, a common learning rate can be used. In this work, we relax this
assumption and propose an optimization framework which accounts for difference
in loss function characteristics across instances. More specifically, our
optimizer learns a dynamic learning rate for each instance present in the
dataset. Learning a dynamic learning rate for each instance allows our
optimization framework to focus on different modes of training data during
optimization. When applied to an image classification task, across different
CNN architectures, learning dynamic learning rates leads to consistent gains
over standard optimizers. When applied to a dataset containing corrupt
instances, our framework reduces the learning rates on noisy instances, and
improves over the state-of-the-art. Finally, we show that our optimization
framework can be used for personalization of a machine learning model towards
a known targeted data distribution.

##  1 Introduction

Figure 1: Eigen-value spectrum of Hessian for ResNet-18 trained on CIFAR100.
For three classes, we plot top-10 eigen-values over the course of training.
The loss function characteristics are different for each class.

Owing to the non-convex nature of most loss functions, coupled with poorly
understood learning dynamics, optimization of Deep Neural Networks (DNNs) is a
challenging task. Recent work can be broadly divided in two categories: (1)
adaptive first order variants of SGD, such as ADAMw [24], RMSprop [12] and
Adam [20], and (2) second order methods such as K-FAC [26]. These methods
adapt the optimization process per parameter and obtain improved convergence
and generalization. The objective function for these frameworks is to minimize
finite-sum problems:
$\mathcal{L}(\theta)=\frac{1}{N}\sum_{i}^{N}L^{i}(\theta)$, where
$L^{i}(\theta)$ denotes the loss on a single training data point. Standard
optimization frameworks assume that the $N$ loss functions come from the same
distribution, and share characteristics such as the curvature, Lipshitz
constant, etc. However, in practice, this is not true. Figure 1 highlights the
difference in loss function characteristics over the course of training for
classes present in CIFAR100.

In contrast to prior work, which models the learning dynamics of sum of finite
problems, in this work, we account for differences in characteristics of
underlying loss functions in the sum. More specifically, for each instance in
the dataset, we associate a dynamic learning rate, and learn it along with
model parameters. The SGD update rule for our framework takes the following
form:

$\theta^{t+1}=\theta^{t}-\frac{\lambda}{N}\sum_{i}^{N}(\mathbf{w}^{i,t}_{inst}\cdot\frac{\partial
L^{i}}{\partial\theta^{t}})$

where $L^{i}$, $\lambda$, $\mathbf{w}^{i,t}_{inst}$ denotes loss, model
parameter learning rate, and multiplicative learning rate correction for
instance $i$ at time step $t$. Having a dynamic learning rate per instance
allows our optimization framework to focus on different modes of data during
the course of optimization. Instead of using a heuristic, we learn the
learning rate for each instance via meta-learning using a held out meta set.
We also extend our framework to learn a dynamic learning rate per class.

The main contributions of our work are:

  1. 1.

We present an optimization framework which learns an adaptive learning rate
per instance in the dataset to account for differences in the loss function
characteristics across instances. We extend this framework to learn adaptive
learning rates per class. We also show that learning dynamic weight-decay
facilitates learning dynamic learning rates on data.

  2. 2.

We show that learning adaptive learning rate on data leads to variance
reduction, and explains faster convergence and improved accuracy.

  3. 3.

We show that in presence of noisy data, our framework reduces the learning
rates on noisy instances, and prioritizes learning from clean instances. Doing
so, we our method outperforms state-of-the-art by a significant margin.

  4. 4.

We show that our framework can be used for personalization of machine learning
models towards a targeted distribution.

##  2 Learning the learning rate for data

As mentioned earlier, the main goal of our work is to learn a dynamic learning
rate for instances present in the train dataset. In this section, we first
formalize this intution and present the framework for learning instance level
learning rates. Next, we will show how our framework can be extended to learn
class level learning rates. We derive our method using stochastic gradient
descent (SGD) as the optimizer for model paramters, however, extension to
other class of optimizers can be done in a similar manner.

###  2.1 Learning instance level learning rate

Let
$\left\\{\left(\mathbf{x}^{i}_{train},y^{i}_{train}\right)\right\\}^{N}_{i=1}$
denote train set, where $\mathbf{x}^{i}\in\mathbb{R}^{d}$ denotes a single
data point in the train set and $y^{i}\in\\{1,...,k\\}$ denotes the
corresponding target. Let $f(\mathbf{x},\theta^{t})$ and $\theta^{t}$ denote
the model and model’s parameters at step $t$ respectively. Let
$L(\mathbf{x}^{i}_{train},y^{i}_{train};\theta^{t})$ denote an arbitrary
differentiable loss function on data point $i$ at time step $t$. In what
follows, we denote $L(\mathbf{x}^{i}_{train},y^{i}_{train};\theta^{t})$ as
$L_{train}^{i}(\theta^{t})$. Let $\mathbf{w}_{inst}^{t}\in\mathbb{R}^{N}$
denote the instance level parameters at time step $t$. Instance level
parameters weigh contribution of instances in the gradient update (see
Equation 1), and can be interpreted as a multiplicative learning rate
correction over the learning rate of model parameter’s optimizer. Note,
learning the learning rate for each instance is equivalent to learning
weighting for each instance. For the ease of explanation, we choose the latter
formulation. Our goal in optimization is to solve for optimal $\hat{\theta}$
by minimizing the weighted loss on the train set:

$\displaystyle\mathcal{L}_{train}(\theta^{t},\mathbf{w}_{inst}^{t})$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}w^{i,t}_{inst}\cdot
L_{train}^{i}(\theta^{t})$ (1)
$\displaystyle\hat{\theta}(\mathbf{w}_{inst}^{t})$
$\displaystyle=\arg\min_{\theta}\mathcal{L}_{train}(\theta,\mathbf{w}_{inst}^{t})$
(2)

The solution to above equation is a function of $\mathbf{w}_{inst}^{t}$, which
is not known a priori. Setting $\mathbf{w}_{inst}^{t}$ as $\mathbf{1}$ at all
time steps recovers the standard gradient descent optimization framework, but
that might not be optimal. This brings us to the question: What is the optimal
value of $\mathbf{w}^{t}_{inst}$ i.e. the learning rate for data points at
time step $t$?

#### Learning dynamic instance level learning rate via meta-learning

In contrast to model parameters, whose optimal value is approximated by
minimizing the loss on train set, we can not approximate the optimal value of
$\mathbf{w}^{t}_{inst}$ by minimizing the loss on train set. Doing so, leads
to a degenerate solution, where $\mathbf{w}_{inst}^{t}=\mathbf{0}$. In
principle, the optimal value of $\mathbf{w}^{t}_{inst}$ is the one, which when
used to compute the gradient update at time step $t$, minimizes the error on a
held-out set (referred as meta set) at convergence, i.e
$\hat{\mathbf{w}}_{inst}^{t}=\arg\min_{\mathbf{w}_{inst}^{t}}\mathcal{L}_{meta}(\hat{\theta}(\mathbf{w}_{inst}^{t}))$.
Here $\hat{\theta}$ denotes model parameters at convergence. The sequence of
model updates from time step $t$ till convergence
($\theta^{t}\rightarrow\hat{\theta}$) can be written as a feed forward
computational graph, allowing us to backpropagate meta-gradient (gradient on
meta set) to $\mathbf{w}^{t}_{inst}$. However, this is not feasible in
practice due to: (1) heavy compute for backpropagating through time steps, (2)
heavy memory foot-print from saving all intermediate representations and (3)
vanishing gradient due to backpropagation through time steps.

To alleviate this issue, we approximate the meta-gradient at convergence with
the meta-gradient at time step $t+1$. More formally, we sample a mini-batch
from train set, and write one step SGD update on model parameters
(${\theta^{t+1}}$) as a function of instance parameters at time step $t$ (see
equation 3). The one step update is used to compute loss on meta set
$\mathcal{L}_{meta}({\theta^{t+1}})$, which is then used to compute the meta-
gradient on instance parameters:

$\displaystyle{\theta^{t+1}}(\mathbf{w}_{inst}^{t})$
$\displaystyle=\theta^{t}-\frac{\lambda}{N}\sum_{i=1}^{N}\mathbf{w}^{i,t}_{inst}\frac{\partial
L_{train}^{i}(\theta^{t})}{\partial\theta^{t}}$ (3)
$\displaystyle\frac{\partial\mathcal{L}_{meta}({\theta^{t+1}})}{\partial
w^{i,t}_{inst}}$
$\displaystyle=\frac{\partial\mathcal{L}_{meta}({\theta^{t+1}})}{\partial{\theta^{t+1}}}\cdot\frac{\partial{\theta^{t+1}}}{\partial
w^{i,t}_{inst}}$ (4)
$\displaystyle\frac{\partial\mathcal{L}_{meta}({\theta^{t+1}})}{\partial
w^{i,t}_{inst}}$
$\displaystyle=\frac{-\lambda}{N}\cdot\frac{\partial\mathcal{L}_{meta}({\theta^{t+1}})}{\partial{\theta^{t+1}}}\cdot\big{[}{\frac{\partial
L_{train}^{i}}{\partial\theta^{t}}}\big{]}^{T}$ (5)

Here, $\lambda$ and $N$ corresponds to the learning rate of the model
optimizer and number of samples in train mini-batch respectively. Using the
meta-gradient on instance-parameters we update the instance-parameters using
first order gradient update rule (see equation 6).

$\displaystyle w^{i,t+1}_{inst}$
$\displaystyle=w^{i,t}_{inst}-\lambda_{w_{inst}}\frac{\partial\mathcal{L}_{meta}({\theta^{t+1}}^{*})}{\partial
w^{i,t}_{inst}}$ (6)

The pseduo code for our method is outlined in Algorithm 1.

Algorithm 1 Learning algorithm for learning the learning rate per instance

0: Train set $\mathcal{D}_{train}$, meta set $\mathcal{D}_{meta}$, model
learning rate $\lambda$, instance parameter learning rate $\lambda_{inst}$,
max iterations $T$.

0: Model parameters at convergence $\theta^{T}$.

1: Initialize model parameters $\theta^{0}$ and instance level parameters
$\mathbf{w}_{inst}^{0}$.

2: for $t=0$ to $T-1$ do

3: $\\{x_{train}^{i},y_{train}^{i}\\}\leftarrow$
SampleMiniBatch($\mathcal{D}_{train}$).

4: $\\{x_{meta}^{j},y_{meta}^{j}\\}\leftarrow$
SampleMiniBatch($\mathcal{D}_{meta}$).

5: $\\{\mathbf{w}_{inst}^{t}\\}\leftarrow$ SampleInstanceParameters.

6: Update model parameters, and express $\theta^{t+1}$ as a function of
$\mathbf{w}_{inst}^{t}$ by Eq. (3).

7: Update $\mathbf{w}_{inst}^{t+1}$ by Eq. (6).

8: end for

#### Analysis of meta-gradient on instance level parameters

From equation 5, we can observe that the meta-gradient on instance parameter
$\mathbf{w}_{inst}^{i,t}$ is proportional to the dot product of $i^{th}$
training sample’s gradient on model parameters at time step $t$ with meta-
gradient on model parameters of samples in the meta set at time step $t+1$.
Therefore, training samples whose gradient aligns with the gradients on meta-
set will obtain a higher weight, leading to an increased learning rate. The
converse holds true as well. For example, if an instance in train set has
wrong label, its gradient will not align with gradient of clean instances in
the meta set. Over the course of learning, the corrupt instance will end up
obtaining a lower value of instance parameter.

###  2.2 Learning class level learning rate

While instance parameters have the flexibility to adapt to each instance
present in the dataset, number of parameters grow with the size of dataset. To
alleviate this issue, another way we can partition a dataset is by leveraging
the class membership of data points. More specifically, we can learn a
learning rate for each class, shared by all the instances present within the
class. Let $\mathbf{w}_{class}^{t}\in\mathbb{R}^{N}$ denote the class
parameters at time step $t$. Similar to equation (3), we can write the one
step look ahead update as a function of class parameters, and use it compute
the meta-gradient on the class parameters:

$\displaystyle\frac{\partial\mathcal{L}_{meta}({\theta_{t+1}}^{*})}{\partial\mathbf{w}^{c,t}_{class}}$
$\displaystyle=-\frac{\lambda.n_{c}}{N}\cdot\frac{\partial\mathcal{L}_{meta}({\theta^{t+1}}^{*})}{\partial{\theta_{t+1}}^{*}}\cdot\big{[}\frac{1}{n_{c}}{\sum\limits_{\begin{subarray}{c}i\\\
y_{i}=c\end{subarray}}^{n_{c}}\frac{\partial
L_{train}^{i}}{\partial\theta_{t}}}\big{]}^{T}$ (7)

Here, $w^{c,t}_{class}$ denotes the weight for $c$ (target class for $i^{th}$
train data point), and $n_{c}$ denotes number of samples in class $c$. As seen
in equation 7, the meta-gradient on class parameters is proportional to the
dot-product of meta-gradient on model parameters with gradient on model
parameters from train set, averaged over instances belonging to class $c$. We
provide detailed derivation in supplementary material.

###  2.3 Meta-learning weight decay regularization

Use of weight-decay as a regularizer is a de-facto standard in training Deep
Neural Networks (DNNs). However, the exact role weight-decay plays in
optimization of modern DNNs is not well understood [8, 13, 24, 39]. In this
section, we highlight the importance of learning the weight-decay coefficient
along with the learning rate for dataset, class or instances. For ease of
explanation, let us consider the case where we are interested in learning
instance level learning rate, and we have the standard weight-decay term added
as a regularizer.

$\displaystyle\mathcal{L}_{train}(\theta^{t},w_{dataset}^{t})$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\mathbf{w}^{i,t}_{inst}\cdot
L_{train}^{i}(\theta^{t})+\lambda_{wd}\|\theta^{t}\|$ (8)

Here $\lambda_{wd}$ denotes the weight-decay coefficient. During the course of
optimization, regardless of the magnitude of the first term, the contribution
of the second term in gradient update is fixed. In our experiments, we found
this to be problematic. When meta-gradient reduces the magnitude of instance
parameter $\mathbf{w}^{i,t}_{inst}$, it leads to a relative increase of
weight-decay component in the gradient update. This leads to destablization of
the training. We solved this problem by treating the weight-decay coefficient
as a learnable parameter, which is learnt along with the learning rate of
class and instances using meta learning setup.

##  3 Experiments

###  3.1 Implementation details

Unless stated otherwise, the following implementation details hold true for
all experiments in the paper. We use SGD optimizer (without momentum and
weight-decay) to learn instance and class level learning rates. We use same
batch-size to sample batches from train-set and meta-set. Apart from clamping
negative learning-rates to 0, we do not employ any form of regularization, and
rely on the meta-gradient to regularize the learning process. We perform
$k$-fold cross validation, where the held-out set is used as both: meta-set
and validation-set (for picking best configuration). For reporting final
numbers, we average out dynamic learning rate trajectory for each class and
instance, and use it to train on the full train-set. We ensure all methods use
the same amount of training data. We report mean and standard-deviation
computed over 3 runs.

###  3.2 Image Classification on CIFAR100

In this section, we show efficacy of our optimization framework when applied
to the task of image classification on CIFAR100 [21] dataset. CIFAR100 dataset
contains 100 classes, 50,000 images in the train set and 10,000 images in the
test set. Therefore, in our framework, along with the model parameters, we
learn 100 and 50,000 dynamic learning rates for class and instances
respectively. We evaluate our framework with ResNet18 [11], VGG16 [35]. We use
standard setup for training both architectures (details in supplementary).

In Figure 2, we compare our optimization framework to other optimizers
commonly used in the deep learning community. Similar to results in [40], when
tuned appropriately, apart from ADAM, all other optimizers obtain performance
comparable to SGD. In both settings, learning class or instance level learning
rate, we outperform these standard optimizers by a significant margin. These
gains over standard optimizers can be attributed to the fact that our
framework adapts the optimization process over samples in dataset instead of
model-parameters.

Across both architectures, learning instance level learning rates performs
more favorably compared to learning class level learning rates. This validates
our hypothesis: loss functions for samples within a class might have different
characteristics, and might benefit from learning sample specific learning
rates. However, class level parameters get more frequent updates compared to
instance level parameters, and hence can achieve faster convergence (see
Figure 2, middle).

| ResNet18 | VGG16
---|---|---
SGD | 77.5 $\pm$ 0.0 | 75.4 $\pm$ 0.2
Momentum [30] | 77.1 $\pm$ 0.3 | 74.1 $\pm$ 0.2
Adam [20] | 74.4 $\pm$ 0.2 | 72.2 $\pm$ 0.5
AdamW [24] | 77.4 $\pm$ 0.1 | 74.5 $\pm$ 0.2
Polyak [31] | 78.0 $\pm$ 0.3 | 74.5 $\pm$ 0.3
LookAhead [40] | 77.2 $\pm$ 0.4 | 75.6 $\pm$ 0.2
Instance-level | 78.6 $\pm$ 0.2 | 76.5 $\pm$ 0.1
Class-level | 78.3 $\pm$ 0.2 | 76.2 $\pm$ 0.2

Figure 2: Table: Comparison of class and instance level optimizer with
different optimizers on CIFAR100. We do a grid search on learning rate and
weight-decay for other optimizers (see supplementary). Lookahead and Polyak
are wrapped around SGD. Figures: Test and train accuracy for different
optimizers for VGG16. Using our optimizer leads to reduced variance, and
better generalization.

###  3.3 Analysis of optimization framework

In this section, we will analyze different components of our optimization
framework.

#### Variance reduction via dynamic learning rates on data

One key hypothesis of our work is: loss functions for different data points
can have different characteristics, and hence might benefit from different
learning rates. In Figure 3 we empirically verify this property on CIFAR100,
using class-level learning rates for training VGG16. In Figure 3 (A, B) we
plot the train loss and test accuracy for the best (green) and the worst (red)
performing class at convergence. Shared learning rate used by SGD works well
for the green class, but is not able to optimize the red class (until learning
rate decay at epoch 150). In contrast, our method accounts for class
performance on the meta-set, and reduces the learning rate for each class in
proportion to that (see Figure 3, C and D). This result provides an
interesting view to our method from the perspective of variance reduction. In
contrast to standard setting where methods have been proposed to reduce
variance in the gradient estimator for the entire mini-batch, our work
performs selective variance reduction. Lowering the learning rate for classes
with worse performance will lower the overall variance in gradient estimator,
leading to faster and stable convergence. In light of recent work [6], which
shows ineffectiveness of standard variance reduction framework in deep
learning, our results indicate that performing selective variance reduction
could be an interesting direction to explore.

Figure 3: To speed up convergence, our framework reduces learning rate on
classes with worse performance on meta set (see text for details). Learning
dynamics for class with best (green) and worst (red) performance at
convergence. A:Train loss. B: Test accuracy. C: Dynamic learning rate for the
classes, along with mean learning rate for all classes (black). D: At each
epoch, we plot the correlation of learning rate of a class with their
performance on heldout meta-set and train-set.

#### Importance of history

| History | ResNet18 | VGG16
---|---|---|---
Instance | ✓ | 78.6 $\pm$ 0.2 | 76.5 $\pm$ 0.1
✗ | 77.8 $\pm$ 0.2 | 75.4 $\pm$ 0.2
Class | ✓ | 78.3 $\pm$ 0.2 | 76.2 $\pm$ 0.2
✗ | 77.8 $\pm$ 0.1 | 75.7 $\pm$ 0.1

Table 1: Impact of retaining history on CIFAR100 for image-classification.

As mentioned earlier, learning learning rates on data points can be
interpreted as learning a weighting on them. Some recent works [16, 32, 34,
38] have used meta-learning based approaches to dynamically assign weights to
instances. In general, these works [16, 34, 38] train a secondary neural
network to assign weights to data-points throughout the course of training, or
[32] perform an online approximation of weights for each sample in the mini-
batch. These frameworks are Markovian in nature, since weights estimated at
each time step are independent of past predictions. In contrast, our framework
treats learning rates on data as learnable parameters, and benefits from past
history of optimization. In Table 1, we establish the importance of retaining
optimization history, by evaluating our framework without the use of history.
Specifically, after each time step, we update the model parameters using the
updated value of instance and class learning rates. Post model parameter
update, we reset the instance and class learning rates to their initial value
of 1. As shown in Table 1, not reusing the history of learnt learning rates
leads to a significant drop in performance across all settings and
architectures on CIFAR100. Another place of comparison with this prior work is
in noisy setting (see Section 3.4), where we outperform these methods by a
significant margin.

#### Importance of learning a dynamic weight-decay

Recently, the importance of weight-decay in optimization of DNNs has gained
much interest [8, 13, 24, 39]. [8] empirically demonstrate that weight-decay
plays an important role in the first few epochs, and does not play as much a
role in the later stages of training. Our results indicates otherwise. In this
work, we show that learning a dynamic weight-decay leads to significant change
in SGD dynamics and also facilitates the learning of learning rates on data.
Figure 4 highlights the change in dynamics of SGD optimizer when weight-decay
is learnt along with the model parameters. Compared to baseline (fixed weight-
decay), the learnt weight-decay adapts to different stages of optimization
(see Figure 4, right). More specifically, post learning rate drop, when model
is most prone to overfitting, dynamic weight decay coefficient increases. This
leads to a temporary drop in performance, but results in better generalization
at convergence. As see in the table (Figure 4, left), learning a dynamic
weight-decay improves performance of all three optimizers.

Dynamic weight decay | ✗ | ✓
---|---|---
SGD | 77.5 $\pm$ 0.0 | 78.0 $\pm$ 0.1
Class-level | 77.7 $\pm$ 0.1 | 78.3 $\pm$ 0.2
Instance-level | 78.2 $\pm$ 0.1 | 78.6 $\pm$ 0.2

Figure 4: Evaluation of dynamic weight-decay on CIFAR100 with ResNet18. Left:
Dynamic weight-decay improves all three optimizers. Center: Train (dashed) and
validation (solid) accuracy for SGD optimizer, with fixed and dynamic weight-
decay. Right: Plot showing the value of dynamic weight-decay when learnt along
with SGD. When the learning rate drops (epoch 80, 120), weight-decay
coefficient increases so as to counteract overfitting, and obtains better
performance at convergence.

###  3.4 Results on Robust Learning

Learning instance-level learning rates can be useful when some of the labels
in the dataset are noisy, where the framework should decrease learning rates
on corrupt instances. In this section, we validate our framework in a
controlled corrupted label setting.

To compare with the relevant state-of-the-art, we follow the common setting in
([16, 32]) to train deep CNNs, where the label of each image is independently
changed to a uniform random class with probability $p$, where $p$ is noise
fraction. The labels of validation data remain clean for evaluation. We
compare our approach with recent state-of-the-art approaches in this setting
[16, 32, 33, 34]. MentorNet [16] and Meta-Weight-Net [34] train an auxillary
neural network to assign weights to samples in the mini-batch. L2RW [32] uses
a held-out set to perform an online approximation of weights for samples in
the mini-batch. Data parameters [33] introduced learnable temperature
parameters per data-point, which scale the gradient contribution of each data
point. Unfortunately, all of these works report results in two distinct
settings: setting A [32] and setting B [34]. While both settings use
WRN-28-10, they differ in learning rate schedules (see details in
supplementary). To make a fair comparion, we report results in both settings.
Similar to setup in [32, 34], we keep 1000 clean images as meta-set. As seen
in Figure 5, our method outperforms other state-of-the-art methods in both
settings under different levels of noise.

| 40% | 60%
---|---|---
Baseline [34] | $51.1\pm 0.4$ | $30.9\pm 0.3$
Focal Loss [23] | $51.2\pm 0.5$ | $27.7\pm 3.7$
Co-teaching [10] | $46.20\pm 0.15$ | $35.7\pm 1.2$
Using 1000 clean images
MentorNet [16] | $61.4\pm 4.0$ | $36.9\pm 1.5$
L2RW [32] | $60.8\pm 0.9$ | $48.2\pm 0.3$
MWNet [34] | $67.7\pm 0.3$ | $58.8\pm 0.1$
Ours (instance-level) | $\bf{69.0\pm 0.2}$ | $\bf{59.6\pm 0.3}$

Setting A

| 40% | 80%
---|---|---
Baseline [32] | $50.66\pm 0.24$ | $8.0$
MentorNet PD [16] | $56.9$ | $14.0$
Data Parameters [33] | $\bf{70.93\pm 0.15}$ | $35.8\pm 1.0$
Using 1000 clean images
MentorNet DD [16] | $67.5$ | $35.0$
L2RW [32] | $61.34\pm 2.06$ | -
Ours (instance-level) | $70.8\pm 0.1$ | $\bf{42.5\pm 0.3}$

Setting B

Figure 5: Tables: On CIFAR100, our methods outperforms state-of-the-art under
varying levels of uniform noise in two settings (see text for details).
Figure: Plot of mean and standard-deviation over instance-level learning rates
for clean and corrupts instances. Our framework learns to reduce the learning
rate on corrupt instances.

###  3.5 Personalizing DNN models

In a traditional setting, machine learning models are trained under empirical
risk minimization (ERM) framework, where train and test set are assumed to be
sampled from the same distribution. These models are optimized to work well
across the entire data distribution. However, this training setup would not be
ideal when at test time only a subset of train distribution is of interest.
This situation can come up in various practical problems: (1) targeting a
certain demographic for recommender systems [28], (2) personalizing models in
health for a certain anomaly or demographic [14, 27], etc. In this setting, an
important question one needs to answer is: What training data should I train
the model on?

We simulate this scenario using the CIFAR100 dataset, which contains 100 fine
grained classes, and 20 super classes (mutually disjoint, contains 5 classes).
In this scenario, despite having the entire dataset annotated, we are
interested in one super class at test time. Below we detail different methods
which can be used in this scenario along with our proposed solution.

Biased training: The problem can be reduced to ERM framework by training the
model on instances belonging to classes present in the super class. This
approach makes an assumption that the other classes present in the dataset are
completely disjoint from the super class. However, some of the discarded
classes might share common low-level features which might be useful to train
the early layers of deep neural network.

Full training: To address the aforementioned limitation, and take advantage of
all the annotated data, one can train the model on all classes of the CIFAR100
dataset. However, this approach makes an assumption that training on all
classes would be beneficial for the classes present in super class.

Transfer Learning: Train model on all 100 classes (full training), followed by
training on the classes present in super class (biased training). A limitation
of this approach is that pretraining model on all 100 classes might bias the
model.

Our solution: The limitation of approaches mentioned above lies in the fact
that it involves making a hard choice regarding the classes present in the
training set. We relax this constraint by using dynamic class-level learning
rates, which can guide the optimization process towards a biased subset
dynamically. The meta set is comprised of instances belonging to the super-
class.

We benchmark our method and the baselines in table below on the CIFAR100
dataset. As seen in table in Figure 6, using our proposed solution we
outperform the other baselines by a significant margin.

Super Class | People | Aquatic | Vehicle 1 | Electrical | Reptiles
---|---|---|---|---|---
Mammals | Devices
Biased Training | $54.1\pm 1.3$ | $69.7\pm 4.5$ | $91.1\pm 0.5$ | $87.0\pm 0.5$ | $77.0\pm 0.5$
Full Training | $55.5\pm 1.6$ | $61.1\pm 1.4$ | $84.7\pm 0.5$ | $77.5\pm 1.1$ | $65.1\pm 1.4$
Transfer Learning | $54.6\pm 1.4$ | $66.4\pm 4.1$ | $91.0\pm 0.2$ | $87.7\pm 0.7$ | $76.2\pm 0.3$
Ours (class-level) | $\bf{61.0\pm 1.2}$ | $\bf{70.7\pm 0.3}$ | $\bf{93.0\pm 0.5}$ | $\bf{88.1\pm 0.8}$ | $\bf{79.5\pm 0.7}$

Figure 6: Table: For the task of personalization towards a super-class, using
dynamic class level learning rates outperforms baselines which involve a
static choice of train data. Figure: Repetable trajectories of learning rate
for classes not present in the super class.

##  4 Related Work

Optimization of DNNs has gained a lot of interest recently [1, 13, 17, 19, 24,
26, 40] . While a full detailed review is beyond the scope of this paper,
here, we give a brief overview of related work most relevant to the material
present in the paper.

Learning adaptive learning rates on data can be interpreted as learning a
weighting on each data point. [16, 34, 38] train an auxillary neural network
to assign weights to data points. [32] performs an online approximation, where
it uses one step look ahead on meta-set to estimate the weights for samples in
the mini-batch. These approaches are Markovian in nature, since weights
estimated at each time step are independent of past-predictions. In contrast,
our method leverages the past history of optimization, and outperforms these
state-of-the-art methods for robust learning (see Section 3.4).

Our work also has connections to importance sampling. [15, 17, 18] propose
approximations for the gradient norm of instances which are used for sampling
data points. Theoretically, sampling data points with high gradient norm
should lead to faster convergence. However, this would not work well in real
world dataset which contain noisy data. In the same spirit, our work also has
connections to the field of curriculum learning [4, 9, 36] and self-paced
learning [7, 22, 29, 37]. These approaches either design hand-crafted
heuristics, or use loss value of data points as a proxy to decide the ordering
of data. All these approaches require coming up with a heuristics which might
not work from one problem domain to another. In contrast, our work can be
interpreted as a soft differentiable form of importance sampling, where the
importance of a sample (learning rate) or curriculum is learnt through meta
gradient.

Data parameters [33] introduced learnable temperature parameters for each
instance and class in the dataset. These parameters controlled the gradient
contribution for each data point, and were learnt using gradient descent. In
similar spirit, [2] introduced learnable robustness parameter per data point,
which generalized different regression losses. Both of these works learn
parameters per data point for robust estimation in classification or
regression. In comparison, our formulation is more generic and can admit any
differentiable loss function. More importantly, due to its meta-learning
framework, our approach allows for robust estimation of noise as well as
personalization.

Other work has proposed optimizing a few hyperparameters, such as kernel
parameters [5], weight decay [3] or others [25], using gradients during
training. However, to the best of our knowledge, none have done so for
learning dynamic learning rates across training per se, nor done so at scale
(e.g. one rate per instance) to achieve state-of-the-art performance.

##  5 Conclusion

In this paper, we have proposed an optimization framework which accounts for
differences in loss function characteristics across instances and classes
present in the dataset. More specifically, our framework learns a dynamic
learning rate for each instance and class present in the dataset. Learning a
dynamic learning rate allows our framework to focus on different modes of
training data. For instance, when presented with noisy dataset, our framework
reduces the learning rate on noisy instances, and focuses on optimizing model
parameters using clean instances. When applied for the task of image-
classification, across different CNN architectures, our framework outperforms
standard optimizers. Finally, for the task of personalization of machine
learning models towards a known data distribution, our framework outperforms
strong baselines.

## References

  * [1] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  In NIPS, 2016.
  * [2] Jonathan T Barron.  A general and adaptive robust loss function.  In CVPR, 2019.
  * [3] Yoshua Bengio.  Gradient-based optimization of hyperparameters.  Neural computation, 2000.
  * [4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In ICML, 2009.
  * [5] Olivier Chapelle, Vladimir Vapnik, Olivier Bousquet, and Sayan Mukherjee.  Choosing multiple parameters for support vector machines.  Machine learning, 2002.
  * [6] Aaron Defazio and Léon Bottou.  On the ineffectiveness of variance reduced optimization for deep learning.  In NeurIPS, 2019.
  * [7] Yanbo Fan, Ran He, Jian Liang, and Baogang Hu.  Self-paced learning: an implicit regularization perspective.  In AAAI, 2017.
  * [8] Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto.  Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence.  In NIPS, 2019.
  * [9] Guy Hacohen and Daphna Weinshall.  On the power of curriculum learning in training deep networks.  arXiv, 2019.
  * [10] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In NeurIPS, 2018.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CPVR, 2016.
  * [12] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky.  Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.
  * [13] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry.  Norm matters: efficient and accurate normalization schemes in deep networks.  In NIPS, 2018.
  * [14] Natasha Jaques, Sara Taylor, Akane Sano, Rosalind Picard, et al.  Predicting tomorrow’s mood, health, and stress level using personalized multitask learning and domain adaptation.  In IJCAI Workshop on artificial intelligence in affective computing, 2017.
  * [15] Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C Lipton, et al.  Accelerating deep learning by focusing on the biggest losers.  arXiv, 2019.
  * [16] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In ICML, 2018.
  * [17] Tyler B Johnson and Carlos Guestrin.  Training deep models faster with robust, approximate importance sampling.  In NeurIPS, 2018.
  * [18] Angelos Katharopoulos and Francois Fleuret.  Not all samples are created equal: Deep learning with importance sampling.  In ICML, 2018.
  * [19] Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, and Sham Kakade.  On the insufficiency of existing momentum schemes for stochastic optimization.  In ICLR, 2018.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [21] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * [22] Yong Jae Lee and Kristen Grauman.  Learning the easy things first: Self-paced visual category discovery.  In CVPR, 2011.
  * [23] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [24] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2019.
  * [25] Dougal Maclaurin, David Duvenaud, and Ryan Adams.  Gradient-based hyperparameter optimization through reversible learning.  In ICML, 2015.
  * [26] James Martens and Roger Grosse.  Optimizing neural networks with kronecker-factored approximate curvature.  In ICML, 2015.
  * [27] Kaare Mikkelsen and Maarten De Vos.  Personalizing deep learning models for automatic sleep staging.  arXiv, 2018.
  * [28] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G Azzolini, et al.  Deep learning recommendation model for personalization and recommendation systems.  arXiv, 2019.
  * [29] Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, and Yueting Zhuang.  Self-paced boost learning for classification.  In IJCAI, 2016.
  * [30] Boris T Polyak.  Some methods of speeding up the convergence of iteration methods.  USSR Computational Mathematics and Mathematical Physics, 1964.
  * [31] Boris T Polyak and Anatoli B Juditsky.  Acceleration of stochastic approximation by averaging.  SIAM journal on control and optimization, 1992.
  * [32] Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun.  Learning to reweight examples for robust deep learning.  In ICML, 2018.
  * [33] Shreyas Saxena, Oncel Tuzel, and Dennis DeCoste.  Data parameters: A new family of parameters for learning a differentiable curriculum.  In NeurIPS, 2019.
  * [34] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng.  Meta-weight-net: Learning an explicit mapping for sample weighting.  In NeurIPS, 2019.
  * [35] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In ICLR, 2015.
  * [36] Valentin I Spitkovsky, Hiyan Alshawi, and Daniel Jurafsky.  Baby steps: How “less is more” in unsupervised dependency parsing.  In NIPS, 2009.
  * [37] James S Supancic and Deva Ramanan.  Self-paced learning for long-term tracking.  In CVPR, 2013.
  * [38] Xinyi Wang, Hieu Pham, Paul Michel, Antonios Anastasopoulos, Graham Neubig, and Jaime Carbonell.  Optimizing data usage via differentiable rewards.  arXiv, 2019.
  * [39] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse.  Three mechanisms of weight decay regularization.  In ICLR, 2019.
  * [40] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton.  Lookahead optimizer: k steps forward, 1 step back.  In NeurIPS, 2019.