# Deep Architectures for Neural Machine Translation

Antonio Valerio Miceli Barone† Jindřich Helcl⋆ Rico Sennrich†
Barry Haddow† Alexandra Birch†
†School of Informatics, University of Edinburgh
⋆Faculty of Mathematics and Physics, Charles University
{amiceli<EMAIL_ADDRESS>
{rico.sennrich<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

It has been shown that increasing model depth improves the quality of neural
machine translation. However, different architectural variants to increase
model depth have been proposed, and so far, there has been no thorough
comparative study.

In this work, we describe and evaluate several existing approaches to
introduce depth in neural machine translation. Additionally, we explore novel
architectural variants, including deep transition RNNs, and we vary how
attention is used in the deep decoder. We introduce a novel "BiDeep" RNN
architecture that combines deep transition RNNs and stacked RNNs.

Our evaluation is carried out on the English to German WMT news translation
dataset, using a single-GPU machine for both training and inference. We find
that several of our proposed architectures improve upon existing approaches in
terms of speed and translation quality. We obtain best improvements with a
BiDeep RNN of combined depth 8, obtaining an average improvement of 1.5 Bleu
over a strong shallow baseline.

We release our code for ease of adoption.

##  1 Introduction

Neural machine translation (NMT) is a well-established approach that yields
the best results on most language pairs (Bojar et al., 2016; Cettolo et al.,
2016). Most systems are based on the sequence-to-sequence model with attention
(Bahdanau et al., 2015) which employs single-layer recurrent neural networks
both in the encoder and in the decoder.

Unlike feed-forward networks where depth is straightforwardly defined as the
number of non-input layers, recurrent neural network architectures with
multiple layers allow different connection schemes (Pascanu et al., 2014) that
give rise to different, orthogonal, definitions of depth (Zhang et al., 2016)
which can affect the model performance depending on a given task. This is
further complicated in sequence-to-sequence models as they contain multiple
sub-networks, recurrent or feed-forward, each of which can be deep in
different ways, giving rise to a large number of possible configurations.

In this work we focus on stacked and deep transition recurrent architectures
as defined by Pascanu et al. (2014). Different types of stacked architectures
have been successfully used for NMT (Zhou et al., 2016; Wu et al., 2016).
However, there is a lack of empirical comparisons of different deep
architectures. Deep transition architectures have been successfully used for
language modeling (Zilly et al., 2016), but not for NMT so far. We evaluate
these architectures, both alone and in combination, varying the connection
scheme between the different components and their depth over the different
dimensions, measuring the performance of the different configurations on the
WMT news translation task.111http://www.statmt.org/wmt17/translation-task.html

Related work includes that of Britz et al. (2017), who have performed an
exploration of NMT architectures in parallel to our work. Their experiments,
which are largely orthogonal to ours, focus on embedding size, RNN cell type
(GRU vs. LSTM), network depth (defined according to the architecture of Wu et
al. (2016)), attention mechanism and beam size. Gehring et al. (2017) recently
proposed a NMT architecture based on convolutions over fixed-sized windows
rather than RNNs, and they reported results for different model depths and
attention mechanism configurations. A similar feedforward architecture which
uses multiple pervasive attention mechanisms rather than convolutions was
proposed by Vaswani et al. (2017), who also report results for different model
depths.

##  2 NMT Architectures

All the architectures that we consider in this work are GRU (Cho et al.,
2014a) sequence-to-sequence transducers (Sutskever et al., 2014; Cho et al.,
2014b) with attention (Bahdanau et al., 2015). In this section we describe the
baseline system and the variants that we evaluated.

###  2.1 Baseline Architecture

As our baseline, we use the NMT architecture implemented in Nematus, which is
described in more depth by Sennrich et al. (2017b). We augment it with layer
normalization (Ba et al., 2016), which we have found to both improve
translation quality and make training considerably faster.

For our discussion, it is relevant that the baseline architecture already
exhibits two types of depth:

  * •

recurrence transition depth in the decoder RNN which consists of two GRU
transitions per output word with an attention mechanism in between, as
described in Firat and Cho (2016).

  * •

feed-forward depth in the attention network that computes the alignment scores
and in the output network that predicts the target words. Both these networks
are multi-layer perceptrons with one tanh hidden layer.

###  2.2 Deep Transition Architectures

In a deep transition RNN (DT-RNN), at each time step the next state is
computed by the sequential application of multiple transition layers,
effectively using a feed-forward network embedded inside the recurrent cell.
In our experiments, these layers are GRU transition blocks with independently
trainable parameters, connected such that the "state" output of one of them is
used as the "state" input of the next one. Note that each of these GRU
transition is not individually recurrent, recurrence only occurs at the level
of the whole multi-layer cell, as the "state" output of the last GRU
transition for the current time step is carried over as the "state" input of
the first GRU transition for the next time step.

Applying this architecture to NMT is a novel contribution.

####  2.2.1 Deep Transition Encoder

As in a baseline shallow Nematus system, the encoder is a bidirectional
recurrent neural network. Let $L_{s}$ be the encoder recurrence depth, then
for the $i$-th source word in the forward direction the forward source word
state $\overrightarrow{h}_{i}\equiv\overrightarrow{h}_{i,L_{s}}$ is computed
as:

$\displaystyle\overrightarrow{h}_{i,1}$
$\displaystyle=\text{GRU}_{1}\left(x_{i},\overrightarrow{h}_{i-1,L_{s}}\right)$
$\displaystyle\overrightarrow{h}_{i,k}$
$\displaystyle=\text{GRU}_{k}\left(0,\overrightarrow{h}_{i,k-1}\right)\text{for
}1<k\leq L_{s}$

where the input to the first GRU transition is the word embedding $x_{i}$,
while the other GRU transitions have no external inputs. Recurrence occurs as
the previous word state $\overrightarrow{h}_{i-1,L_{s}}$ enters the
computation in the first GRU transition for the current word.
The reverse source word states are computed similarly and concatenated to the
forward ones to form the bidirectional source word states
$C\equiv\left\\{\left[\overrightarrow{h}_{i,L_{s}}\overleftarrow{h}_{i,L_{s}}\right]\right\\}$.

####  2.2.2 Deep Transition Decoder

$\ldots$$\ldots$$\ldots$ Figure 1: Deep transition decoder

The deep transition decoder is obtained by extending the baseline decoder in a
similar way. Recall that the baseline decoder of Nematus already has a
transition depth of two, with the first GRU transition receiving as input the
embedding of the previous target word and the second GRU transition receiving
as input a context vector computed by the attention mechanism. We extend this
decoder architecture to an arbitrary transition depth $L_{t}$ as follows:

$\displaystyle s_{j,1}$
$\displaystyle=\text{GRU}_{1}\left(y_{j-1},s_{j-1,L_{t}}\right)$
$\displaystyle s_{j,2}$
$\displaystyle=\text{GRU}_{2}\left(\text{ATT}(C,s_{j,1}),s_{j,1}\right)$
$\displaystyle s_{j,k}$
$\displaystyle=\text{GRU}_{k}\left(0,s_{j,k-1}\right)\text{for }2<k\leq L_{t}$

where $y_{j-1}$ is the embedding of the previous target word and
$\text{ATT}(C,s_{i,1})$ is the context vector computed by the attention
mechanism. GRU transitions other than the first two do not have external
inputs. The target word state vector $s_{j}\equiv s_{j,L_{t}}$ is then used by
the feed-forward output network to predict the current target word. A diagram
of this architecture is shown in Figure 1.

The output network can be also made deeper by adding more feed-forward hidden
layers.

###  2.3 Stacked architectures

A stacked RNN is obtained by having multiple RNNs (GRUs in our experiments)
run for the same number of time steps, connected such that at each step the
bottom RNN takes "external" inputs from the outside, while each of the higher
RNN takes as its "external" input the "state" output of the one below it.
Residual connections between states at different depth (He et al., 2016) are
also used to improve information flow. Note that unlike deep transition GRUs,
here each GRU transition block constitutes a cell that is individually
recurrent, as it has its own state that is carried over between time steps.

####  2.3.1 Stacked Encoder

In this work we consider two types of bidirectional stacked encoders: an
architecture similar to Zhou et al. (2016) which we denote here as alternating
encoder (Figure 2), and one similar to Wu et al. (2016) which we denote as
biunidirectional encoder (Figure 3).

Our contribution is the empirical comparison of these architectures, both in
isolation and in combination with the deep transition architecture.

We do not consider stacked unidirectional encoders (Sutskever et al., 2014) as
bidirectional encoders have been shown to outperform them (e.g. Britz et al.
(2017)).

##### Alternating Stacked Encoder

$\ldots$$\ldots$$\ldots$$\ldots$$\ldots$$\ldots$$\ldots$ Figure 2: Alternating
stacked encoder (Zhou et al., 2016).

The forward part of the encoder consists of a stack of GRU recurrent neural
networks, the first one processing words in the forward direction, the second
one in the backward direction, and so on, in alternating directions. For an
encoder stack depth $D_{s}$, and a source sentence length $N$, the forward
source word state $\overrightarrow{w}_{i}\equiv\overrightarrow{w}_{i,D_{s}}$
is computed as:

$\displaystyle\overrightarrow{w}_{i,1}$
$\displaystyle=\overrightarrow{h}_{i,1}=\text{GRU}_{1}\left(x_{i},\overrightarrow{h}_{i-1,1}\right)$
$\displaystyle\overrightarrow{h}_{i,2k}$
$\displaystyle=\text{GRU}_{2k}\left(\overrightarrow{w}_{i,2k-1},\overrightarrow{h}_{i+1,2k}\right)$
$\displaystyle\text{ for }1<2k\leq D_{s}$
$\displaystyle\overrightarrow{h}_{i,2k+1}$
$\displaystyle=\text{GRU}_{2k+1}\left(\overrightarrow{w}_{i,2k},\overrightarrow{h}_{i-1,2k+1}\right)$
$\displaystyle\text{ for }1<2k+1\leq D_{s}$
$\displaystyle\overrightarrow{w}_{i,j}$
$\displaystyle=\overrightarrow{h}_{i,j}+\overrightarrow{w}_{i,j-1}$
$\displaystyle\text{ for }1<j\leq D_{s}$

where we assume that $\overrightarrow{h}_{0,k}$ and
$\overrightarrow{h}_{N+1,k}$ are zero vectors. Note the residual connections:
at each level above the first one, the word state of the previous level
$\overrightarrow{w}_{i,j-1}$ is added to the recurrent state of the GRU cell
$\overrightarrow{h}_{i,j}$ to compute the the word state for the current level
$\overrightarrow{w}_{i,j}$.

The backward part of the encoder has the same structure, except that the first
level of the stack processes the words in the backward direction and the
subsequent levels alternate directions.

The forward and backward word states are then concatenated to form
bidirectional word states
$C\equiv\left\\{\left[\overrightarrow{w}_{i,D_{s}}\overleftarrow{w}_{i,D_{s}}\right]\right\\}$.
A diagram of this architecture is shown in Figure 2.

##### Biunidirectional Stacked Encoder

$\ldots$$\ldots$$\ldots$$\ldots$$\ldots$ Figure 3: Biunidirectional stacked
encoder (Wu et al., 2016).

In this encoder the forward and backward parts are shallow, as in the baseline
architecture. Their word states are concatenated to form shallow bidirectional
word states
$w_{i}\equiv\left[\overrightarrow{w}_{i,1}\overleftarrow{w}_{i,1}\right]$ that
are then used as inputs for subsequent stacked GRUs which operate only in the
forward sentence direction, hence the name "biunidirectional". Since residual
connections are also present, the higher depth GRUs have a state size twice
that of the base ones. This architecture has shorter maximum information
propagation paths than the alternating encoder, suggesting that it may be less
expressive, but it has the advantage of enabling implementations with higher
model parallelism. A diagram of this architecture is shown in Figure 3.

In principle, alternating and biunidirectional stacked encoders can be
combined by having $D_{sa}$ alternating layers followed by $D_{sb}$
unidirectional layers.

####  2.3.2 Stacked Decoder

$\ldots$$\ldots$$\ldots$ Figure 4: Stacked RNN decoder

A stacked decoder can be obtained by stacking RNNs which operate in the
forward sentence direction. A diagram of this architecture is shown in Figure
4.

Note that the base RNN is always a conditional GRU (cGRU, Firat and Cho, 2016)
which has transition depth at least two due to the way that the context
vectors generated by the attention mechanism are used in Nematus. This opens
up the possibility of several architectural variants which we evaluated in
this work:

##### Stacked GRU

The higher RNNs are simple GRUs which receive as input the state from the
previous level of the stack, with residual connections between the levels.

$\displaystyle s_{j,1,1}$
$\displaystyle=\text{GRU}_{1,1}\left(y_{j-1},s_{j-1,1,2}\right)$
$\displaystyle c_{j,1}$ $\displaystyle=\text{ATT}(C,s_{j,1,1})$ $\displaystyle
s_{j,1,2}$ $\displaystyle=\text{GRU}_{1,2}\left(c_{j,1},s_{j,1,1}\right)$
$\displaystyle r_{j,1}$ $\displaystyle=s_{j,1,2}$ $\displaystyle s_{j,k,1}$
$\displaystyle=\text{GRU}_{k}\left(r_{j,k-1},s_{j-1,k,1}\right)$
$\displaystyle r_{j,k}$ $\displaystyle=s_{j,k,1}+r_{j,k-1}$
$\displaystyle\text{for }1<k\leq D_{t}$

Note that the higher levels have transition depth one, unlike the base level
which has two.

##### Stacked rGRU

The higher RNNs are GRUs whose "external" input is the concatenation of the
state below and the context vector from the base RNN. Formally, the states
$s_{j,k,1}$ of the higher RNNs are computed as:

$\displaystyle s_{j,k,1}$
$\displaystyle=\text{GRU}_{k}\left(\left[r_{j,k-1},c_{j,1}\right],s_{j-1,k,1}\right)$
$\displaystyle\text{for }1<k\leq D_{t}$

This is similar to the deep decoder by Wu et al. (2016).

##### Stacked cGRU

The higher RNNs are conditional GRUs, each with an independent attention
mechanism. Each level has two GRU transitions per step $j$, with a new context
vector $c_{j,k}$ computed in between:

$\displaystyle s_{j,k,1}$
$\displaystyle=\text{GRU}_{k,1}\left(r_{j,k-1},s_{j-1,k,1}\right)$
$\displaystyle c_{j,k}$ $\displaystyle=\text{ATT}(C,s_{j,k,1})$ $\displaystyle
s_{j,k,2}$ $\displaystyle=\text{GRU}_{k,2}\left(c_{j,k},s_{j,1,1}\right)$
$\displaystyle\text{for }1<k\leq D_{t}$

Note that unlike the stacked GRU and rGRU, the higher levels have transition
depth two.

##### Stacked crGRU

The higher RNNs are conditional GRUs but they reuse the context vectors from
the base RNN. Like the cGRU there are two GRU transition per step, but they
reuse the context vector $c_{j,1}$ computed at the first level of the stack:

$\displaystyle s_{j,k,1}$
$\displaystyle=\text{GRU}_{k,1}\left(r_{j,k-1},s_{j-1,k,1}\right)$
$\displaystyle s_{j,k,2}$
$\displaystyle=\text{GRU}_{k,2}\left(c_{j,1},s_{j,1,1}\right)$
$\displaystyle\text{for }1<k\leq D_{t}$

###  2.4 BiDeep architectures

We introduce the BiDeep RNN, a novel architecture obtained by combining deep
transitions with stacking.

A BiDeep encoder is obtained by replacing the $D_{s}$ individually recurrent
GRU cells of a stacked encoder with multi-layer deep transition cells each
composed by $L_{s}$ GRU transition blocks.

For instance, the BiDeep alternating encoder is defined as follows:

$\displaystyle\overrightarrow{w}_{i,1}$
$\displaystyle=\overrightarrow{h}_{i,1}=\text{DTGRU}_{1}\left(x_{i},\overrightarrow{h}_{i-1,1}\right)$
$\displaystyle\overrightarrow{h}_{i,2k}$
$\displaystyle=\text{DTGRU}_{2k}\left(\overrightarrow{w}_{i,2k-1},\overrightarrow{h}_{i+1,2k}\right)$
$\displaystyle\text{ for }1<2k\leq D_{s}$
$\displaystyle\overrightarrow{h}_{i,2k+1}$
$\displaystyle=\text{DTGRU}_{2k+1}\left(\overrightarrow{w}_{i,2k},\overrightarrow{h}_{i-1,2k+1}\right)$
$\displaystyle\text{ for }1<2k+1\leq D_{s}$
$\displaystyle\overrightarrow{w}_{i,j}$
$\displaystyle=\overrightarrow{h}_{i,j}+\overrightarrow{w}_{i,j-1}$
$\displaystyle\text{ for }1<j\leq D_{s}$

where each multi-layer cell $\text{DTGRU}_{k}$ is defined as:

$\displaystyle v_{k,1}$
$\displaystyle=\text{GRU}_{k,1}\left(\text{in}_{k},\text{state}_{k}\right)$
$\displaystyle v_{k,t}$
$\displaystyle=\text{GRU}_{k,t}\left(0,v_{k_{t}-1}\right)\text{for }1<k\leq
L_{s}$ $\displaystyle\text{DTGRU}_{k}$
$\displaystyle\left(\text{in}_{k},\text{state}_{k}\right)=v_{k,L_{s}}$

It is also possible to have different transition depths at each stacking
level.

BiDeep decoders are similarly defined, replacing the recurrent cells (GRU,
rGRU, cGRU or crGRU) with deep transition multi-layer cells.

##  3 Experiments

All experiments were performed with Nematus (Sennrich et al., 2017b),
following Sennrich et al. (2017a) in their choice of preprocessing and
hyperparameters. For experiments with deep models, we increase the depth by a
factor of 4 compared to the baseline for most experiments; in preliminary
experiments, we observed diminishing returns for deeper models.

We trained on the parallel English–German training data of WMT-2017 news
translation task, using newstest2013 as validation set. We used early-stopping
on the validation cross-entropy and selected the best model based on
validation Bleu.

We report cross-entropy (CE) on newstest2013, training speed (on a single
Titan X (Pascal) GPU), and the number of parameters. For translation quality,
we report case-sensitive, detokenized Bleu, measured with mteval-v13a.pl, on
newstest2014, newstest2015, and newstest2016.

We release the code under an open source license, including it in the official
Nematus repository.222https://github.com/EdinburghNLP/nematus The
configuration files needed to replicate our experiments are available in a
separate repository.333https://github.com/Avmb/deep-nmt-architectures

###  3.1 Layer Normalization

encoder | CE | Bleu | parameters (M) | training speed | early stop
---|---|---|---|---|---
|  | 2014 | 2015 | 2016 |  | (words/s) | ($10^{4}$ minibatches)
baseline | 49.98 | 21.2 | 23.8 | 28.4 |  098.0 | 3350 | 44
+layer normalization | 47.53 | 21.9 | 24.7 | 29.3 |  098.1 | 2900 | 29
alternating (depth 4) | 49.25 | 21.8 | 24.6 | 28.9 | 135.8 | 2150 | 46
+layer normalization | 46.29 | 22.6 | 25.2 | 30.5 | 135.9 | 1600 | 29
Table 1: Layer normalization results. English$\to$German WMT17 data.

Our first experiment is concerned with layer normalization. We are interested
to see how essential layer normalization is for our deep architectures, and
compare the effect of layer normalization on a baseline system, and a system
with an alternating encoder with stacked depth 4. Results are shown in Table
1. We find that layer normalization is similarly effective for both the
shallow baseline model and the deep encoder, yielding an average improvement
of 0.8–1 Bleu, and reducing training time substantially. Therefore we use it
for all the subsequent experiments.

###  3.2 Deep Encoders

encoder | depth | CE | Bleu | parameters (M) | training speed
---|---|---|---|---|---
| s. bidir. | s. forw. | trans. |  | 2014 | 2015 | 2016 |  | (words/s)
shallow | 1 | - | 1 | 47.53 | 21.9 | 24.7 | 29.3 |  0 98.1 | 2900
alternating | 4 | - | 1 | 46.29 | 22.6 | 25.2 | 30.5 | 135.9 | 1600
biunidirectional | 1 | 3 | 1 | 46.79 | 22.4 | 25.4 | 30.0 | 173.7 | 1500
deep transition | 1 | - | 4 | 46.54 | 22.9 | 25.4 | 30.2 | 117.0 | 1900
Table 2: Deep encoder results. English$\to$German WMT17 data. Parameters and
speed are highlighted for the deep recurrent models.

In Table 2 we report experimental results for different architectures of deep
encoders, while the decoder is kept shallow.

We find that all the deep encoders perform substantially better than baseline
(+0.5–+1.2 Bleu), with no consistent quality differences between each other.
In terms of number of parameters and training speed, the deep transition
encoder performs best, followed by the alternating stacked encoder and finally
the biunidirectional encoder (note that we trained on a single GPU, the
biunidirectional encoder may be comparatively faster on multiple GPUs due to
its higher model parallelism).

###  3.3 Deep Decoders

decoder | high RNN | decoder RNN depth | output | CE | Bleu | params. | training speed
---|---|---|---|---|---|---|---
| stacked | trans. | type | depth |  | 2014 | 2015 | 2016 | (M) | (words/s)
shallow | - | 1 | 1 | 1 | 47.53 | 21.9 | 24.7 | 29.3 |  0 98.1 | 2900
stacked | GRU | 4 | 1 | 1 | 46.73 | 21.8 | 24.6 | 29.5 | 117.0 | 2250
stacked | rGRU | 4 | 1 | 1 | 46.72 | 22.1 | 25.0 | 29.4 | 135.9 | 2150
stacked | cGRU | 4 | 1 | 1 | 44.76 | 22.8 | 25.5 | 29.6 | 164.3 | 1300
stacked | crGRU | 4 | 1 | 1 | 45.88 | 22.5 | 24.7 | 29.7 | 145.4 | 1750
deep transition | - | 1 | 8 | 1 | 45.98 | 22.4 | 24.9 | 30.0 | 117.0 | 2200
deep output | - | 1 | 1 | 4 | 47.21 | 21.5 | 24.2 | 28.7 |  098.9 | 2850
Table 3: Deep decoder results. English$\to$German WMT17 data. Parameters and
speed are highlighted for the deep recurrent models.

Table 3 shows results for different decoder architectures, while the encoder
is shallow. We find that the deep decoders all improve the cross-entropy, but
the Bleu results are more varied: deep output444deep feed-forward output with
shallow RNNs in both the encoder and decoder decreases Bleu scores (but note
that the baseline has already some depth), stacked GRU performs similarly to
the baseline (-0.1–+0.2 Bleu) and stacked rGRU possibly slightly better
(+0.1–+0.2 Bleu).

Other deep RNN decoders achieve higher gains. The best results (+0.6 Bleu on
average) are achieved by the stacked conditional GRU with independent multi-
step attention (cGRU). This decoder, however, is the slowest one and has the
most parameters.

The deep transition decoder performs well (+0.5 Bleu on average) in terms of
quality and is the fastest and smallest of all the deep decoders that have
shown quality improvements.

The stacked conditional GRU with reused attention (crGRU) achieves smaller
improvements (+0.3 Bleu on average) and has speed and size intermediate
between the deep transition and stacked cGRU decoders.

###  3.4 Deep Encoders and Decoders

encoder | decoder | decoder high | encoder depth | decoder depth | CE | Bleu | params. | training speed
---|---|---|---|---|---|---|---|---
|  | RNN type | bidir. forw. trans. | stacked | trans. |  | 2014 | 2015 | 2016 | (M) | (words/s)
shallow | shallow | - | 1 | - | 1 | 1 | 1 | 47.53 | 21.9 | 24.7 | 29.3 |  0 98.1 | 2900
deep tran. | shallow | - | 1 | - | 4 | 1 | 1 | 46.54 | 22.9 | 25.4 | 30.2 | 117.0 | 1900
(Zhou et al., 2016) (ours) |  |  |  |  |  |  |  |  |  |  |
alternating | stacked | GRU | 4 | - | 1 | 4 | 1 | 45.89 | 22.9 | 25.3 | 30.1 | 154.9 | 1480
(Wu et al., 2016) (ours) |  |  |  |  |  |  |  |  |  |  |
biunidir. | stacked | rGRU | 1 | 3 | 1 | 4 | 1 | 46.15 | 22.4 | 24.7 | 29.6 | 211.5 | 1280
alternating | stacked | rGRU | 4 | - | 1 | 4 | 1 | 46.00 | 23.0 | 25.7 | 30.5 | 173.7 | 1400
alternating | stacked | cGRU | 4 | - | 1 | 4 | 1 | 44.32 | 22.9 | 25.7 | 29.8 | 202.1 |  0970
deep tran. | deep tran. | - | 1 | - | 4 | 1 | 8 | 45.52 | 22.7 | 25.7 | 30.1 | 136.0 | 1570
BiDeep altern. | BiDeep | rGRU | 2 | - | 2 | 2 | 4/2 | 43.52 | 23.1 | 25.5 | 30.6 | 145.4 | 1480
BiDeep altern. | BiDeep | rGRU | 4 | - | 2 | 4 | 4/2 | 43.26 | 23.4 | 26.0 | 31.0 | 214.7 |  0980
alternating | stacked | rGRU | 8 | - | 1 | 8 | 1 | 44.32 | 22.9 | 25.5 | 30.5 | 274.6 |  0880
Table 4: Deep encoder–decoder results. English$\to$German WMT17 data.
Transition depth 4/2 means 4 in the base RNN of the stack and 2 in the higher
RNNs. The last two models are large and their results are highlighted
separately.

Table 4 shows results for models where both the encoder and the decoder are
deep, in addition to the results of the best deep encoder (the deep transition
encoder) + shallow decoder reported here for ease of comparison.

Compared to deep transition encoder alone, we generally see improvements in
cross-entropy, but not in Bleu. We evaluate architectures similar to Zhou et
al. (2016) (alternating encoder + stacked GRU decoder) and (Wu et al., 2016)
(biunidirectional encoder + stacked rGRU decoder), though they are not
straight replications since we used GRU cells rather than LSTMs and the
implementation details are different. We find that the former architecture
performs better in terms of Bleu scores, model size and training speed.

The other variants of alternating encoder + stacked or deep transition decoder
perform similarly to alternating encoder + stacked rGRU decoder, but do not
improve Bleu scores over the best deep encoder with shallow decoder. Applying
the BiDeep architecture while keeping the total depth the same yields small
improvements over the best deep encoder (+0.2 Bleu on average), while the
improvement in cross-entropy is stronger. We conjecture that deep decoders may
be better at handling subtle target-side linguistic phenomena that are not
well captured by the 4-gram precision-based Bleu evaluation.

Finally, we evaluate a subset of architectures with a combined depth that is 8
times that of the baseline. Among the large models, the BiDeep model yields
substantial improvements (average +0.6 Bleu over the best deep encoder, +1.5
Bleu over the shallow baseline), in addition to cross-entropy improvements.
The stacked-only model, on the other hand, performs similarly to the smaller
models, despite having even more parameters than the BiDeep model. This shows
that it is useful to combine deep transitions with stacking, as they provide
two orthogonal kinds of depth that are both beneficial for neural machine
translation.

###  3.5 Error Analysis

One theoretical difference between a stacked RNN and a deep transition RNN is
that the distance in the computation graph between timesteps is increased for
deep transition RNNs. While this allows for arguably more expressive
computations to be represented, in principle it could reduce the ability to
remember information over long distances, since each layer may lose
information during forward computation or backpropagation. This may not be a
significant issue in the encoder, as the attention mechanism provides short
paths from any source word state to the decoder, but the decoder contains no
such shortcuts between its states, therefore it might be possible that this
negatively affects its ability to model long-distance relationships in the
target text, such as subject–verb agreement.

Here, we seek to answer this question by testing our models on Lingeval97
(Sennrich, 2017), a test set which provides contrastive translation pairs for
different types of errors. For the example of subject-verb agreement,
contrastive translations are created from a reference translation by changing
the grammatical number of the verb, and we can measure how often the NMT model
prefers the correct reference over the contrastive variant.

In Figure 5, we show accuracy as a function of the distance between subject
and verb. We find that information is successfully passed over long distances
by the deep recurrent transition network. Even for decisions that require
information to be carried over 16 or more words, or at least 128 GRU
transitions555some decisions may not require the information to be passed on
the target side because the decisions may be possible based on source-side
information., the deep recurrent transition network achieves an accuracy of
over 92.5% ($N=560$), higher than the shallow decoder (91.6%), and similar to
the stacked GRU (92.7%). The highest accuracy (94.3%) is achieved by the
BiDeep network.

$0$$4$$8$$12$$16$$0.8$$0.85$$0.9$$0.95$$1$distanceaccuracyshallow GRUstacked
GRUdeep transition GRUBiDeep GRU$\geq$ 16 Figure 5: Subject-verb agreement
accuracy as a function of distance between subject and verb.

##  4 Conclusions

In this work we presented and evaluated multiple architectures to increase the
model depth of neural machine translation systems.

We showed that alternating stacked encoders (Zhou et al., 2016) outperform
biunidirectional stacked encoders Wu et al. (2016), both in accuracy and
(single-GPU) speed. We showed that deep transition architectures, which we
first applied to NMT, perform comparably to the stacked ones in terms of
accuracy (Bleu, cross-entropy and long-distance syntactic agreement), and
better in terms of speed and number of parameters.

We found that depth improves Bleu scores especially in the encoder. Decoder
depth, however, still improves cross-entropy if not strongly Bleu scores.

The best results are obtained by our BiDeep architecture which combines both
stacked depth and transition depth in both the (alternating) encoder and the
decoder, yielding better accuracy for the same number of parameters than
systems with only one kind of depth.

We recommend to use combined architectures when maximum accuracy is the goal,
or use deep transition architectures when speed or model size are a concern,
as deep transition performs very positively in the quality/speed and
quality/size trade-off.

While this paper only reports results for one translation direction, the
effectiveness of the presented architectures across different data conditions
and language pairs was confirmed in follow-up work. For the shared news
translation task of this year’s Conference on Machine Translation (WMT17), we
built deep models for 12 translation directions, using a deep transition
architecture or a stacked architecture (alternating encoder and rGRU decoder),
and observe improvements for the majority of translation directions Sennrich
et al. (2017a).

## Acknowledgments

The research presented in this publication was conducted in cooperation with
Samsung Electronics Polska sp. z o.o. - Samsung R&D Institute Poland. eu-
logo.pngThis project received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreements 645452 (QT21), 644402
(HimL) and 688139 (SUMMA).

## References

  * Ba et al. (2016) Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016.  Layer Normalization.  CoRR abs/1607.06450.
  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural Machine Translation by Jointly Learning to Align and Translate.  In Proceedings of the International Conference on Learning Representations (ICLR).
  * Bojar et al. (2016) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016.  Findings of the 2016 Conference on Machine Translation (WMT16).  In Proceedings of the First Conference on Machine Translation, Volume 2: Shared Task Papers. Association for Computational Linguistics, Berlin, Germany, pages 131–198.
  * Britz et al. (2017) Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V. Le. 2017.  Massive Exploration of Neural Machine Translation Architectures.  CoRR abs/1703.03906.
  * Cettolo et al. (2016) Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. 2016.  Report on the 13th IWSLT Evaluation Campaign.  In IWSLT 2016. Seattle, USA.
  * Cho et al. (2014a) Kyunghyun Cho, B van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a.  On the properties of neural machine translation: Encoder-decoder approaches.  In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8), 2014.
  * Cho et al. (2014b) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar, pages 1724–1734.
  * Firat and Cho (2016) Orhan Firat and Kyunghyun Cho. 2016.  Conditional Gated Recurrent Unit with Attention Mechanism.  https://github.com/nyu-dl/dl4mt-tutorial/blob/master/docs/cgru.pdf.  Published online, version adbaeea.
  * Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017\.  Convolutional Sequence to Sequence Learning.  CoRR abs/1705.03122.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. pages 770–778.
  * Pascanu et al. (2014) Razvan Pascanu, Çağlar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. 2014.  How to Construct Deep Recurrent Neural Networks.  In International Conference on Learning Representations 2014 (Conference Track).
  * Sennrich (2017) Rico Sennrich. 2017.  How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs.  In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. Association for Computational Linguistics, Valencia, Spain, pages 376–382.
  * Sennrich et al. (2017a) Rico Sennrich, Alexandra Birch, Anna Currey, Ulrich Germann, Barry Haddow, Kenneth Heafield, Antonio Valerio Miceli Barone, and Philip Williams. 2017a.  The University of Edinburgh’s Neural MT Systems for WMT17.  In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers. Copenhagen, Denmark.
  * Sennrich et al. (2017b) Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017b.  Nematus: a Toolkit for Neural Machine Translation.  In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Valencia, Spain, pages 65–68.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In Advances in neural information processing systems. pages 3104–3112.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention Is All You Need.  arXiv preprint arXiv:1706.03762 .
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016.  Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation.  CoRR abs/1609.08144.
  * Zhang et al. (2016) Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan R Salakhutdinov, and Yoshua Bengio. 2016.  Architectural Complexity Measures of Recurrent Neural Networks.  In Advances in Neural Information Processing Systems 29. pages 1822–1830.
  * Zhou et al. (2016) Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016.  Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation.  TACL 4:371–383.
  * Zilly et al. (2016) Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2016.  Recurrent highway networks.  arXiv preprint arXiv:1607.03474 .


# Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms

Albert Jiménez Sanfiz
AIP Labs
<EMAIL_ADDRESS>
&Mohamed Akrout
AIP Labs
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Backpropagation is the default algorithm for training deep neural networks due
to its simplicity, efficiency and high convergence rate. However, its
requirements make it impossible to be implemented in a human brain. In recent
years, more biologically plausible learning methods have been proposed. Some
of these methods can match backpropagation accuracy, and simultaneously
provide other extra benefits such as faster training on specialized hardware
(e.g., ASICs) or higher robustness against adversarial attacks. While the
interest in the field is growing, there is a necessity for open-source
libraries and toolkits to foster research and benchmark algorithms. In this
paper, we present BioTorch, a software framework to create, train, and
benchmark biologically motivated neural networks. In addition, we investigate
the performance of several feedback alignment methods proposed in the
literature, thereby unveiling the importance of the forward and backward
weight initialization and optimizer choice. Finally, we provide a novel
robustness study of these methods against state-of-the-art white and black-box
adversarial attacks.

##  1 Introduction

Backpropagation (BP) [1] is the dominant algorithm to compute the gradients of
a learning system with respect to a predefined loss function. Its simplicity,
efficiency, and high accuracy and convergence rates, make it the de facto
algorithm to train neural networks. However, there is evidence that such an
algorithm could not be biologically implemented by the human brain. One of the
main reasons is the need of BP to have symmetric forward and backward synaptic
weights (i.e., the weight transport or the weight symmetry problem) [1, 2, 3,
4, 5]. Since synapses are unidirectional in the brain, feedforward and
feedback connections must be physically distinct.

To overcome the aforementioned limitation, recent studies in learning
algorithms have focused on the intersection between neuroscience and machine
learning by studying more biologically-plausible algorithms. One of the main
family of methods proposed is based on neural activity differences to encode
errors (NGRAD) [6], namely, target propagation (TP) and its variants [7, 8, 9,
10, 11, 12]. The other family is known as alignment methods, which employ
distinct forward and feedback synaptic weights, where the latter are
randomized [13, 14, 15], thereby providing noisy and synthetic gradient
training schemes. Neural networks trained with alignment methods learn to
adapt the forward synaptic weights so the alignment between both matrices
increase, making learning possible. Therefore, random feedback weights can
transmit useful teaching signals to neurons along the network depth [13]. The
scope of this work will be focused on feedback alignment (FA) methods.

Early studies [13, 15], have shown that shallow networks using FA and direct
FA (DFA) methods were able to learn good feature representations of complete
simple tasks such digit recognition in MNIST. Deeper architectures were
trained in [11] with FA and TP methods showing their inability to learn in
complex visual tasks such as ImageNet [16]. The forward-backward weight
asymmetry of these methods can lead to vanishing or exploding gradients
because the backward weighst remain fixed during the training. Several works
[17, 18, 19, 20, 21] have shown, however, that increasing the alignment of the
feedforward and backward weights, alone or in combination with random feedback
weight matrices, can help prevent that behaviour. Some works aim to control
the magnitudes of the weights at training time. For example, in [19], the
authors experiment constraining the magnitude of the forward weights after
every training iteration scaling them by the initial weights $L_{2}$ norm. In
[17], the authors transport the sign of the weight matrix, and propose Batch
Manhattan (BM) as a parameter update rule, where the gradient magnitudes are
discarded and the updates are proportional to the sign. Another technique
proven to be useful is careful initialization of the forward and backward
weight matrices. Using variance preserving methods like Xavier [22] helps
ensure the variance of the activations are the same across every layer,
preventing the forward and backward signals from exploding or vanishing. These
lastest works reach or surpass BP metrics on MNIST or CIFAR10 [23] and scale
to more difficult tasks, obtaining a close performance to BP on ImageNet.

In addition to providing synthetic gradient training schemes, it has been
shown that alignment methods can increase the robustness of deep learning
models against adversarial attacks [24]. Adversarial vulnerability is an
intrinsic property of all classifiers [25] and state-of-the-art defense
techniques rely on robust pre-trained models and expensive adversarial
training with Projected Gradient Descent (PGD) [26]. It was demonstrated in
[27] that fine-tuning pre-trained models using the DFA method increases the
robustness when training with a photonic co-processor for physical parameter
obfuscation [28]. Finally, the additional interest for feedback alignment is
also driven by their ability to allow forward and feedback weights to live
locally in application-specific integrated circuits (ASICs), which ultimately
allows for time and energy savings [29, 30, 31].

While the research and interest on feedback alignment methods is growing, we
noticed a lack of open-source libraries and public algorithmic benchmarks. In
this work, we provide BioTorch, a modular and extensible framework with the
implementations of five alignment methods: FA, DFA, and sign concordant
variants (uSF, brSF, frSF), whose code is available at
https://github.com/jsalbert/biotorch.

Our contributions can be summarized as follows:

  1. i)

We provide the first open-source software framework that allows for the
creation of custom neural networks with feedback alignment layers, the
automatic conversion of model architectures, reproducible training,
evaluation, and benchmarking (in both accuracy and robustness against
adversarial attacks) via configurable experiments.

  2. ii)

We carried out a benchmark study of the performance of feedback alignment
algorithms on different datasets. First, we benchmarked a shallow network on
MNIST and Fashion-MNIST. Second, we increased the network’s depth as we
investigated the impact of different optimizers on the performance of all
alignment methods on CIFAR-10. Finally, we benchmarked all methods by training
a ResNet-18 on ImageNet.

  3. iii)

We performed a novel robustness study of feedback alignment algorithms against
a set of state-of-the-art white and black-box adversarial attacks.

##  2 Feedback Alignment Methods and Their Potential Adversarial Robustness

Given a labeled dataset $\cup_{k}\\{(\mathbf{x}^{(k)},\mathbf{y}^{(k)})\\}$
and cost function $\mathcal{L}$, the training of a neural network of $N$
layers alternates between forward passes, to perform inference and compute an
error signal, and backward passes, to send the error signal back and update
the weights. The output of the $i$th layer, $\mathbf{y}_{i}$, is computed by
forwarding the output signal of the $(i-1)$th signal, $\mathbf{y}_{i-1}$, from
layer $(i-1)$ as follows

$\mathbf{y}_{i}=\mathbf{\phi}(\mathbf{z}_{i}),~{}\textrm{with}~{}~{}\mathbf{z}_{i}=\mathbf{W}_{i}\;\mathbf{y}_{i-1}+\mathbf{b}_{i}.$
(1)

The backpropagation equation [1] for the weight updates is given by:

$\delta\mathbf{W}_{i}=-\left(\left(\mathbf{W}_{i+1}^{\top}\,\delta\mathbf{z}_{i+1}\right)\,\odot\,\phi^{\prime}(\mathbf{z}_{i})\right)\mathbf{y}_{i-1}^{\top},~{}\textrm{with}~{}~{}\delta\mathbf{z}_{i+1}=\frac{\partial\mathcal{L}}{\partial\mathbf{z}_{i+1}}.$
(2)

###  2.1 Alignment Methods

(a) BP (b) FA (c) DFA

Figure 1: Schematic overview of different error transportation methods. Blue
arrows indicate paths involving the forward pass weight matrix, $\mathbf{W}$,
and gray arrows indicate backward paths involving a fixed weight matrix
$\mathbf{B}$.

Feedback Alignment (FA) The weight update (2) of layer $i$ requires the
knowledge of the matrix $\mathbf{W}_{i+1}^{\top}$ and is biologically
implausible because it requires that neurons send to each other large numbers
of synaptic weights (i.e. weight transport). This fact has motivated the
substitution of the matrix $\mathbf{W}_{i+1}^{\top}$ with a random synaptic
weights matrix $\mathbf{B}_{i+1}$ to account for a separate backward pass
needed to avoid the weight transport problem [14]. Therefore, the weight
update becomes:

$\delta\mathbf{W}_{i}=-\left(\left(\mathbf{B}_{i+1}\,\delta\mathbf{z}_{i+1}\right)\,\odot\,\phi^{\prime}(\mathbf{z}_{i})\right)\mathbf{y}_{i-1}^{\top},~{}\textrm{with}~{}~{}\delta\mathbf{z}_{i+1}=\frac{\partial\mathcal{L}}{\partial\mathbf{z}_{i+1}}.$
(3)

Direct Feedback Alignment (DFA) While the weight update in feedback alignment
is computed recursively across layers, it is possible to project the error
propagation by directly backwarding the derivative of the loss at the last
layer, $\delta\mathbf{z}_{N}$, to all layers [15]. This results in the
following update:

$\delta\mathbf{W}_{i}=-\left(\left(\mathbf{B}_{i}\,\delta\mathbf{z}_{N}\right)\,\odot\,\phi^{\prime}(\mathbf{z}_{i})\right)\mathbf{y}_{i-1}^{\top},~{}\textrm{with}~{}~{}\delta\mathbf{z}_{N}=\frac{\partial\mathcal{L}}{\partial\mathbf{z}_{N}}.$
(4)

where $\mathbf{B}_{i}$ is a fixed random matrix of appropriate shape (i.e.,
input dimension of layer $i$ $\times$ output dimension of the last layer $N$).

Uniform Sign-concordant Feedbacks (uSF) This method transports the sign of the
forward matrices by assuming synaptic weights with unit magnitudes, i.e.,
$\mathbf{B}_{i}=\textrm{sign}(\mathbf{W}_{i}^{\top})~{}\forall i$.

Batchwise Random Magnitude Sign-concordant Feedbacks (brSF) Instead of
assuming a unit magnitude for the synaptic weights of the $i$th layer, this
method re-draws their magnitude $\left|\mathbf{R}_{i}\right|$ after each
update such that
$\mathbf{B}_{i}=\left|\mathbf{R}_{i}\right|\odot\textrm{sign}(\mathbf{W}_{i}^{\top})~{}\forall
i$.

Fixed Random Magnitude Sign-concordant Feedbacks (frSF) This is a variation of
the brSF method where the magnitude of weights $\left|\mathbf{R}_{i}\right|$
is not redrawn after each update, but rather fixed and initialized at the
start of the training.

Note that for the uSF method to converge, it is common to scale the sign
matrix by the initialization rule applied to create the weight matrix [17,
19]. For the brSF and frSF methods, we highlight the application of the
absolute value of the feedback weights matrix, as opposed to [17], to
transport the sign correctly and make convergence possible without the need
for an update parameter rule like Batch Manhattan.

###  2.2 Adversarial Robustness

White-box adversarial attacks are very sensitive to the quality of gradients
as they rely on them to perturb the input and fool the model. The noise
introduced in those gradients when they are computed using the methods
described in Section 2.1, could decrease the effectiveness of such attacks on
models trained with feedback alignment.

To investigate the robustness of deep networks trained with alignment methods,
it is important to select attacks based on different principles. Black-box
attacks have recently become more popular as their strategies are designed to
work without requiring access to the target model. We chose to benchmark on
four state-of-the-art white-box attacks: fast gradient sign method (FGSM)
[25], projected gradient descent (PGD) [26], averaged PGD (APGD) [32], and PGD
used in the TRADES training (TPGD) [33]; and two state-of-the-art black-box
attacks: Few-Pixel [34] and Square [35]. See the description of these attacks
in Appendix 6.3.

##  3 BioTorch

BioTorch is an open-source Python library specializing in biologically
plausible learning algorithms that runs PyTorch [36] in the backend for its
efficiency and auto-differentiation routines. Its design focuses on the
principles of research reproducibility [37] and integration with PyTorch
training and testing patterns. BioTorch holds implementations of all the
feedback alignment methods mentioned in Section 2. At the moment, it is
focused on computer vision tasks, as these have been the most explored in the
related literature.

###  3.1 BioTorch Interface

BioTorch’s intuitive interface, allows the user to create neural network
models with feedback alignment methods in an effortless manner. Its main
features are highlighted below.

Model importing: BioTorch comes with a list of predefined 25 state-of-the-art
architectures, such as ResNet-50, DenseNet-161 and MNasNet. To create one of
these models, it is enough to import the model from the path
biotorch.models.<method>, e.g.:

⬇

# Create a ResNet-18 model with FA layers

from biotorch.models.fa import resnet18

model = resnet18()

Custom model creation using alignment layers: By importing individual Linear
or Convolutional layers implementing alignment methods, the architecture of a
neural network model can be defined as follows:

⬇

# Import a convolutional and linear layer using the uSF alignment method

import torch.nn.functional as F

from biotorch.layers.usf import Conv2d, Linear

# Define the custom model using the imported layers

class Model(nn.Module):

def __init__(self):

super(Model, self).__init__()

self.conv1 = Conv2d(in_channels=3, out_channels=128, kernel_size=3)

self.fc = Linear(in_features=128, out_features=10)

def forward(self, x):

out = F.relu(self.conv1(x))

out = F.avg_pool2d(out, out.size()[3])

return self.fc(out)

model = Model()

Automatic conversion of an existing model architecture: It is possible to use
the BioModule wrapper of BioTorch to automatically convert an existing PyTorch
model to use any of the alignment methods aforementioned.

⬇

from torchvision.models import alexnet

from biotorch.module.biomodule import BioModule

# Convert Alex-Net from using BP to the frSF alignment method

model = BioModule(module=alexnet(), mode=’frsf’)

###  3.2 Configurable and Reproducible Experiments

The current version of BioTorch allows the user to benchmark different
alignment algorithms on a selection of datasets. All the datasets supported
are downloaded using the Torchvision library, except for ImageNet
(ILSVRC2012), which is pulled from the official source.

Diligent measures were taken to ensure correct experiment reproducibility.
Experiments are run via configuration files that contain all the training
hyperparameters, a dataset, and selected metrics. We include the choice of a
fixed seed to set deterministic mathematical and CUDA operations [38] to
guarantee that the experiments can be replicated. The best performing model on
the validation set, the original configuration file, test results, and
TensorBoard logs for all the metrics are saved at the end of each experiment.
All experiments in Section 4 were run with the following command using the
appropriate .yaml configuration file:

⬇

#!/bin/bash

python benchmark.py \--config benchmark_config_example.yaml

We included an explained example of a configuration file in the Appendix 6.2
and we refer the reader to the code repository where interactive Colaboratory
notebooks and a complete description of the configuration file are provided.

###  3.3 Trackable Alignment Metrics

To observe and measure the behaviour of the algorithms during training,
BioTorch includes two key metrics of alignment methods.

The backward-forward weight norm ratio During the training of networks with
feedback alignment methods, the magnitude of the forward weight matrices
$\left\lVert\mathbf{W}^{\top}_{i}\right\rVert_{2}$ changes while the magnitude
of feedback weight matrices $\left\lVert\mathbf{B}_{i}\right\rVert_{2}$ is
fixed. This weight update asymmetry can lead to network convergence issues.
Indeed, as observed in [19], the ratio between the norms of the backward and
forward weights, i.e.,
$\left\lVert\mathbf{B}_{i}\right\rVert_{2}~{}/~{}\left\lVert\mathbf{W}^{\top}_{i}\right\rVert_{2}$,
determine if the trained networks are at risk of experiencing
vanishing/exploding gradients when BP does not, even while using the same
training hyperparameters.

The backward-forward weight alignment For a given layer $i$, the angle between
every pair of transposed forward and backward weight matrices, namely,
($\mathbf{W}^{\top}_{i},\mathbf{B}_{i}$), can be computed to measure how close
the considered alignment method is to the case with symmetric weights. Angles
close to $0$ denote higher alignment and thus approaching the alignment of BP.

##  4 Experiments

In accordance with the techniques proposed in past works, we kept $\mathbf{W}$
and $\mathbf{B}$ in the same magnitude scale to improve training in asymmetric
conditions. In particular, we experimented constraining the forward weights,
discarding the magnitude of the gradients using only their sign or clipping
them, and providing a careful initialization of parameters. At the end, the
below two choices were the most impactful.

Variance preserving initialization We initialized our layers weight matrices,
$\mathbf{W}$ and $\mathbf{B}$, using Xavier Uniform initialization [22]. For
the uSF method, we scaled the sign of the weights by the standard deviation of
Xavier initialization. For the brSF and frSF methods, we used the absolute
value of the random feedback weight matrix to prevent the change of the sign,
in contrast with the implementation of [17]. Our experiments with these
initializations were successful to train all the networks, independently of
the network depth.

The choice of Adam optimizer A review of empirical experiments of prior
studies, reveals a variety of different optimizer choices. More specifically,
[15] chose RMSprop [39], [11, 19] made use of Adam [40], and [17, 18, 20] used
SGD. Our experiments show a significant gap in the classification accuracy of
the models depending on the choice of the optimizer, specially for FA and DFA,
even after tuning their respective learning rate. For this reason, we show the
results of our experiments for both SGD and Adam optimizers.

###  4.1 Accuracy Benchmark

MNIST & Fashion-MNIST We start our empirical study by benchmarking all the
alignment methods for LeNet [41] on the MNIST and Fashion-MNIST datasets. The
networks were trained with the SGD optimizer setting a momentum of $0.9$, and
weight decay of $10^{-3}$. We trained for $100$ epochs, decreasing the initial
learning rate by a factor of $2$ at the $50th$ and $75th$ epoch. The rest of
network hyperparameters can be found in the Appendix (Section 6.1).

The results of our implementation (Table 1) are close to those reported in
[19] for similar benchmarking conditions. We observe that the performances of
FA and DFA are close to BP, and that sign concordant methods match BP
performance on MNIST. If we increase the dataset difficulty (Fashion-MNIST),
the performance gap between backpropagation and the other methods, specially
with the ones not using sign concordant feedback, also increases.

| Dataset
---|---
| MNIST | Fashion MNIST
BP | 0.91 | 9.20
FA | 1.7 | 13.06
DFA | 1.61 | 12.81
uSF | 0.94 | 9.69
brSF | 0.91 | 10.02
frSF | 0.97 | 9.61
Table 1: Top-1 test error (%) of LeNet on the MNIST and Fashion-MNIST
datasets.

CIFAR-10 & ImageNet What happens when we scale the application of alignment
methods to deeper architectures and more challenging tasks? To answer this
question, we benchmark three different networks on CIFAR-10. We start with a
modified 5-layer LeNet network, and continue increasing the depth with a
ResNet-20, followed by a ResNet-56 [42]. We applied the same dataset split as
in [42] where a 5K images from the training set were sampled and used as
validation test, while the evaluation is performed on the unseen test set. All
the training details and network hyperparameters can be found in the Appendix
Section 6.1.

The use of an adaptive parameter independent optimizer as Adam outperformed
SGD in all the network configurations for the FA and DFA methods, as shown in
Table 2. These methods are the ones where the asymmetry in the backward pass
is larger.

| LeNet | ResNet-20 | ResNet-56
---|---|---|---
| SGD | Adam | SGD | Adam | SGD | Adam
BP | 14.23 | 15.92 | 8.63 | 10.01 | 8.3 | 7.83
FA | 46.69 | 40.67 | 32.16 | 29.59 | 34.88 | 29.23
DFA | 54.21 | 37.59 | 45.94 | 32.16 | 38.01 | 32.02
uSF | 16.22 | 16.34 | 10.05 | 10.59 | 8.2 | 9.19
brSF | 16.02 | 17.08 | 11.02 | 11.08 | 8.69 | 10.13
frSF | 16.86 | 16.83 | 11.2 | 11.22 | 9.49 | 10.02
Table 2: Top-1 test error (%) on the CIFAR-10 dataset. Three networks ranging
from shallower to deeper are benchmarked using BP and alignment methods.

The significant improvement brought by Adam is expected since it maintains
per-parameter learning rates that are adapted based on the average of the
second moments of the gradients inherited from RMSProp. Therefore, it yields
better performances under noisy gradients. To confirm this observation, we
plot in Fig. 2 the backward-forward norm weight ratios for the DFA method for
both SGD and Adam optimizers. There, it is seen that SGD has driven the norm
weight ratios of the first layers of the network very close to 0, which means
that the forward weight matrices $\mathbf{W}_{i}$ were updated to reach much
larger values than the ones in the backward weight matrices $\mathbf{B}_{i}$.
This is due to the direct error projection of the error from the last layer to
each layer in the DFA method, thereby sidestepping the small norms of the
gradients computed using chain rule.

(a) DFA (SGD) (b) DFA (Adam)

Figure 2: Weight ratios for the methods FA and DFA when training a ResNet56 on
CIFAR-10 for SGD in (a) and Adam in (b).

The same observation does not hold for the FA method, where the weight norm
ratio of the first layers does not vanish to 0 as shown in Fig. 3. We also see
that Adam achieves smaller backward-forward weight alignment angles compared
to SGD for both LeNet and ResNet-20.

(a) LeNet (SGD)

(b) ResNet-20 (SGD)

(c) LeNet (Adam)

(d) ResNet-20 (Adam)

Figure 3: Matrix alignment and weight ratios for FA and different network and
optimizer configurations. LeNet with SGD or Adam optimizer in (a) and (c).
ResNet-20 with SGD or Adam optimizer in (b) and (d)

Furthermore, we investigate the effect of the difference between the
distributions used to initialize the forward and backward weights,
$\mathbf{W}_{i}$ and $\mathbf{B}_{i}$, respectively. To this end, we
initialized the backward weights with the Xavier initialization. Then, we run
two separate experiments in which we initialized the forward weights with a
Kaiming Uniform 111PyTorch default Kaiming init. is defined as
nn.init.kaiming_uniform_(weight, a=math.sqrt(5)) (PyTorch default layer
initialization), and then with a Normal distribution $\mathcal{N}(0,1)$. The
results are shown in Fig. 4.

(a) Kaiming (SGD)

(b) $\mathcal{N}(0,1)$ (SGD)

(c) Kaiming (Adam)

(d) $\mathcal{N}(0,1)$ (Adam)

Figure 4: Matrix alignment and weight ratios for uSF for a ResNet-20 network
trained with SGD and Adam on CIFAR10. The forward weight matrices are
initialized with Kaiming in (a) and (c) and with $\mathcal{N}(0,1)$ in (b) and
(d).

We noticed how Adam was able to get a better alignment faster than SGD, and
outperforming it in accuracy when there was a mismatch between the
initialization methods. Furthermore, we observed that the mismatch of
initialization compounds with network depth, making it more difficult to reach
convergence as more layers are added.

Finally, we benchmark all the methods training a ResNet-18 network on
ImageNet. Our results in Fig. 4 and Table 4 showing the top-1 classification
error rate in the validation set, comply with the values previously reported
in [18]. The training details are described in the Appendix Section 6.1.

| Top -1 Error Rate (%)
---|---
BP | 30.39
FA | 85.25
DFA | 82.45
uSF | 34.97
brSF | 37.21
frSF | 36.5
Table 3: Top-1 ImageNet validation error for all the alignment methods.

Table 4: Top-1 ImageNet validation error (%) for a ResNet-18 network trained
with all the feedback alignment methods.

###  4.2 Adversarial Robustness Study

In this section, we benchmark the robustness of our ResNet-20 models trained
on CIFAR-10 against the white-box and black-box attacks described in Appendix
Section 6.3 using the open-source Torchattacks package [43]. The evaluation of
the white-box attacks was performed on the full CIFAR-10 test set, while the
black-box attacks were evaluated on 500 test images only, due to their
significant model query complexities.

White-box attacks We vary the perturbation magnitude, $\epsilon$, from 0 to
0.1 with a step size of 0.01. We used the $L_{\infty}$ as the distance measure
for PGD, APGD and TPGD. Fig. 5 depicts the ResNet-20 accuracy as function of
$\epsilon$. There, when trained with FA and DFA, the model accuracy does not
decrease significantly, unlike the models transporting either full weights or
their sign only. We also notice that frSF and brSF, which transport the sign
of the weight along with a random magnitude, deliver noisier gradients than
uSF, thereby increasing the accuracy of the model as function of $\epsilon$.
For both SGD and Adam optimizers, and in accordance with the accuracy vs.
robustness trade-off [33], the accuracy of the models in Table 2 is
complementary to their robustness.

(a) FGSM (SGD)

(b) PGD (SGD)

(c) APGD (SGD)

(d) TPGD (SGD)

(e) FGSM (Adam)

(f) PGD (Adam)

(g) APGD (Adam)

(h) TPGD (Adam)

Figure 5: Accuracy of ResNet-20 on CIFAR-10 when trained with SGD or Adam
under different white-box adversarial attacks: FGSM in (a) and (e), PGD in (b)
and (f), APGD in (c) and (g), and TRADES attack in (d) and (h).

Black-box attacks These family of attacks don’t intrinsically rely on
gradients and only have access to the target model outputs. Figs. 6(a) and
6(c) show that the Few-Pixel attack is effective at finding adversarial
examples as the number of function of the number of pixels allowed to change
increases. FA, and sign concordant methods are more resistant to this attack
than BP. However, the degradation of the model robustness under the Square
attack, depicted in Fig. 6(b) and 6(d), shows that the training of networks
using synthetic gradient schemes did not provide a performance improvement
against it.

(a) Few-pixel (SGD)

(b) Square (SGD)

(c) Few-pixel (Adam)

(d) Square (Adam)

Figure 6: Accuracy of ResNet-20 on CIFAR-10 when trained with SGD or Adam
under two black-box adversarial attacks: few-pixel in (a) and (c), and Square
in (b) and (d).

##  5 Conclusion

We have presented BioTorch, an open-source project that includes the
implementation of feedback alignment methods and a framework for benchmarking
them. To the best of our knowledge, this is the first framework that allows
for effortless training and evaluation of neural networks using feedback
alignment methods. Along with the description of BioTorch’s main features, we
have provided an accuracy and robustness benchmark of the methods mentioned in
the Section 2.1 of this paper.

While BioTorch is a step forward for the rigorous evaluation of biologically-
motivated models, there are a number of extensions that could improve it, such
as increasing support for other types of layers, including other existing
biologically-motivated algorithms, or adding benchmarks for different domains
and tasks beyond computer vision. We hope the release of our software will
help the research community and encourage the design of new algorithms
inspired by the human brain.

## References

  * [1] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning representations by back-propagating errors.  nature, 323(6088):533–536, 1986.
  * [2] Stephen Grossberg.  Competitive learning: From interactive activation to adaptive resonance.  Cognitive science, 11(1):23–63, 1987.
  * [3] David G Stork.  Is backpropagation biologically plausible.  In International Joint Conference on Neural Networks, volume 2, pages 241–246. IEEE Washington, DC, 1989.
  * [4] Francis Crick.  The recent excitement about neural networks.  Nature, 337(6203):129–132, 1989.
  * [5] Lakshminarayan V Chinta and Douglas B Tweed.  Adaptive optimal control without weight transport.  Neural computation, 24(6):1487–1518, 2012.
  * [6] Timothy P Lillicrap, Adam Santoro, Luke Marris, Colin J Akerman, and Geoffrey Hinton.  Backpropagation and the brain.  Nature Reviews Neuroscience, 21(6):335–346, 2020.
  * [7] Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio.  Difference target propagation.  In Joint european conference on machine learning and knowledge discovery in databases, pages 498–515. Springer, 2015.
  * [8] Yoshua Bengio.  How auto-encoders could provide credit assignment in deep networks via target propagation.  arXiv preprint arXiv:1407.7906, 2014.
  * [9] Geoffrey Hinton.  How to do backpropagation in a brain.  In Invited talk at the NIPS’2007 deep learning workshop, volume 656, 2007.
  * [10] Yann Le Cun.  Learning process in an asymmetric threshold network.  In Disordered systems and biological organization, pages 233–240. Springer, 1986.
  * [11] Sergey Bartunov, Adam Santoro, Blake A Richards, Luke Marris, Geoffrey E Hinton, and Timothy Lillicrap.  Assessing the scalability of biologically-motivated deep learning algorithms and architectures.  arXiv preprint arXiv:1807.04587, 2018.
  * [12] Yoshua Bengio.  Deriving differential target propagation from iterating approximate inverses.  arXiv preprint arXiv:2007.15139, 2020.
  * [13] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman.  Random feedback weights support learning in deep neural networks.  arXiv preprint arXiv:1411.0247, 2014.
  * [14] Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman.  Random synaptic feedback weights support error backpropagation for deep learning.  Nature communications, 7(1):1–10, 2016.
  * [15] Arild Nøkland.  Direct feedback alignment provides learning in deep neural networks.  arXiv preprint arXiv:1609.01596, 2016.
  * [16] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [17] Qianli Liao, Joel Leibo, and Tomaso Poggio.  How important is weight symmetry in backpropagation?  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  * [18] Will Xiao, Honglin Chen, Qianli Liao, and Tomaso Poggio.  Biologically-plausible learning algorithms can scale to large datasets.  arXiv preprint arXiv:1811.03567, 2018.
  * [19] Theodore H Moskovitz, Ashok Litwin-Kumar, and LF Abbott.  Feedback alignment in deep convolutional networks.  arXiv preprint arXiv:1812.06488, 2018.
  * [20] Mohamed Akrout, Collin Wilson, Peter C Humphreys, Timothy Lillicrap, and Douglas Tweed.  Deep learning without weight transport.  arXiv preprint arXiv:1904.05391, 2019.
  * [21] Daniel Kunin, Aran Nayebi, Javier Sagastuy-Brena, Surya Ganguli, Jonathan Bloom, and Daniel Yamins.  Two routes to scalable credit assignment without weight symmetry.  In International Conference on Machine Learning, pages 5511–5521. PMLR, 2020.
  * [22] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * [23] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [24] Mohamed Akrout.  On the adversarial robustness of neural networks without weight transport.  arXiv preprint arXiv:1908.03560, 2019.
  * [25] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [26] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv preprint arXiv:1706.06083, 2017.
  * [27] Alessandro Cappelli, Ruben Ohana, Julien Launay, Laurent Meunier, and Iacopo Poli.  Ropust: Improving robustness through fine-tuning with photonic processors and synthetic gradients.  In ICML 2021 Workshop on Adversarial Machine Learning, 2021.
  * [28] Alessandro Cappelli, Ruben Ohana, Julien Launay, Laurent Meunier, Iacopo Poli, and Florent Krzakala.  Adversarial robustness by design through analog computing and synthetic gradients.  arXiv preprint arXiv:2101.02115, 2021.
  * [29] Yu-Hsin Chen, Joel Emer, and Vivienne Sze.  Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks.  ACM SIGARCH Computer Architecture News, 44(3):367–379, 2016.
  * [30] K Hyoukjun, P Michael, and K Tushar.  Maestro: an open-source infrastructure for modeling dataflows within deep learning accelerators.  arXiv preprint arXiv:1805.02566, 2018.
  * [31] Julien Launay, Iacopo Poli, Kilian Müller, Gustave Pariente, Igor Carron, Laurent Daudet, Florent Krzakala, and Sylvain Gigan.  Hardware beyond backpropagation: a photonic co-processor for direct feedback alignment.  arXiv preprint arXiv:2012.06373, 2020.
  * [32] Roland S Zimmermann.  Comment on" adv-bnn: Improved adversarial defense through robust bayesian neural network".  arXiv preprint arXiv:1907.00895, 2019.
  * [33] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically principled trade-off between robustness and accuracy.  In International Conference on Machine Learning, pages 7472–7482. PMLR, 2019.
  * [34] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai.  One pixel attack for fooling deep neural networks.  IEEE Transactions on Evolutionary Computation, 23(5):828–841, 2019\.
  * [35] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein.  Square attack: a query-efficient black-box adversarial attack via random search.  In European Conference on Computer Vision, pages 484–501. Springer, 2020.
  * [36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [37] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle.  Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program).  arXiv preprint arXiv:2003.12206, 2020.
  * [38] Reproducibility - pytorch 1.9.0.
  * [39] Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky.  Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.  Cited on, 14(8):2, 2012.
  * [40] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [41] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [42] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [43] Hoki Kim.  Torchattacks: A pytorch repository for adversarial attacks.  arXiv preprint arXiv:2010.01950, 2020.

##  6 Appendix

###  6.1 Experiment Details and Hyperparameters

For all datasets, initial learning rates where selected by performing a grid
search within the values [0.1, 0.01, 0.001, 0.0001].

MNIST & Fashion-MNIST The modified LeNet network architecture chosen
corresponds to the first column of Table 5. Networks were trained with the
Stochastic Gradient Descent (SGD) optimizer with a momentum of $0.9$, and
weight decay of $10^{-3}$. Images were resized to $32\times 32$ prior to being
input to the network. We trained with a batch size of 64 for 100 epochs in one
GPU. We decreased the initial learning rate by a factor of $10$ at the $50$th
and the $75$th epoch. The initial learning rates chosen for every method can
be found in Table 6.

LeNet layers | MNIST | CIFAR
---|---|---
Input | 1$\times$32$\times$32 | 3$\times$32$\times$32
1 | conv: 5$\times$5$\times$6, ReLU | conv: 3x$\times$3$\times$32, ReLU
2 | maxpool 2$\times$2 Stride: 2 | maxpool 2$\times$2 Stride: 2
3 | conv: 5$\times$5$\times$16, ReLU | conv: 3$\times$3$\times$64, ReLU
4 | maxpool 2$\times$2 Stride: 2 | maxpool 2$\times$2 Stride: 2
5 | conv: 5$\times$5$\times$120, ReLU | conv: 3$\times$3$\times$128, ReLU
6 | linear: 84, ReLU | linear: 256, ReLU
7 | linear: 10 | linear: 10
Table 5: LeNet Architectures: the convolutional layers format is $K\times
K\times C$, where $K$ is the filter size, $C$ is the number of output
channels. Stride is 1 by default if not further specified. The maxpool layer
format is $K\times K$, where $K$ is the filter size. The linear format is $C$,
where $C$ specifies the number of output neurons.

CIFAR10 The modified LeNet network architecture chosen corresponds to the
second column of Table 5. The ResNet-20 and ResNet-56 architectures [42] have
not been further modified. Networks using the SGD optimizer were trained with
a momentum of $0.9$ and weight decay of $10^{-4}$. Networks using Adam were
trained with the same weight decay and betas parameters equal to
$[0.9,0.999]$. We trained with a batch size of $128$ for $250$ epochs in one
GPU. We decreased the initial learning rate by a factor of $10$ at the
$100$th, $150$th and $200$th epoch. All the initial learning rates for both
SGD and Adam can be found in Table 6 for the LeNet architecture and in Tables
7 and 8 for the ResNet ones.

Dataset | Optimizer | BP | FA | DFA | uSF | brSF | frSF
---|---|---|---|---|---|---|---
MNIST & Fashion-MNIST | SGD | 0.001 | 0.001 | 0.0001 | 0.001 | 0.001 | 0.001
CIFAR10 | SGD | 0.1 | 0.001 | 0.0001 | 0.1 | 0.1 | 0.1
Adam | 0.001 | 0.0001 | 0.0001 | 0.001 | 0.001 | 0.001
Table 6: Initial learning rates for LeNet style networks trained on MNIST, Fashion-MNIST and CIFAR-10, for SGD and Adam optimizers. Optimizer | BP | FA | DFA | uSF | brSF | frSF
---|---|---|---|---|---|---
SGD | 0.1 | 0.1 | 0.001 | 0.1 | 0.1 | 0.1
Adam | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001
Table 7: Initial learning rates for ResNet-20 trained on CIFAR-10, for the SGD
and the Adam optimizers.

ImageNet A ResNet-18 network is trained with a batch size of $256$ and 2 GPUs
for $75$ epochs using SGD with a initial learning rate of $0.1$. A scheduler
decreased the learning rate by a factor of $10$ at the $20$th, the $40$th and
the $60$th epoch. We used a weight decay of $10^{-4}$ and a momentum of $0.9$.
For DFA we used Adam with an initial learning rate of $0.001$. At training
time, a random resized crop of dimensions $224\times 224$ of the original
image or its horizontal flip with the per-pixel mean subtracted is used. When
testing, the image is resized to $256\times 256$ and then a center crop of
$224\times 224$ is used as input to the network.

Optimizer | BP | FA | DFA | uSF | brSF | frSF
---|---|---|---|---|---|---
SGD | 0.1 | 0.01 | 0.0001 | 0.1 | 0.1 | 0.1
Adam | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001
Table 8: Initial learning rates for ResNet-56 trained on CIFAR-10, for the SGD
and the Adam optimizers.

###  6.2 BioTorch Configuration File

Configuration files are structured in 6 parts as shown in the example in
Listing. 1:

Experiment This section includes general information about the experiment.

  1. name (string): Name of the experiment.

  2. output_dir (string): Folder where the outputs of the experiment will be saved.

  3. seed (int): Random seed for the experiment.

  4. deterministic (boolean): If true, will apply PyTorch deterministic operations to guarantee reproducibility.

Data This section includes information about the data that will be used to
train the model.

  1. dataset (string): Name of the dataset (see supported datasets in the code repository)

  2. dataset_path (string): If dataset is not supported by Torchvision, a path can be provided (e.g., ImageNet)

  3. target_size (int): Images will be resized to the target size (if specified)

  4. num_workers (int): Number of threads for data loading

Model This section includes information about the model. For a list of
supported architectures, initializations and feedback alignment methods we
redirect the reader to the code repository.

  1. architecture (string): Name of the model architecture

  2. mode/type (string): Name of the feedback alignment method

  3. mode/options/init (string): Layer initialization method

  4. mode/options/gradient_clip (boolean): If true, will apply gradient clipping (-1, 1)

Training Parameters This section includes information about the training
hyperparameters. As it is very standard and for brevity, we redirect the
reader to the configuration file example and the code repository to see the
methods supported. We highlight the inclusion of the tracking alignment
metrics.

  1. metrics/weight_alignment (boolean): Track the angle between layers forward and backward weights

  2. metrics/weight_ratio (boolean): Track the norm ratio between layers forward and backward weights

Infrastructure This section includes information about the hardware.

  1. gpus/ (int/list): If -1 will use CPU, else will use the corresponding GPUs devices

Evaluation This section includes information about the evaluation.

  1. evaluation/ (boolean): If true, it will run an evaluation in the test and store the metrics after the training

⬇

1# 1. Experiment

2experiment

3 name "fa_lr_0.1"

4 output_dir "trained_models_final/cifar10/resnet20/"

5 # For reproducibility

6 seed 2021

7 deterministic true

8

9# 2. Data

10data

11 dataset "cifar10_benchmark"

12 dataset_path null

13 target_size 32

14 num_workers 0

15

16# 3. Model

17model

18 architecture "resnet20"

19 mode

20 type "fa"

21 options

22 init "xavier"

23 gradient_clip false

24

25 pretrained false

26 # checkpoint:

27 loss_function

28 name "cross_entropy"

29

30# 4. Training Parameters

31training

32 hyperparameters

33 epochs 250

34 batch_size 128

35

36 optimizer

37 type "SGD"

38 lr 0.1

39 weight_decay 0.0001

40 momentum 0.9

41

42 lr_scheduler

43 type "multistep_lr"

44 gamma 0.1

45 milestones [100, 150, 200]

46

47 metrics

48 top_k 5

49 display_iterations 500

50 weight_alignment true

51 weight_ratio true

52

53# 5. Infrastructure

54infrastructure

55 gpus 0

56

57# 6. Evaluation

58evaluation true

Listing 1: BioTorch configuration file example to train a ResNet-20 FA model
on CIFAR10.

###  6.3 Adversarial Attacks Description

FGSM One of the earliest versions of to generate an adversarial example
$x^{\textrm{adv}}$ using a model to attack $f$ is the Fast Gradient Sign
Method (FGSM) [25] which updates the original image $x$, whose label is
$y^{*}$, using the direction of the sign of the gradient with respect to the
image as follows:

$x^{\textrm{adv}}=x+\epsilon\;\textrm{sign}\big{(}\nabla_{x}\,\mathcal{L}(f(x),y^{*})\big{)}$
(5)

PGD, APGD and TPGD The Projected Gradient Descent (PGD) [26] is a variant of
FGSM attack applied multiple times with a step size $\alpha$. That is to say:

$x^{\textrm{adv}}_{0}=x,\quad\quad
x^{\textrm{adv}}_{t+1}=\textrm{Clip}_{x}^{\epsilon}\bigg{(}x^{\textrm{adv}}_{t}+\alpha\;\textrm{sign}\big{(}\nabla_{x^{\textrm{adv}}_{t}}\,\mathcal{L}(f(x^{\textrm{adv}}_{t}),y^{*})\big{)}\bigg{)}$
(6)

where $x^{\textrm{adv}}_{t}$ denotes the adversarial example after $t$-steps
and $\textrm{Clip}_{x}^{\epsilon}(\cdot)$ refers to the function that clips
its parameter to be within the $\epsilon$-ball of center the original input
$x$. The averaged PGD (APGD)[32] estimates the gradient using the
approximation
$\frac{1}{k}\,\sum_{i}^{k}\nabla_{x^{\textrm{adv}}_{t}}\,\mathcal{L}(f(x^{\textrm{adv}}_{t}),y^{*})\approx\mathbb{E}\left[\nabla_{x^{\textrm{adv}}_{t}}\,\mathcal{L}(f(x^{\textrm{adv}}_{t}),y^{*})\right]$.
The TPGD variant of PGD was used in the TRADES adersarial training [33] with
the only difference of the KL divergence loss instead of the cross-entropy
loss function.

Few-Pixel Carefully designed perturbations that are even unrecognizable by
human eyes can be appropriately perturbed using the one-pixel attack [34]
based on differential evolution. While common adversarial attacks are
constructed by perturbating all pixels with an overall constraint on the
strength of the accumulated modification (e.g., the perturbation $\epsilon$ in
FGSM and the ball radius $\epsilon$ and the step size $\alpha$ in PGD), the
few-pixel attack limits the number of pixels to modify without confining the
strength of modification. In this paper we consider the n-pixel attack where
$\textrm{n}\in\\{1,2,3,4,5\\}$.

Square: Without the need for the local gradient information, the square attack
[35] is a score-based that is based on a randomized search scheme which
selects localized square shaped updates at random positions to iteratively
generate adversarial samples on the surface of the $l_{\infty}$\- or
$l2$-balls. This attack outperforms many more sophisticated attacks in terms
of success rate and query efficiency.