2019 Pages Lukas JendeleContributed equally.1<EMAIL_ADDRESS>
and Ondrej Skopek1<EMAIL_ADDRESS>
1 Department of Computer Science, ETH Zürich
and Anton S. Becker2,3<EMAIL_ADDRESS>
2 Institute of Diagnostic and Interventional Radiology, University Hospital
Zürich
3 Department of Health Sciences and Technology, ETH Zürich
and Ender Konukoglu4<EMAIL_ADDRESS>
4 Computer Vision Laboratory, ETH Zürich

# Adversarial Augmentation for Enhancing Classification of Mammography Images

###### Abstract

Supervised deep learning relies on the assumption that enough training data is
available, which presents a problem for its application to several fields,
like medical imaging. On the example of a binary image classification task
(breast cancer recognition), we show that pretraining a generative model for
meaningful image augmentation helps enhance the performance of the resulting
classifier. By augmenting the data, performance on downstream classification
tasks could be improved even with a relatively small training set. We show
that this “adversarial augmentation” yields promising results compared to
classical image augmentation on the example of breast cancer classification.

###### keywords:

Generative Adversarial Network, Augmentation, CycleGAN

##  1 Introduction

Deep learning in computer vision has achieved great results in the past few
years [Deng et al. (2009), Karras et al. (2017)]. Most of these have been
enabled by more computational power and large amounts of data. Unfortunately,
in many scientific fields such as medical imaging, there are usually several
orders of magnitude fewer data samples to work with than in large-scale
computer vision datasets. Leaving aside issues like anonymization and privacy,
this poses several specific problems for anyone wishing to use medical imaging
datasets:

  1. 1.

Scarcity — Data is hard to obtain, usually only a few samples are available
per dataset.

  2. 2.

Bias — Medical and other small datasets usually contain many more negative
(healthy) images than positive ones (with a valid and confirmed illness). The
reason for that is that the data usually comes from a real-world diagnostic
process, where data is obtained even at a low suspicion threshold, since the
potential harms of the imaging procedure are far outweighed by the benefit of
a prompt diagnosis. Furthermore, in screening settings a large population of
completely symptom-free subjects is deliberately examined.

Often present are also “confirmation” images – for a patient with a positive
finding, many more images will be made to confirm the diagnosis and monitor
the progress. This only increases the bias, as the dataset then has several
positive images of the same patient. In other fields, variations on these
processes also exist, all resulting in a similar bias.

  3. 3.

Noise — Introduced by capturing devices, errors made during data processing or
storage, or from a naturally noisy population (e.g. synthetic implants, marker
wires, or prior surgery related to the illness).

All three of these issues pose a significant challenge for training
classification models. In this work, we aim to partially alleviate the first
two problems in the context of binary image classification. Our contributions
are the following:

  1. 1.

We train a generative model that has the ability to transform data from one
class to the other and back with a CycleGAN architecture [Zhu et al. (2017)].

  2. 2.

We show that the classifier is partially fooled into thinking that the
transformed images are of the respective real class-label distributions.

  3. 3.

We show that the performance of the classifier may improve when its training
data is augmented with the transformed images, in comparison to classical
image augmentation.

##  2 Related work

Generative Adversarial Networks (GANs), proposed by Goodfellow et al. (2014),
have shown great potential for generating or modifying images. Many studies
focused on image augmentation using GANs Shrivastava et al. (2017); Mueller et
al. (2018); Bousmalis et al. (2018). The application to the medical domain is
logical, because it is generally difficult to obtain data there, and all
datasets are naturally heavily imbalanced. Shin et al. (2018) focuses on brain
MRI augmentation using paired image-to-image translation similar to the
pix2pix approach Isola et al. (2017).

However, paired images (e.g. the same breast in the same view with and without
cancer) are very hard to obtain. Thus, we focus on unpaired image
augmentation. In their work on CycleGAN, Zhu et al. (2017) used a pair of GANs
coupled with a cycle-consistent loss for unpaired image-to-image translation,
and succeed in converting images between two domains (e.g. horses to zebras).
In our work, we adopt this idea to generate cancerous features into or remove
them from mammography images.

Parallel to our research, Sun et al. (2018) have applied the CycleGAN
architecture to augment brain and liver MRI scans. Aligned with our work, they
show that such augmentation boosts the classifier’s performance.

For the actual cancerous lesion detection, there have been several studies
utilizing deep neural networks on image patches, like Lévy and Jain (2016).
The reason for looking at smaller patches is mostly because of dimensionality
reduction. There have also been attempts at detection by training on whole
images Ribli et al. (2018); Shen (2017); Hussain et al. (2017). They all
augment the dataset by translating, rotating, or flipping the images to
improve the system’s performance, which we also compare to in our experiments.

##  3 Model

Our approach consists of two models, trained separately. In the first step, we
train a specific GAN architecture to learn a transformation from the domain of
images of one class label to the domain of images of the other class labels.
In the second step, we use the generative model to augment a Faster R-CNN
classifier Ren et al. (2015) to improve its performance.

###  3.1 Generative augmentation model

[Healthy $\to$ cancer]  [Cancer $\to$ healthy]

Figure 1: Given an image dataset with two classes (cancerous and healthy
breast scans), the generative model learns to transform images from one class
to the other.

The generative model is based on CycleGAN Zhu et al. (2017). Its goal is to
perform unpaired translation of images from one domain to another, and back.
It achieves this by training two generator–discriminator pairs and introducing
a cycle-consistency loss. In our case, we apply it to generate and remove
cancerous features from mammography images. Figure 1 shows the output of the
generative model on two training samples.

More formally, CycleGAN transforms images from a domain $X$ to another domain
$Y$. For that, it uses two independent mappings, $G_{Y}:X\to Y$ and
$G_{X}:Y\to X$. To train these mappings directly, one would need paired
images, which are very hard to obtain (for example, the same patient’s image
with and without breast cancer, in the exact same orientation). Instead,
CycleGAN uses a GAN-like loss of introducing a discriminator, that attempts to
differentiate generated images from the empirical domain $\hat{X}=G_{X}(Y)$
from the real images from $X$ by learning a mapping
$D_{Y}:(X\cup\hat{X})\to[0,1]$ (analogously for $D_{X}$).

Furthermore, it adds a cycle-consistency loss $\mathcal{L}_{cyc}$, which
enforces the "identity" property $X\approx G_{X}(G_{Y}(X))$. All of this is
analogously done for domain $Y$ as well. Figure 2 shows a simple diagram of
the model.

The loss is composed of the following partial loss terms. The first is the
classic adversarial GAN loss, where $D_{X}$ is the discriminator of the GAN on
domain $X$, and $G_{X}$ is the generator of samples in the domain $X$ given a
sample from $Y$.

$\mathcal{L}_{GAN}(G_{Y},D_{Y})=\mathrm{E}_{y\sim
p_{data}(y)}\left[\log{D_{Y}(x)}\right]+\mathrm{E}_{x\sim
p_{data}(x)}\left[\log{\left(1-D_{Y}(G_{Y}(x))\right)}\right]$

For training stability reasons, our implementation uses the alternative LSGAN
Mao et al. (2017) loss function, with parameters $a=-1$ and $b=c=0$.

$\mathcal{L}_{GAN}(G_{Y},D_{Y})=\frac{1}{2}\mathrm{E}_{y\sim
p_{data}(y)}\left[(D_{Y}(x)-1)^{2}\right]+\frac{1}{2}\mathrm{E}_{x\sim
p_{data}(x)}\left[(D_{Y}(G_{Y}(x)))^{2}\right]$

The second loss term corresponds to cycle-consistency losses for both
directions.

$\mathcal{L}_{cyc}(G_{X},G_{Y})=\mathrm{E}_{x\sim
p_{data}(x)}\left[||x-G_{X}(G_{Y}(x))||_{1}\right]+\mathrm{E}_{y\sim
p_{data}(y)}\left[||y-G_{Y}(G_{X}(y))||_{1}\right]$

The loss of the final model sums all the partial loss terms with constant
weights (regarded as hyperparameters).

$\mathcal{L}(G_{X},D_{X},G_{Y},D_{Y})=\mathcal{L}_{GAN}(G_{X},D_{X})+\mathcal{L}_{GAN}(G_{Y},D_{Y})+\lambda_{cyc}\mathcal{L}_{cyc}(G_{X},G_{Y})$

The objective of training is summarized by the following optimization problem.

$G_{X}^{*},G_{Y}^{*}=\arg\min_{G_{X},G_{Y}}\max_{D_{X},D_{Y}}\mathcal{L}(G_{X},D_{X},G_{Y},D_{Y}).$

####  3.1.1 Conditioning on regions of interest

To enhance the usefulness of our model, we add another input modality into to
our generative model that represents regions of interest in the picture. For
example, for breast cancer imaging, this modality could contain a boolean mask
indicating segmented regions with “suspicious” (potentially cancerous) tissue.
This also allows for encoding of various invariants into the dataset. By
varying the additional mask position spatially, we obtain several variants of
the transformed image, which together encode spatial equivariance of cancerous
tissue, which might not be represented in the original dataset due to a low
number of samples. The datasets we use all contain masks (of varying quality)
with highlighted lesions or benign masses of the same dimension as the image.

To model the additional data source, we append another channel to our input
image and let the model train using both the original image and the mask as
both input and output. The generator now obtains a “two channel” image, and
produces two channels instead of one. The final loss function is a sum of our
$\cal{L}$ loss function applied to each channel individually. The rest of the
model remains the same. The changes in the formulation of both generators and
discriminators are the following (shown for $X\to Y$, $Z$ is the domain of
masks):

$\displaystyle G_{Y}$ $\displaystyle:(X,Z)\to(Y,Z)$ $\displaystyle D_{Y}$
$\displaystyle:(X\cup\hat{X},Z)\to[0,1]$

Figure 2: CycleGAN model diagram with the cycle-consistency loss
$\mathcal{L}_{cyc}$, only shown $X\to Y$.

####  3.1.2 Removing checkerboard artifacts

Empirically, models with deconvolutional layers tend to exhibit “checkerboard”
artifacts, especially when trained for longer amounts of time Odena et al.
(2016). Therefore in our experiments, we 1) substitute a deconvolution with
nearest-neighbor upsampling followed by a convolution, and 2) we initialize
the kernel weights using ICNR Aitken et al. (2017). Generally, deconvolution
preserves more details and produces less blurry results compared to upsampling
and followed by convolution. We also evaluated bilinear upsampling, but it
empirically produced more artifacts than nearest-neighbor upsampling.

###  3.2 Neural classification model

The classifier model used for all experiments was an adaptation of Faster
R-CNN Ren et al. (2015) that ranked second in the DREAM breast cancer
detection challenge proposed by Ribli et al. (2018). Faster R-CNN is a
convolution-based network capable of classifying and localizing objects in an
image. Pure classification networks (predicting a binary answer) are easier to
train, and thus more commonly used for mammography images. However, we believe
that localizing malignant tumors is important if the system was to be
implemented in clinical routine, since it helps in verifying the decision. The
network is based on ResNet-50, a 50 layered network with residual connections
pretrained on ImageNet Deng et al. (2009). Similarly to Ribli et al. (2018),
we also changed the following parameters: we enabled the proposal network, and
changed the proposal non-maximal suppression threshold to 0.5.

##  4 Experiments

To validate our ideas and claims, we propose several simple experiments in the
domain of breast cancer recognition from 2D mammography images.

###  4.1 Model implementation

The generative augmentation models for all our experiments are based on the
CycleGAN Zhu et al. (2017) architecture, and are implemented in
TensorFlow111Based on the TensorFlow research CycleGAN implementation:
https://github.com/tensorflow/models Abadi et al. (2016). More details about
the architectures and training procedures are provided in Appendix A.

###  4.2 Datasets

There are several datasets that relate to breast cancer diagnosis. In most of
these one can observe the limitations that we outlined in the introduction.
For our experiments, we used the following datasets: (1) BCDR Guevara Lopez et
al. (2012), Breast Cancer Digital Repository, several datasets from Portugal;
(2) INbreast Moreira et al. (2012), INbreast digital breast database, also
from Portugal. Samples with a BiRads classification greater than 3 were
considered as positive (cancerous), lower than 3 were considered negative
(healthy); and (3) CBIS–DDSM Sawyer Lee et al. (2016), Curated Breast Imaging
Subset of DDSM (Digital Database for Screening Mammography) from the USA.

Dataset | Cancerous | Healthy
---|---|---
BCDR-1 | 55 | 199
BCDR-2 | 44 | 651
INbreast | 100 | 270
CBIS | 672 | 960

Dataset | Cancerous | Healthy
---|---|---
Training | 655 | 1538
Evaluation | 116 | 272
Testing | 100 | 270

Table 1: Number of samples in the various datasets.

For the generative model, we use BCDR-1 and BCDR-2 (merged together) for
training. For the classifier, we use both BCDR datasets along with CBIS with
an 85% training and 15% evaluation split. Due to a high noise ratio in CBIS,
we only used it for the classifier. We use the held-out INbreast dataset as a
test dataset for both models. All images were downscaled to $256\times 204$
pixels due to hardware limitations. We also experimented with $512\times 408$
pixels, but the image quality was poorer. Table 1 shows the number of samples
in the respective datasets.

###  4.3 Training a classifier

Our Faster R-CNN Ribli et al. (2018) based classifier was trained to localize
malignant and benign lesions. We convert the pixel masks into a set of
bounding box by applying Otsu threshold segmentation and taking the bounding
box around every disconnected region. Images with no lesions or lesions with a
bounding box area smaller than $10$ pixels were discarded, as R-CNN doesn’t
need to train on “negative” images. For each image, the model predicts a set
of bounding boxes, corresponding scores, and classes. For evaluation, we treat
an image as positive (cancerous) if any of the bounding boxes score with a
malignant class is higher than a chosen, constant confidence threshold.

We train the classifier on different datasets for a maximum of 100,000 steps
(batch size 8) and pick the best model based on ROC AUC Bradley (1997) on the
evaluation set. Based on inspection of the evaluation set loss, we empirically
chose the models trained for $47,500$ steps (for all model variants).

###  4.4 “Fooling” a trained classifier

As a first step, we want to see if our classifier, trained only on original
images, is “fooled” by the generated images. In other words, for correctly
classified images, how many times does the label change after we run the
images through the generative augmentation model? We evaluate this question on
all of our test data (see Section 5).

Figure 3: Evaluation diagram whether the generator $G_{Y}$ fooled the trained
classifier into thinking that the generated image $G_{Y}(x)$ is from domain
$Y$.

###  4.5 Improving the classifier

Secondly, we evaluate if a classifier trained in the same way on a mixed
dataset of original and augmented images using the generative model performs
better in terms of both classification metrics and “being fooled” We also
compare the model to standard augmentation techniques such as image
translation, rotation, and horizontal flipping. We use the same
training/evaluation/testing split, but balance the training dataset by
converting all the healthy images to cancerous ones, and adding them to our
dataset. We then balance the dataset in a similar way as in Section 4.3.

##  5 Results

To visualize the results of our generative augmentation models, we show a
random uniform selection of images augmented by our generative model from the
INbreast test dataset in Figure 4 and 5 (Appendix B).

Classifier training data | Correctly clf. % | Fooled % | ROC AUC % | F1 score %
---|---|---|---|---
Original | $76.22\pm 4.08$ | $38.49\pm 3.31$ | $83.50\pm 1.47$ | $62.53\pm 0.40$
Classically augmented | $80.54\pm 0.47$ | $33.34\pm 2.53$ | $79.05\pm 1.94$ | $62.63\pm 2.50$
GAN-augmented | $80.99\pm 1.96$ | $30.91\pm 8.83$ | $82.04\pm 0.57$ | $63.81\pm 2.04$
Table 2: Fooling and improving the classifier evaluated on the test dataset
INbreast (different patient population than the training set). GAN-augmented
images are from the unconditioned GAN model because of better image quality.
Each run was repeated three times — shown are the average and the standard
deviation for each value.

The first and second columns of Table 2 show that the classifier learns to be
less fooled by our generative augmentation model if we augment the training
set images using the same model, which confirms the intuition that this makes
the classifier slightly more robust.

As shown in the first row of Table 2, the classifier performs reasonably well
when trained on the original dataset and evaluated on a test split from that
dataset (both in terms of ROC AUC and F1 score). The F1 score is computed
using a custom bounding box proposal confidence threshold of $0.23$, same as
in Ribli et al. (2018).

When the training set images are augmented by our GAN (third row), the average
ROC AUC goes down slightly, but the error margin is too big to produce a
conclusive result. As was previously shown by Becker et al. (2018b, a) and our
subjective assessment, this suggests that the new GAN-generated data might be
challenging to classify for our classifier. The same conclusion applies for
the experiment where we augment the training set images using traditional
image augmentation techniques.

##  6 Discussion

Overall, our GAN training has been very prone to checkerboard and “S”-shaped
artifacts, as can be partially seen in Figures 4 and 5 (Appendix B). We also
experimented with both higher ($512\times 408$ px) and lower resolutions
($256\times 208$ px) of images: the lower resolutions generally had fewer
artifacts and faster training times, but a higher resolution is desirable when
thinking about moving to full-field mammographic images in the future.
Unfortunately, due to GPU memory limitations resolution could not be further
increased. Our GAN models and RCNN-based classifiers train in less than 24
hours on an NVIDIA TITAN Xp GPU.

The classifier results are inconclusive, and it is not clear that adding our
augmented images helps the classifier achieve better performance or not. We
hypothesize that this might be due to the noise in our data, as the results of
Sun et al. (2018) suggest that the overall method is sound and can improve
classifier performance if applied well.

##  7 Future work

Possible future improvements to our work include investigating upscaling the
resolution without obtaining artifacts with approaches similar to Wang et al.
(2018), stabilizing the conditioned model training and results, and also
leveraging that model fully to augment the images in pre-specified places. For
a more detailed image, we could explore approaches similar to Self-Attention
GAN Zhang et al. (2018), which promises to pay close attention to parts of the
input image for output generation. This would also help in interpreting the
resulting changes done by the GAN. Unfortunately, this approach is very
memory-expensive.

Traditionally, Variational Autoencoders Kingma and Welling (2013) (VAEs) lack
detail in the output images and GANs lack “truthfulness” — they may
overgenerate parts of the image Sajjadi et al. (2018). As a more hybrid
approach, we could combine a VAE with a GAN to model both the location and the
image details jointly with one model, similarly to the approaches in Liu et
al. (2017); Huang et al. (2018); Andermatt et al. (2018). To simplify the
model, one could also try using a StarGAN-like Choi et al. (2018) approach by
only using one generator/discriminator pair which is conditioned by the class
label, instead of using two generators and discriminators.

##  8 Conclusion

In our work, we have shown that for binary classification on images, there
exists a simple way to potentially increase prediction accuracy by generative
dataset augmentation. Leveraging the idea behind CycleGAN, we have designed a
GAN that is able to translate images from one class label to the other, and
use that property to augment the training dataset of a classifier into a
bigger, more balanced, and less sparse dataset. We have provided a proof of
concept implementation and shown that on the challenging noisy example case of
breast cancer recognition from mammography images, we may be able to help
improve performance of classifiers. This suggests our generative augmentation
model learns a meaningful approximation of the manifolds of our class labels.

We would like to thank the Computer Vision Lab at ETH Zürich for providing us
with computational resources.

## References

  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, Xiaoqiang Zheng, and Google Brain.  TensorFlow: A System for Large-Scale Machine Learning.  In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16)_ , pages 265–284, 2016.  ISBN 978-1-931971-33-1.  10.1038/nn.3331.  URL https://www.usenix.org/conference/osdi16/technical-sessions/presentation/abadi.
  * Aitken et al. (2017) Andrew Aitken, Christian Ledig, Lucas Theis, Jose Caballero, Zehan Wang, and Wenzhe Shi.  Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize.  _arXiv preprint arXiv:1707.02937_ , 2017.
  * Andermatt et al. (2018) Simon Andermatt, Antal Horváth, Simon Pezold, and Philippe C. Cattin.  Pathology Segmentation using Distributional Differences to Images of Healthy Origin.  _CoRR_ , abs/1805.10344, 2018.  URL http://arxiv.org/abs/1805.10344.
  * Becker et al. (2018a) Anton S Becker, Lukas Jendele, Ondrej Skopek, Nicole Berger, Soleen Ghafoor, Magda Marcon, and Ender Konukoglu.  Injecting and removing malignant features in mammography with CycleGAN: Investigation of an automated adversarial attack using neural networks.  _arXiv preprint arXiv:1811.07767_ , 2018a.
  * Becker et al. (2018b) Anton S. Becker, Lukas Jendele, Ondrej Skopek, Soleen Ghafoor, Nicole Berger, Magda Marcon, and Ender Konukoglu.  Generative Neural Network Inserting or Removing Cancer into Mammograms Fools Radiologists and Deep Learning Alike: Example of an Adversarial Attack.  In _Proceedings of the RSNA Annual Meeting_ , 2018b.
  * Bousmalis et al. (2018) Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Yunfei Bai, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, et al.  Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4243–4250. IEEE, 2018.
  * Bradley (1997) Andrew P Bradley.  The use of the area under the ROC curve in the evaluation of machine learning algorithms.  _Pattern recognition_ , 30(7):1145–1159, 1997\.
  * Choi et al. (2018) Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on_ , pages 248–255. IEEE, 2009.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Nets.  In _Advances in neural information processing systems_ , pages 2672–2680, 2014.
  * Guevara Lopez et al. (2012) Miguel Angel Guevara Lopez, Naimy González Posada, Daniel Moura, Raúl Pollán, José Franco-Valiente, César Ortega, Manuel Del Solar, Guillermo Díaz-Herrero, Isabel Pereira M A Ramos, Joana Pinheiro Loureiro, Teresa Cardoso Fernandes, and Bruno Ferreira M Araújo.  BCDR: A Breast Cancer Digital Repository, 2012.
  * Hahnloser et al. (2000) Richard H. R. Hahnloser, Rahul Sarpeshkar, Misha A. Mahowald, Rodney J. Douglas, and H. Sebastian Seung.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.  _Nature_ , 405, 2000.  URL http://dx.doi.org/10.1038/35016072.
  * Huang et al. (2018) Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal Unsupervised Image-to-image Translation.  In _ECCV_ , 2018.
  * Hussain et al. (2017) Zeshan Hussain, Francisco Gimenez, Darvin Yi, and Daniel Rubin.  Differential Data Augmentation Techniques for Medical Imaging Classification Tasks.  In _AMIA Annual Symposium Proceedings_ , volume 2017, page 979. American Medical Informatics Association, 2017.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-Image Translation with Conditional Adversarial Networks.  _CVPR_ , 2017.
  * Karras et al. (2017) Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Lei Ba.  Adam: A Method for Stochastic Optimization.  _International Conference on Learning Representations (ICRL)_ , 2015\.  URL http://arxiv.org/abs/1412.6980.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling.  Auto-Encoding Variational Bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Lévy and Jain (2016) Daniel Lévy and Arzav Jain.  Breast Mass Classification from Mammograms using Deep Convolutional Neural Networks.  _arXiv preprint arXiv:1612.00542_ , 2016.
  * Liu et al. (2017) Ming-Yu Liu, Thomas Breuel, and Jan Kautz.  Unsupervised image-to-image translation networks.  In _Advances in Neural Information Processing Systems_ , pages 700–708, 2017.
  * Maas et al. (2013) Andrew L Maas, Awni Y Hannun, and Andrew Y Ng.  Rectifier Nonlinearities Improve Neural Network Acoustic Models, 2013\.
  * Mao et al. (2017) Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least Squares Generative Adversarial Networks.  In _Computer Vision (ICCV), 2017 IEEE International Conference on_ , pages 2813–2821. IEEE, 2017.
  * Moreira et al. (2012) Inês C Moreira, Igor Amaral, Inês Domingues, António Cardoso, Maria João Cardoso, and Jaime S Cardoso.  INbreast: Toward a Full-field Digital Mammographic Database.  _Academic radiology_ , 19(2):236–248, 2012.
  * Mueller et al. (2018) Franziska Mueller, Florian Bernard, Oleksandr Sotnychenko, Dushyant Mehta, Srinath Sridhar, Dan Casas, and Christian Theobalt.  GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB.  In _Proceedings of Computer Vision and Pattern Recognition (CVPR)_ , 2018.  URL https://handtracker.mpi-inf.mpg.de/projects/GANeratedHands/.
  * Nair and Hinton (2010) Vinod Nair and Geoffrey E Hinton.  Rectified Linear Units improve Restricted Boltzmann Machines.  In _Proceedings of the 27th International Conference on Machine Learning (ICML)_ , pages 807–814, 2010.
  * Odena et al. (2016) Augustus Odena, Vincent Dumoulin, and Chris Olah.  Deconvolution and Checkerboard Artifacts.  _Distill_ , 2016.  10.23915/distill.00003.  URL http://distill.pub/2016/deconv-checkerboard.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 91–99, 2015.  URL https://arxiv.org/abs/1506.01497.
  * Ribli et al. (2018) Dezso Ribli, Anna Horváth, Zsuzsa Unger, Péter Pollner, and István Csabai.  Detecting and classifying lesions in mammograms with Deep Learning.  _Scientific Reports_ , 8(1):4165, 2018.  ISSN 2045-2322.  10.1038/s41598-018-22437-z.  URL https://doi.org/10.1038/s41598-018-22437-z.
  * Sajjadi et al. (2018) M. S. M. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly.  Assessing Generative Models via Precision and Recall.  In _Workshop on Theoretical Foundations and Applications of Deep Generative Models (TADGM) at the 35th International Conference on Machine Learning (ICML)_ , 2018.
  * Sawyer Lee et al. (2016) Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, and Daniel Rubin.  Curated Breast Imaging Subset of DDSM.  _The Cancer Imaging Archive_ , 2016.  http://dx.doi.org/10.7937/K9/TCIA.2016.7O02S9CY.
  * Shen (2017) Li Shen.  End-to-end Training for Whole Image Breast Cancer Diagnosis using An All Convolutional Design.  _arXiv preprint arXiv:1708.09427_ , 2017.
  * Shin et al. (2018) Hoo-Chang Shin, Neil A Tenenholtz, Jameson K Rogers, Christopher G Schwarz, Matthew L Senjem, Jeffrey L Gunter, Katherine P Andriole, and Mark Michalski.  Medical image synthesis for data augmentation and anonymization using generative adversarial networks.  In _International Workshop on Simulation and Synthesis in Medical Imaging_ , pages 1–11. Springer, 2018.
  * Shrivastava et al. (2017) Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russell Webb.  Learning from Simulated and Unsupervised Images through Adversarial Training.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2242–2251, 2017.
  * Sun et al. (2018) Liyan Sun, Jiexiang Wang, Xinghao Ding, Yue Huang, and John Paisley.  An Adversarial Learning Approach to Medical Image Synthesis for Lesion Removal.  _arXiv preprint arXiv:1810.10850_ , 2018.
  * Wang et al. (2018) Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018.
  * Zhang et al. (2018) Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena.  Self-Attention Generative Adversarial Networks.  _arXiv:1805.08318_ , 2018.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks.  In _Computer Vision (ICCV), 2017 IEEE International Conference on_ , 2017.

##  Appendix A Model implementation

We train all our GAN models for $40,000$ steps, using a learning rate of
$1\cdot 10^{-4}$ for the discriminators and $2\cdot 10^{-4}$ for the
generators. The optimization is performed using Adam Kingma and Ba (2015) and
a batch size of 1. All code is available on
GitHub222https://github.com/BreastGAN/augmentation.

The architectures of both discriminators are the same: 4 convolutional layers
with reflection padding, with filters of size 64, 128, 256, 512 and stride 2
for all layers except for the last one that has stride 1, with a LeakyReLU
activation function Hahnloser et al. (2000); Nair and Hinton (2010); Maas et
al. (2013): $\max(0.2x,x)$. All the convolutions have a kernel size of
$4\times 4$. The output is subsequently flattened to one channel using a
stride 1 convolution, with a sigmoid activation function.

Both generator networks consist of two convolutions with stride 2 to compress
the dimensionality of the image followed by 9 ResNet blocks (2 convolutions
layers each). Lastly, the result is upsampled using two additional
convolutional layers as described in Section 3.1.2. All the generator layers
use ReLU activation functions.

##  Appendix B Random samples from our GAN augmentation models

[Healthy (top) to cancerous (bottom).]

[Cancerous (top) to healthy (bottom).]

Figure 4: Random samples of images from our trained GAN (without masks,
$512\times 408$ px).

[Healthy (top) to cancerous (bottom), mask (middle).]

[Cancerous (top) to healthy (bottom), mask (middle).]

Figure 5: Random samples of images from our trained GAN (with masks,
$512\times 408$ px).


# Highway Networks

Rupesh Kumar Srivastava  Klaus Greff  Jürgen Schmidhuber  The Swiss AI Lab
IDSIA Istituto Dalle Molle di Studi sull’Intelligenza Artificiale Università
della Svizzera italiana (USI) Scuola universitaria professionale della
Svizzera italiana (SUPSI) Galleria 2, 6928 Manno-Lugano, Switzerland

###### Abstract

There is plenty of theoretical and empirical evidence that depth of neural
networks is a crucial ingredient for their success. However, network training
becomes more difficult with increasing depth and training of very deep
networks remains an open problem. In this extended abstract, we introduce a
new architecture designed to ease gradient-based training of very deep
networks. We refer to networks with this architecture as highway networks,
since they allow unimpeded information flow across several layers on
_information highways_. The architecture is characterized by the use of gating
units which learn to regulate the flow of information through a network.
Highway networks with hundreds of layers can be trained directly using
stochastic gradient descent and with a variety of activation functions,
opening up the possibility of studying extremely deep and efficient
architectures.

deep learning

Note: A full paper extending this study is available at
http://arxiv.org/abs/1507.06228, with additional references, experiments and
analysis.

##  1 Introduction

Many recent empirical breakthroughs in supervised machine learning have been
achieved through the application of deep neural networks. Network depth
(referring to the number of successive computation layers) has played perhaps
the most important role in these successes. For instance, the top-5 image
classification accuracy on the 1000-class ImageNet dataset has increased from
$\sim$84% (Krizhevsky et al., 2012) to $\sim$95% (Szegedy et al., 2014;
Simonyan & Zisserman, 2014) through the use of ensembles of deeper
architectures and smaller receptive fields (Ciresan et al., 2011a, b, 2012) in
just a few years.

On the theoretical side, it is well known that deep networks can represent
certain function classes exponentially more efficiently than shallow ones
(e.g. the work of Håstad (1987); Håstad & Goldmann (1991) and recently of
Montufar et al. (2014)). As argued by Bengio et al. (2013), the use of deep
networks can offer both computational and statistical efficiency for complex
tasks.

However, training deeper networks is not as straightforward as simply adding
layers. Optimization of deep networks has proven to be considerably more
difficult, leading to research on initialization schemes (Glorot & Bengio,
2010; Saxe et al., 2013; He et al., 2015), techniques of training networks in
multiple stages (Simonyan & Zisserman, 2014; Romero et al., 2014) or with
temporary companion loss functions attached to some of the layers (Szegedy et
al., 2014; Lee et al., 2015).

In this extended abstract, we present a novel architecture that enables the
optimization of networks with virtually arbitrary depth. This is accomplished
through the use of a learned gating mechanism for regulating information flow
which is inspired by Long Short Term Memory recurrent neural networks
(Hochreiter & Schmidhuber, 1995). Due to this gating mechanism, a neural
network can have paths along which information can flow across several layers
without attenuation. We call such paths _information highways_ , and such
networks _highway networks_.

In preliminary experiments, we found that highway networks as deep as 900
layers can be optimized using simple Stochastic Gradient Descent (SGD) with
momentum. For up to 100 layers we compare their training behavior to that of
traditional networks with normalized initialization (Glorot & Bengio, 2010; He
et al., 2015). We show that optimization of highway networks is virtually
independent of depth, while for traditional networks it suffers significantly
as the number of layers increases. We also show that architectures comparable
to those recently presented by Romero et al. (2014) can be directly trained to
obtain similar test set accuracy on the CIFAR-10 dataset without the need for
a pre-trained teacher network.

###  1.1 Notation

We use boldface letters for vectors and matrices, and italicized capital
letters to denote transformation functions. $\mathbf{0}$ and $\mathbf{1}$
denote vectors of zeros and ones respectively, and $\mathbf{I}$ denotes an
identity matrix. The function $\sigma(x)$ is defined as
$\sigma(x)=\frac{1}{1+e^{-x}},x\in\mathbb{R}$.

##  2 Highway Networks

A _plain_ feedforward neural network typically consists of $L$ layers where
the $l^{th}$ layer ($l\in\\{1,2,...,L\\}$) applies a non-linear transform $H$
(parameterized by $\mathbf{W_{H,l}}$) on its input $\mathbf{x_{l}}$ to produce
its output $\mathbf{y_{l}}$. Thus, $\mathbf{x_{1}}$ is the input to the
network and $\mathbf{y_{L}}$ is the network’s output. Omitting the layer index
and biases for clarity,

$\mathbf{y}=H(\mathbf{x},\mathbf{W_{H}}).$ (1)

$H$ is usually an affine transform followed by a non-linear activation
function, but in general it may take other forms.

For a highway network, we additionally define two non-linear transforms
$T(\mathbf{x},\mathbf{W_{T}})$ and $C(\mathbf{x},\mathbf{W_{C}})$ such that

$\mathbf{y}=H(\mathbf{x},\mathbf{W_{H}})\cdotp
T(\mathbf{x},\mathbf{W_{T}})+\mathbf{x}\cdot C(\mathbf{x},\mathbf{W_{C}}).$
(2)

We refer to $T$ as the _transform_ gate and $C$ as the _carry_ gate, since
they express how much of the output is produced by transforming the input and
carrying it, respectively. For simplicity, in this paper we set $C=1-T$,
giving

$\mathbf{y}=H(\mathbf{x},\mathbf{W_{H}})\cdotp
T(\mathbf{x},\mathbf{W_{T}})+\mathbf{x}\cdot(1-T(\mathbf{x},\mathbf{W_{T}})).$
(3)

The dimensionality of $\mathbf{x},\mathbf{y},H(\mathbf{x},\mathbf{W_{H}})$ and
$T(\mathbf{x},\mathbf{W_{T}})$ must be the same for Equation (missing) 3 to be
valid. Note that this re-parametrization of the layer transformation is much
more flexible than Equation (missing) 1. In particular, observe that

$\mathbf{y}=\begin{cases}\mathbf{x},&\text{if
}T(\mathbf{x},\mathbf{W_{T}})=\mathbf{0},\\\
H(\mathbf{x},\mathbf{W_{H}}),&\text{if
}T(\mathbf{x},\mathbf{W_{T}})=\mathbf{1}.\end{cases}$ (4)

Similarly, for the Jacobian of the layer transform,

$\frac{d\mathbf{y}}{d\mathbf{x}}=\begin{cases}\mathbf{I},&\text{if
}T(\mathbf{x},\mathbf{W_{T}})=\mathbf{0},\\\
H^{\prime}(\mathbf{x},\mathbf{W_{H}}),&\text{if
}T(\mathbf{x},\mathbf{W_{T}})=\mathbf{1}.\end{cases}$ (5)

Thus, depending on the output of the transform gates, a highway layer can
smoothly vary its behavior between that of a plain layer and that of a layer
which simply passes its inputs through. Just as a plain layer consists of
multiple computing units such that the $i^{th}$ unit computes
$y_{i}=H_{i}(\mathbf{x})$, a highway network consists of multiple blocks such
that the $i^{th}$ block computes a _block state_ $H_{i}(\mathbf{x})$ and
_transform gate output_ $T_{i}(\mathbf{x})$. Finally, it produces the _block
output_
$y_{i}=H_{i}(\mathbf{x})*T_{i}(\mathbf{x})+x_{i}*(1-T_{i}(\mathbf{x}))$, which
is connected to the next layer.

###  2.1 Constructing Highway Networks

As mentioned earlier, Equation (missing) 3 requires that the dimensionality of
$\mathbf{x},\mathbf{y},H(\mathbf{x},\mathbf{W_{H}})$ and
$T(\mathbf{x},\mathbf{W_{T}})$ be the same. In cases when it is desirable to
change the size of the representation, one can replace $\mathbf{x}$ with
$\mathbf{\hat{x}}$ obtained by suitably sub-sampling or zero-padding
$\mathbf{x}$. Another alternative is to use a plain layer (without highways)
to change dimensionality and then continue with stacking highway layers. This
is the alternative we use in this study.

Convolutional highway layers are constructed similar to fully connected
layers. Weight-sharing and local receptive fields are utilized for both $H$
and $T$ transforms. We use zero-padding to ensure that the block state and
transform gate feature maps are the same size as the input.

###  2.2 Training Deep Highway Networks

For plain deep networks, training with SGD stalls at the beginning unless a
specific weight initialization scheme is used such that the variance of the
signals during forward and backward propagation is preserved initially (Glorot
& Bengio, 2010; He et al., 2015). This initialization depends on the exact
functional form of $H$.

For highway layers, we use the transform gate defined as
$T(\mathbf{x})=\sigma(\mathbf{W_{T}}^{T}\mathbf{x}+\mathbf{b_{T}})$, where
$\mathbf{W_{T}}$ is the weight matrix and $\mathbf{b_{T}}$ the bias vector for
the transform gates. This suggests a simple initialization scheme which is
independent of the nature of $H$: $b_{T}$ can be initialized with a negative
value (e.g. -1, -3 etc.) such that the network is initially biased towards
_carry_ behavior. This scheme is strongly inspired by the proposal of Gers et
al. (1999) to initially bias the gates in a Long Short-Term Memory recurrent
network to help bridge long-term temporal dependencies early in learning. Note
that $\sigma(x)\in(0,1),\forall x\in\mathbb{R}$, so the conditions in Equation
(missing) 4 can never be exactly true.

In our experiments, we found that a negative bias initialization was
sufficient for learning to proceed in very deep networks for various zero-mean
initial distributions of $W_{H}$ and different activation functions used by
$H$. This is significant property since in general it may not be possible to
find effective initialization schemes for many choices of $H$.

Figure 1: Comparison of optimization of plain networks and highway networks of
various depths. All networks were optimized using SGD with momentum. The
curves shown are for the best hyperparameter settings obtained for each
configuration using a random search. Plain networks become much harder to
optimize with increasing depth, while highway networks with up to 100 layers
can still be optimized well.

##  3 Experiments

###  3.1 Optimization

Very deep plain networks become difficult to optimize even if using the
variance-preserving initialization scheme form (He et al., 2015). To show that
highway networks do not suffer from depth in the same way we train run a
series of experiments on the MNIST digit classification dataset. We measure
the cross entropy error on the training set, to investigate optimization,
without conflating them with generalization issues.

We train both plain networks and highway networks with the same architecture
and varying depth. The first layer is always a regular fully-connected layer
followed by 9, 19, 49, or 99 fully-connected plain or highway layers and a
single softmax output layer. The number of units in each layer is kept
constant and it is 50 for highways and 71 for plain networks. That way the
number of parameters is roughly the same for both. To make the comparison fair
we run a random search of 40 runs for both plain and highway networks to find
good settings for the hyperparameters. We optimized the initial learning rate,
momentum, learning rate decay rate, activation function for $H$ (either ReLU
or tanh) and, for highway networks, the value for the transform gate bias
(between -1 and -10). All other weights were initialized following the scheme
introduced by (He et al., 2015).

The convergence plots for the best performing networks for each depth can be
seen in Figure 1. While for 10 layers plain network show very good
performance, their performance significantly degrades as depth increases.
Highway networks on the other hand do not seem to suffer from an increase in
depth at all. The final result of the 100 layer highway network is about 1
order of magnitude better than the 10 layer one, and is on par with the 10
layer plain network. In fact, we started training a similar 900 layer highway
network on CIFAR-100 which is only at 80 epochs as of now, but so far has
shown no signs of optimization difficulties. It is also worth pointing out
that the highway networks always converge significantly faster than the plain
ones.

###  3.2 Comparison to Fitnets

Network | Number of Layers | Number of Parameters | Accuracy
---|---|---|---
Fitnet Results reported by Romero et al. (2014) |  |  |
Teacher | 5 |  $\sim$9M | 90.18%
Fitnet 1 | 11 |  $\sim$250K | 89.01%
Fitnet 2 | 11 |  $\sim$862K | 91.06%
Fitnet 3 | 13 |  $\sim$1.6M | 91.10%
Fitnet 4 | 19 |  $\sim$2.5M | 91.61%
Highway networks |  |  |
Highway 1 (Fitnet 1) | 11 |  $\sim$236K | 89.18%
Highway 2 (Fitnet 4) | 19 |  $\sim$2.3M | 92.24%
Highway 3* | 19 |  $\sim$1.4M | 90.68%
Highway 4* | 32 |  $\sim$1.25M | 90.34%
Table 1: CIFAR-10 test set accuracy of convolutional highway networks with
rectified linear activation and sigmoid gates. For comparison, results
reported by Romero et al. (2014) using maxout networks are also shown. Fitnets
were trained using a two step training procedure using soft targets from the
trained Teacher network, which was trained using backpropagation. We trained
all highway networks directly using backpropagation. * indicates networks
which were trained only on a set of 40K out of 50K examples in the training
set.

Deep highway networks are easy to optimize, but are they also beneficial for
supervised learning where we are interested in generalization performance on a
test set? To address this question, we compared highway networks to the thin
and deep architectures termed _Fitnets_ proposed recently by Romero et al.
(2014) on the CIFAR-10 dataset augmented with random translations. Results are
summarized in Table 1.

Romero et al. (2014) reported that training using plain backpropogation was
only possible for maxout networks with depth up to 5 layers when number of
parameters was limited to $\sim$250K and number of multiplications to
$\sim$30M. Training of deeper networks was only possible through the use of a
two-stage training procedure and addition of soft targets produced from a pre-
trained shallow teacher network (hint-based training). Similarly it was only
possible to train 19-layer networks with a budget of 2.5M parameters using
hint-based training.

We found that it was easy to train highway networks with number of parameters
and operations comparable to fitnets directly using backpropagation. As shown
in Table 1, Highway 1 and Highway 4, which are based on the architecture of
Fitnet 1 and Fitnet 4 respectively obtain similar or higher accuracy on the
test set. We were also able to train thinner and deeper networks: a 19-layer
highway network with $\sim$1.4M parameters and a 32-layer highway network with
$\sim$1.25M parameter both perform similar to the teacher network of Romero et
al. (2014).

##  4 Analysis

In Figure 2 we show some inspections on the inner workings of the
best111obtained via random search over hyperparameters to minimize the best
training set error achieved using each configuration 50 hidden layer fully-
connected highway networks trained on MNIST (top row) and CIFAR-100 (bottom
row). The first three columns show, for each transform gate, the bias, the
mean activity over 10K random samples, and the activity for a single random
sample respectively. The block outputs for the same single sample are
displayed in the last column.

The transform gate biases of the two networks were initialized to -2 and -4
respectively. It is interesting to note that contrary to our expectations most
biases actually decreased further during training. For the CIFAR-100 network
the biases increase with depth forming a gradient. Curiously this gradient is
inversely correlated with the average activity of the transform gates as seen
in the second column. This indicates that the strong negative biases at low
depths are not used to shut down the gates, but to make them more selective.
This behavior is also suggested by the fact that the transform gate activity
for a single example (column 3) is very sparse. This effect is more pronounced
for the CIFAR-100 network, but can also be observed to a lesser extent in the
MNIST network.

The last column of Figure 2 displays the block outputs and clearly visualizes
the concept of “information highways”. Most of the outputs stay constant over
many layers forming a pattern of stripes. Most of the change in outputs
happens in the early layers ($\approx 10$ for MNIST and $\approx 30$ for
CIFAR-100). We hypothesize that this difference is due to the higher
complexity of the CIFAR-100 dataset.

In summary it is clear that highway networks actually utilize the gating
mechanism to pass information almost unchanged through many layers. This
mechanism serves not just as a means for easier training, but is also heavily
used to route information in a trained network. We observe very selective
activity of the transform gates, varying strongly in reaction to the current
input patterns.

Figure 2: Visualization of certain internals of the blocks in the best 50
hidden layer highway networks trained on MNIST (top row) and CIFAR-100 (bottom
row). The first hidden layer is a plain layer which changes the dimensionality
of the representation to 50. Each of the 49 highway layers (y-axis) consists
of 50 blocks (x-axis). The first column shows the transform gate biases, which
were initialized to -2 and -4 respectively. In the second column the mean
output of the transform gate over 10,000 training examples is depicted. The
third and forth columns show the output of the transform gates and the block
outputs for a single random training sample.

##  5 Conclusion

Learning to route information through neural networks has helped to scale up
their application to challenging problems by improving credit assignment and
making training easier (Srivastava et al., 2015). Even so, training very deep
networks has remained difficult, especially without considerably increasing
total network size.

Highway networks are novel neural network architectures which enable the
training of extremely deep networks using simple SGD. While the traditional
plain neural architectures become increasingly difficult to train with
increasing network depth (even with variance-preserving initialization), our
experiments show that optimization of highway networks is not hampered even as
network depth increases to a hundred layers.

The ability to train extremely deep networks opens up the possibility of
studying the impact of depth on complex problems without restrictions. Various
activation functions which may be more suitable for particular problems but
for which robust initialization schemes are unavailable can be used in deep
highway networks. Future work will also attempt to improve the understanding
of learning in highway networks.

## Acknowledgments

This research was supported by the by EU project “NASCENCE” (FP7-ICT-317662).
We gratefully acknowledge the support of NVIDIA Corporation with the donation
of the Tesla K40 GPUs used for this research.

## References

  * Bengio et al. (2013) Bengio, Yoshua, Courville, Aaron, and Vincent, Pascal.  Representation learning: A review and new perspectives.  _Pattern Analysis and Machine Intelligence, IEEE Transactions on_ , 35(8):1798–1828, 2013.  URL http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6472238.
  * Ciresan et al. (2011a) Ciresan, Dan, Meier, Ueli, Masci, Jonathan, and Schmidhuber, Jürgen.  A committee of neural networks for traffic sign classification.  In _Neural Networks (IJCNN), The 2011 International Joint Conference on_ , pp. 1918–1921. IEEE, 2011a.  URL http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6033458.
  * Ciresan et al. (2012) Ciresan, Dan, Meier, Ueli, and Schmidhuber, Jürgen.  Multi-column deep neural networks for image classification.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012.
  * Ciresan et al. (2011b) Ciresan, DC, Meier, Ueli, Masci, Jonathan, Gambardella, Luca M, and Schmidhuber, Jürgen.  Flexible, high performance convolutional neural networks for image classification.  In _IJCAI_ , 2011b.  URL http://www.aaai.org/ocs/index.php/IJCAI/IJCAI11/paper/download/3098/3425%0020http://dl.acm.org/citation.cfm?id=2283603.
  * Gers et al. (1999) Gers, Felix A., Schmidhuber, Jürgen, and Cummins, Fred.  Learning to forget: Continual prediction with LSTM.  In _ICANN_ , volume 2, pp. 850–855, 1999.  URL http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=818041.
  * Glorot & Bengio (2010) Glorot, Xavier and Bengio, Yoshua.  Understanding the difficulty of training deep feedforward neural networks.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 249–256, 2010.  URL http://machinelearning.wustl.edu/mlpapers/paper_files/AISTATS2010_GlorotB10.pdf.
  * Håstad (1987) Håstad, Johan.  _Computational limitations of small-depth circuits_.  MIT press, 1987.  URL http://dl.acm.org/citation.cfm?id=SERIES9056.27031.
  * Håstad & Goldmann (1991) Håstad, Johan and Goldmann, Mikael.  On the power of small-depth threshold circuits.  _Computational Complexity_ , 1(2):113–129, 1991\.  URL http://link.springer.com/article/10.1007/BF01272517.
  * He et al. (2015) He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian.  Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification.  _arXiv:1502.01852 [cs]_ , February 2015.  URL http://arxiv.org/abs/1502.01852.
  * Hochreiter & Schmidhuber (1995) Hochreiter, Sepp and Schmidhuber, Jürgen.  Long short term memory.  Technical Report FKI-207-95, Technische Universität München, München, August 1995.  URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.51.3117.
  * Krizhevsky et al. (2012) Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , 2012.  URL http://books.nips.cc/papers/files/nips25/NIPS2012_0534.pdf.
  * Lee et al. (2015) Lee, Chen-Yu, Xie, Saining, Gallagher, Patrick, Zhang, Zhengyou, and Tu, Zhuowen.  Deeply-supervised nets.  pp. 562–570, 2015.  URL http://jmlr.org/proceedings/papers/v38/lee15a.html.
  * Montufar et al. (2014) Montufar, Guido F, Pascanu, Razvan, Cho, Kyunghyun, and Bengio, Yoshua.  On the number of linear regions of deep neural networks.  In _Advances in Neural Information Processing Systems_. 2014.  URL http://papers.nips.cc/paper/5422-on-the-number-of-linear-regions-of-deep-neural-networks.pdf.
  * Romero et al. (2014) Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua.  FitNets: Hints for thin deep nets.  _arXiv:1412.6550 [cs]_ , December 2014.  URL http://arxiv.org/abs/1412.6550.
  * Saxe et al. (2013) Saxe, Andrew M., McClelland, James L., and Ganguli, Surya.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  _arXiv:1312.6120 [cond-mat, q-bio, stat]_ , December 2013.  URL http://arxiv.org/abs/1312.6120.
  * Simonyan & Zisserman (2014) Simonyan, Karen and Zisserman, Andrew.  Very deep convolutional networks for large-scale image recognition.  _arXiv:1409.1556 [cs]_ , September 2014.  URL http://arxiv.org/abs/1409.1556.
  * Srivastava et al. (2015) Srivastava, Rupesh Kumar, Masci, Jonathan, Gomez, Faustino, and Schmidhuber, Jürgen.  Understanding locally competitive networks.  In _International Conference on Learning Representations_ , 2015.  URL http://arxiv.org/abs/1410.1165.
  * Szegedy et al. (2014) Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew.  Going deeper with convolutions.  _arXiv:1409.4842 [cs]_ , September 2014.  URL http://arxiv.org/abs/1409.4842.