11institutetext: IT University of Copenhagen, Denmark
11email<EMAIL_ADDRESS>22institutetext: Universidad Nacional de
San Martín, Argentina
22email<EMAIL_ADDRESS>33institutetext: CONICET -
Universidad Nacional del Litoral, Argentina
33email<EMAIL_ADDRESS>
# Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging

Dovile Juodelyte 11 Yucheng Lu 11 Amelia Jiménez-Sánchez 11 Sabrina Bottazzi
22 Enzo Ferrante 33 Veronika Cheplygina 11

###### Abstract

Transfer learning has become an essential part of medical imaging
classification algorithms, often leveraging ImageNet weights. However, the
domain shift from natural to medical images has prompted alternatives such as
RadImageNet, often demonstrating comparable classification performance.
However, it remains unclear whether the performance gains from transfer
learning stem from improved generalization or shortcut learning. To address
this, we investigate potential confounders – whether synthetic or sampled from
the data – across two publicly available chest X-ray and CT datasets. We show
that ImageNet and RadImageNet achieve comparable classification performance,
yet ImageNet is much more prone to overfitting to confounders. We recommend
that researchers using ImageNet-pretrained models reexamine their model
robustness by conducting similar experiments. Our code and experiments are
available at https://github.com/DovileDo/source-matters.

###### Keywords:

Transfer Learning Classification Domain Shift Shortcuts

##  1 Introduction

Machine learning models hold immense promise for revolutionizing healthcare.
However, their deployment in real-world clinical settings is hindered by
various challenges, with one of the most critical being their hidden reliance
on spurious features [24]. Recent research has highlighted the detrimental
effects of this reliance, including bias against demographic subgroups [2],
limited generalization across hospitals [25], and the risk of clinical errors
that may harm patients [17].

Despite transfer learning becoming a cornerstone in medical imaging, its
impact on model generalization remains largely unexplored. Pre-training on
ImageNet has become a standard practice due to its success in 2D image
classification. While some studies have explored alternative medical source
datasets for pre-training [3, 15, 23, 26, 14], ImageNet continues to serve as
a strong baseline.

Recent literature suggests that the size of the source dataset may matter more
than its domain or composition [19, 8, 6]. However, [13] demonstrated
performance improvements through source dataset pruning. In this context, we
argue that cross-domain transfer can be problematic, especially when source
dataset selection is solely based on classification performance, as it may
inadvertently lead to shortcut learning rather than genuine improvements in
generalization.

In this paper, we investigate how the domain of the source dataset affects
model generalization. First, we conceptualize confounding factors in medical
images and systematically assess model robustness by generating synthetic or
sampling real-world confounders commonly found in chest X-rays and CT scans.
Second, we compare models pre-trained on natural (ImageNet) and medical
(RadImageNet) datasets across X-ray and CT tasks and show substantial
differences in robustness to shortcut learning despite comparable predictive
performance. While transfer learning has been observed to enhance model
robustness [11, 12], our results suggest that it may not hold true when
transferring across domains, cautioning against using ImageNet pre-trained
models in medical contexts due to their susceptibility to shortcut learning.
Furthermore, our findings highlight the limitations of conventional
performance metrics based on i.i.d. datasets, which fail to discern between
genuine improvements in generalization and shortcut learning. Thus, we
advocate for a more nuanced evaluation of transfer learning effectiveness to
ensure the reliability and safety of machine learning applications in clinical
settings.

##  2 Method

Figure 1: MICCAT: Medical Imaging Contextualized Confounder Taxonomy.
Instances of confounders investigated in this paper are highlighted in bold.

###  2.1 MICCAT: towards a standardized taxonomy for medical imaging
confounders

To the best of our knowledge, there is no standardized taxonomy for
classifying potential confounders in medical images. Thus, to better structure
our robustness analysis, we propose a new taxonomy: Medical Imaging
Contextualized Confounder Taxonomy (MICCAT).

Previous work has shown that standard demographic attributes such as sex, age,
or ethnicity may act as confounders, leading to shortcut learning and
potentially disadvantaging historically underserved subgroups [2]. However,
solely focusing on standard protected demographic attributes may overlook
other specific factors related to clusters of patients for which the systems
tend to fail [7]. In MICCAT, we identify these as ‘contextualized
confounders’, as they are often domain or context-specific, associated with
particular image modalities, organs, hospitalization conditions, or diseases.

First, MICCAT differentiates between patient level and environment level
confounders. At the patient level, we make a distinction between standard
demographic attributes (e.g., sex, age, race) and contextualized anatomical
confounders, which arise from inherent anatomical properties of the organs and
human body or disease variations in images. This distinction is crucial as
standard demographic attributes often serve as proxies for underlying causes
of learned shortcuts. For instance, ethnicity may proxy skin color in
dermatoscopic images. Identifying the true shortcut cause allows for more
targeted interventions to mitigate biases. We define the concept of
environment level confounders, which stem from contextualized external or
imaging confounders. The former encompass physical or virtual elements in
images due to external factors like hospitalization devices or image tags,
while the latter include characteristics related to the imaging modality
itself, such as noise, motion blur, or differences in intensities due to
equipment or acquisition parameters. Figure 1 illustrates this taxonomy with
examples for each category. While we have found over 20 papers with various
shortcut examples, space constraints prevent listing them all. We recommend
[2, 20] for detailed discussions on specific examples and [18] for an
interesting analysis of common pitfalls that humans encounter when analyzing
medical images, which can lead to shortcut learning.

Confounders studied in this paper. We cover the confounder taxonomy by
investigating four examples of confounders:

  * •

An external confounder (_a tag_) placed in the upper left corner of the image,
representing confounding features introduced by various imaging devices across
or within hospitals (Fig. 5(a)).

  * •

Two typical imaging confounders: _denoising_ (Fig. 2(c)), widely used by
various vendors to reduce noise for enhanced readability [9], and _Poisson
noise_ (Fig. 2(d)), originating from quantum statistics of photons, which
cannot be mitigated through hardware engineering, unlike noise introduced by
circuit-related artifacts [22].

  * •

A patient-level confounder where we leverage _patient gender_ , which is
easily accessible in metadata, as a proxy for a broader spectrum of anatomical
confounders. We use the same label for this variable as in the original
dataset.

###  2.2 Experimental Design

We investigate the impact of source dataset domain on model generalization by
comparing ImageNet [5] and RadImageNet [15] models, which are fine-tuned using
binary prediction tasks for findings in open-access chest X-ray (NIH CXR14
[21]) and CT (LIDC-IDRI [1]) datasets curated to include systematically
controlled confounders.

(a)

(b)

(c)

(d)

Figure 2: Synthetic artifacts: (a) A tag with a red arrow for reference, (b) a
zoomed-in view of the original image, (c) Denoising by low-pass filter with
cutoff frequency (see Eq. 1) of $D_{0}=200$px, and (d) Poisson noise with
$N_{0}=2\times 10^{6}$ (see Eq. 2). The parameters used here are to emphasize
subtle local variations such as the smoothing effect of the low-pass filter
and the graininess introduced by the Poisson noise. For our experiments, we
use $D_{0}=500$px and $N_{0}=2\times 10^{7}$ which are imperceptible.

Confounder generation. A tag is placed further away from the edges (starting
at $200\times 200$px in the original image of $1024\times 1024$px), to ensure
it remains intact during training despite augmentations applied (Fig. 5(a)).

The simplest method for Denoising is applying low-pass filtering which entails
converting the input image from the spatial to the frequency domain using
Discrete Fourier Transform (DFT), followed by element-wise multiplication with
the low-pass filter $H_{LPF}(u,v)$ to generate the filtered image:

$H_{LPF}(u,v)=\left\\{\begin{array}[]{ll}1,&D(u,v)\leq D_{0}\\\
0,&\mbox{otherwise}\end{array}\right.$ (1)

where $D(u,v)$ represents the distance from the origin in the frequency
domain, and $D_{0}$ is the specified cutoff frequency. In our experiments, we
set $D_{0}=500$px. Subsequently, the high-frequency suppressed image is
reconstructed in the spatial domain via the Inverse Discrete Fourier Transform
(IDFT), resulting in a smoothing effect (see Fig. 2(c)).

Poisson noise originating from quantum statistics of photons is formulated as
a Poisson random process:

$\left(p_{r}+N_{p}\right)=\mathcal{P}\left(p_{r}\right)$ (2)

where $N_{p}$ represents Poisson noise, which notably affects image quality
under low-dose conditions (e.g., low-dose CT and X-ray screenings), while the
linear recording $p_{r}=\exp\left(-p_{a}\right)N_{0}$ is obtained via the
reversed conversion from attenuation $p_{a}$ given the prior information of
the source intensity $N_{0}$. To simulate low-dose screening, we add Poisson
noise to the image (Fig. 2(d)), adjusting the $N_{0}$ parameter to control
noise levels. We aim for minimal noise, setting $N_{0}=2\times 10^{7}$ after
visually examining the noise to ensure it remains barely imperceptible.

Patient gender is sampled to correlate ‘Female’ with the label.

Evaluation. To investigate shortcut learning systematically, we construct
development datasets for fine-tuning, focusing on a binary classification
task. We introduce previously mentioned confounders (e.g., ‘Female’) into the
positive class with a controlled probability
$p_{\text{art}}\in\\{0,0.1,0.2,0.5,0.8,1\\}$ to deliberately influence the
learning process, replicating scenarios where real-world data may contain
confounders. To assess the presence of shortcut learning, we evaluate the
fine-tuned models using a dedicated out-of-distribution (o.o.d.) test set. In
this test set, we introduce the same artifact used during fine-tuning to the
negative class with $p_{\text{art}}=1$, such that the models are tested on
instances where artifacts appear in the opposite class compared to what they
encountered during training. Fine-tuned model classification performance is
evaluated using the AUC (area under the receiver operating characteristic
curve).

Table 1: Target datasets used for fine-tuning. T: tag, D: denoising, N: noise.

# images in % split % class split Image Batch Task Confounder test/dev
train/val pos/neg size size Lung mass (NIH CXR14 [21]) T, D, N 83/248 90/10
30/70 512 $\times$ 512 32 Lung mass (LIDC-IDRI [1]) T, D, N 1710/500 80/20
50/50 362 $\times$ 362 32 Atelectasis (NIH CXR14 [21]) Gender 400/400 85/15
50/50 256 $\times$ 256 64

Medical targets. We create separate binary classification tasks for lung mass
detection using subsets of images sourced from two datasets: the chest X-ray
NIH CXR14 [21] subset annotated by clinicians [16], and the chest CT dataset
LIDC-IDRI [1] annotated by four radiologists. From the latter, we sample
paired positive and negative 2D slices from the original 3D scans using nodule
ROI annotations, representing any kind of lesions and their nearby slices
without remarkable findings. We include synthetic artifacts (a tag, denoising,
and Poisson noise) in both tasks. For the case where patient gender serves as
the confounding feature, we sample posterior to anterior (PA) images from the
NIH CXR14 dataset [21] to construct a binary classification task for
atelectasis. We deliberately limit the size of our development datasets,
encompassing both balanced and unbalanced class distributions to cover a
spectrum of clinical scenarios. Data splits for training, validation, and
testing preserve class distribution and are stratified by patient. Further
details are available in Table 1.

Fine-tuning details. We use ResNet50 [10] as the backbone with average pooling
and a dropout layer (0.5 probability). The models are trained using cross-
entropy loss with Adam optimizer (learning rate: $1\times 10^{-5}$) for a
maximum of 200 epochs with early stopping after 30 epochs of no improvement in
validation loss (AUC for the balanced tasks). This configuration, established
during early tuning, proved flexible enough to accommodate different
initializations and target datasets. During training, we apply image
augmentations including random rotation (up to 10 degrees), width and height
shifts, shear, and zoom, all set to 0.1, with a fill mode set to ‘nearest’.
Models were implemented using Keras [4] library and fine-tuned on an NVIDIA
Tesla A100 GPU card.

##  3 Results and Discussion

Figure 3: Mean AUC across five-fold cross-validation with 95% confidence
interval for lung mass (left and middle) and atelectasis (right) prediction in
chest X-rays. Increasing correlation between artifact (a tag (left), denoising
(middle), or gender (right)) and the label leads to decreased o.o.d.
performance (on o.o.d. test set as described in Sec. 2.2) (top row), while
i.i.d. performance increases (bottom row). RadImageNet’s pre-trained models
exhibit lesser degradation in o.o.d. performance compared to ImageNet’s pre-
trained models, suggesting that ImageNet may over-rely on spurious
correlations in the target dataset.

RadImageNet is robust to shortcut learning. Figure 3 shows that both ImageNet
and RadImageNet pre-trained models exhibit comparable performance on
independent and identically distributed (i.i.d.) test sets, however, when
subjected to o.o.d. test sets, notable differences emerge. Specifically,
ImageNet’s o.o.d. performance on X-rays, confounded by tag, denoising, and
patient gender, drops more compared to RadImageNet, indicating ImageNet’s
higher reliance on spurious correlations. This could be because certain
features, for instance, a tag (letters), may serve as a discriminative feature
in ImageNet, e.g., for the computer keyboard class. However, RadImageNet is
invariant to such features as they are not consistently associated with
specific labels across different classes, and this invariance transfers to the
target task. Similar trends were observed in the CT dataset, with the o.o.d.
AUC decreasing from 0.84 to 0.02 for ImageNet, and to 0.22 for RadImageNet
(for tag); and from 0.7 to 0.01 for ImageNet, and from 0.83 only to 0.6 for
RadImageNet (for denoising).

Figure 4: Mean o.o.d. AUC across five-fold cross-validation with a 95%
confidence interval for lung mass prediction in chest X-rays and CT scans. In
X-rays (top row), both ImageNet and RadImageNet pre-trained models show
similar reliance on Poisson noise. However, RadImageNet is more robust in CT
scans (bottom row). When the confounding feature is high vs low noise level,
both ImageNet and RadImageNet are less sensitive (right column), compared to
noise vs no noise (left column).

Although tag and denoising are designed to replicate real-world artifacts,
they lack the diversity found in real-world scenarios. Patient gender presents
a more realistic confounder. Here, the performance gap between ImageNet and
RadImageNet is smaller (by 0.12 on average for $p_{\text{art}}\geq 0.1$) yet
remains statistically significant (permutation test,
$0.008<p\text{-value}<0.032$, for $p_{\text{art}}\geq 0.1$, detailed in Table
2 in the supplementary material). This suggests that RadImageNet’s resilience
to shortcuts extends to more realistic variations in confounders, further
emphasizing its robustness in medical image classification.

Random initialization appears to be robust to shortcut learning, with
consistent o.o.d. performance as $p_{\text{art}}$ increases. However, this is
mainly due to the unbalanced class distribution in the lung mass prediction
task within the NIH CXR14 dataset, where randomly initialized models tend to
predict the overrepresented negative class. Conversely, in the case of a
balanced class distribution in the CT target dataset, the o.o.d. performance
of randomly initialized models deteriorates to a similar degree as that of
ImageNet-initialized models.

Shortcuts come in all shapes and sizes. ImageNet and RadImageNet both heavily
rely on Poisson noise in X-rays (Fig. 4, upper left) but RadImageNet shows
greater robustness to noise in CT scans compared to ImageNet (Fig. 4, lower
left). It is important to note that Poisson noise manifests differently in
X-rays and CT scans. In X-rays, Poisson noise introduces graininess
characterized by random and pixel-wise independent variations, while in CT
scans, it appears as streak artifacts structurally correlated to projections
and thus not random in the image domain (see Fig. 5 in the supplementary
material).

To understand the impact of this difference, we directly introduce Poisson
noise $N_{0}=2\times 10^{7}$ in the image domain for CT scans, mimicking the
pixel-wise independence seen in X-rays. However, since CT scans inherently
contain noise, this introduces a confounding feature of high versus low levels
of noise, as opposed to the original confounder of noise versus no noise.

To simulate a corresponding scenario in X-rays, we generate two levels of
Poisson noise: $N_{0}=2\times 10^{7}$ for the positive images and
$N_{0}=1\times 10^{7}$ for the negative images (reversed for the o.o.d. test
set). Both models exhibit a smaller drop in o.o.d. performance across
modalities, indicating a reduced reliance on the noise shortcut (Fig. 4, right
column). This suggests that discerning between high and low noise levels
presents a more challenging task than simply detecting the presence of noise.

RadImageNet maintains its robustness in CT scans, while in X-rays, RadImageNet
relies on noise to a similar extent as ImageNet. This may be explained by the
absence of X-ray images in RadImageNet’s pre-training data, leading to a lack
of robust X-ray representations that would resist pixel-wise independent noise
– a phenomenon less common in CT, MR, and ultrasound, modalities encompassed
in RadImageNet. This highlights that even transferring from a medical source
of a different modality may lead to overfitting on confounders.

In our exploration of the confounder taxonomy, we found that RadImageNet
models are generally more robust to shortcut learning. However, there is some
variability within the category of imaging confounders, and the importance of
the source dataset domain in anatomical confounders seems to be lower.
Expanding the scope to include more examples of confounders would offer a more
comprehensive understanding of the taxonomy landscape and provide insights
into the nuances within each category, facilitating better-informed source
dataset selection and evaluation strategies.

##  4 Conclusion

Our study sheds light on the critical role that the domain of the source
dataset plays in model generalization in medical imaging tasks. By
systematically investigating confounders typically found in X-rays and CT
scans, we have uncovered substantial differences in robustness to shortcut
learning between models pre-trained on natural and medical image datasets. Our
findings caution against the blind application of transfer learning across
domains. We advocate for a more nuanced evaluation to improve the reliability
and safety of machine learning applications in clinical settings.

####  4.0.1 Data use declaration.

This study uses publicly available pre-trained ImageNet [5] (from Keras [4])
and RadImageNet [15] weights as well as publicly available medical imaging
datasets NIH CXR14 [21] (available at https://nihcc.app.box.com/v/ChestXray-
NIHCC), and LIDC-IDRI [1].

####  4.0.2 Acknowledgements

This study was funded by the Novo Nordisk Foundation (grant number
NNF21OC0068816). The authors acknowledge the National Cancer Institute and the
Foundation for the National Institutes of Health as well as NIH Clinical
Center, and their critical role in the creation of the free publicly available
LIDC/IDRI Database and NIH CXR14 dataset used in this study.

## References

  * [1] Armato III, S.G., McLennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D.R., Henschke, C.I., Hoffman, E.A., Kazerooni, E.A., MacMahon, H., Van Beek, E.J.R., Yankelevitz, D., Biancardi, A.M., Bland, P.H., Brown, M.S., Engelmann, R.M., Laderach, G.E., Max, D., Pais, R.C., Qing, D.P.Y., Roberts, R.Y., Smith, A.R., Starkey, A., Batra, P., Caligiuri, P., Farooqi, A., Gladish, G.W., Jude, C.M., Munden, R.F., Petkovska, I., Quint, L.E., Schwartz, L.H., Sundaram, B., Dodd, L.E., Fenimore, C., Gur, D., Petrick, N., Freymann, J., Kirby, J., Hughes, B., Casteele, A.V., Gupte, S., Sallam, M., Heath, M.D., Kuhn, M.H., Dharaiya, E., Burns, R., Fryd, D.S., Salganicoff, M., Anand, V., Shreter, U., Vastagh, S., Croft, B.Y., Clarke, L.P.: Data From LIDC-IDRI [Data set]. The Cancer Imaging Archive (2015)
  * [2] Banerjee, I., Bhattacharjee, K., Burns, J.L., Trivedi, H., Purkayastha, S., Seyyed-Kalantari, L., Patel, B.N., Shiradkar, R., Gichoya, J.: “Shortcuts” Causing Bias in Radiology Artificial Intelligence: Causes, Evaluation, and Mitigation. Journal of the American College of Radiology 20(9), 842–851 (Sep 2023)
  * [3] Cheplygina, V.: Cats or cat scans: Transfer learning from natural or medical image source data sets? Current Opinion in Biomedical Engineering 9, 21–27 (2019)
  * [4] Chollet, F., et al.: Keras (2015)
  * [5] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. pp. 248–255. IEEE (2009)
  * [6] Entezari, R., Wortsman, M., Saukh, O., Shariatnia, M.M., Sedghi, H., Schmidt, L.: The Role of Pre-training Data in Transfer Learning (Mar 2023), arXiv:2302.13602 [cs]
  * [7] von Euler-Chelpin, M., Lillholm, M., Vejborg, I., Nielsen, M., Lynge, E.: Sensitivity of screening mammography by density and texture: a cohort study from a population-based screening program in Denmark. Breast cancer research: BCR 21(1), 111 (Oct 2019)
  * [8] Gavrikov, P., Keuper, J.: Does Medical Imaging learn different Convolution Filters? (Oct 2022), arXiv:2210.13799 [cs, eess]
  * [9] Hasegawa, A., Ishihara, T., Thomas, M.A., Pan, T.: Noise reduction profile: A new method for evaluation of noise reduction techniques in ct. Medical physics 49(1), 186–200 (2022)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778. IEEE, Las Vegas, NV, USA (Jun 2016)
  * [11] Hendrycks, D., Lee, K., Mazeika, M.: Using pre-training can improve model robustness and uncertainty. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 2712–2721. PMLR (09–15 Jun 2019)
  * [12] Islam, M., Li, Z., Glocker, B.: Robustness Stress Testing in Medical Image Classification. In: Sudre, C.H., Baumgartner, C.F., Dalca, A., Mehta, R., Qin, C., Wells, W.M. (eds.) Uncertainty for Safe Utilization of Machine Learning in Medical Imaging. pp. 167–176. Lecture Notes in Computer Science, Springer Nature Switzerland, Cham (2023)
  * [13] Jain, S., Salman, H., Khaddaj, A., Wong, E., Park, S.M., Mądry, A.: A data-based perspective on transfer learning. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3613–3622. IEEE, Vancouver, BC, Canada (Jun 2023)
  * [14] Juodelyte, D., Jiménez-Sánchez, A., Cheplygina, V.: Revisiting hidden representations in transfer learning for medical imaging. Transactions on Machine Learning Research (2023)
  * [15] Mei, X., Liu, Z., Robson, P.M., Marinelli, B., Huang, M., Doshi, A., Jacobi, A., Cao, C., Link, K.E., Yang, T., Wang, Y., Greenspan, H., Deyer, T., Fayad, Z.A., Yang, Y.: RadImageNet: An Open Radiologic Deep Learning Research Dataset for Effective Transfer Learning. Radiology: Artificial Intelligence 4(5), e210315 (Sep 2022), publisher: Radiological Society of North America
  * [16] Nabulsi, Z., Sellergren, A., Jamshy, S., Lau, C., Santos, E., Kiraly, A.P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., Yu, J., Kalidindi, S.R., Etemadi, M., Garcia-Vicente, F., Melnick, D., Corrado, G.S., Peng, L., Eswaran, K., Tse, D., Beladia, N., Liu, Y., Chen, P.H.C., Shetty, S.: Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19. Scientific Reports 11(1), 15523 (Sep 2021)
  * [17] Oakden-Rayner, L., Dunnmon, J., Carneiro, G., Ré, C.: Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In: ACM Conference on Health, Inference, and Learning. pp. 151–159 (2020)
  * [18] Peh, W.C.: Pitfalls in diagnostic radiology. Springer (2014)
  * [19] Ramanujan, V., Nguyen, T., Oh, S., Farhadi, A., Schmidt, L.: On the connection between pre-training data diversity and fine-tuning robustness. Advances in Neural Information Processing Systems 36 (2023)
  * [20] Sun, S., Koch, L.M., Baumgartner, C.F.: Right for the wrong reason: Can interpretable ml techniques detect spurious correlations? In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 425–434. Springer (2023)
  * [21] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3462–3471. IEEE Computer Society, Los Alamitos, CA, USA (jul 2017)
  * [22] Wei, K., Fu, Y., Zheng, Y., Yang, J.: Physics-based noise modeling for extreme low-light photography. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(11), 8520–8537 (2021)
  * [23] Wen, Y., Chen, L., Deng, Y., Zhou, C.: Rethinking pre-training on medical imaging. Journal of Visual Communication and Image Representation 78, 103145 (Jul 2021)
  * [24] Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V.X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., Ossorio, P.N., Thadaney-Israni, S., Goldenberg, A.: Do no harm: a roadmap for responsible machine learning for health care. Nature Medicine 25(9), 1337–1340 (Sep 2019)
  * [25] Zech, J.R., Badgeley, M.A., Liu, M., Costa, A.B., Titano, J.J., Oermann, E.K.: Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study. PLOS Medicine 15(11), e1002683 (Nov 2018), publisher: Public Library of Science
  * [26] Zhou, Z., Sodha, V., Pang, J., Gotway, M.B., Liang, J.: Models Genesis. Medical Image Analysis 67, 101840 (Jan 2021)

## Supplementary material

Table 2: Permutation t-test for the hypothesis that the o.o.d. AUC of ImageNet is lower than that of RadImageNet for atelectasis prediction in chest X-rays confounded by patient gender. $p_{\text{art}}$ | $t\text{-statistic}$ | $p\text{-value}$
---|---|---
0 | -0.5 | 0.817
0.1 | -4.1 | 0.008
0.2 | -2.8 | 0.032
0.5 | -4.2 | 0.008
0.8 | -4.8 | 0.008
1 | -5.9 | 0.008

(a) Original CT scan

(b) Poisson noise

Figure 5: Here $N_{0}=4\times 10^{3}$ to emphasize streak artifacts introduced
by the Poisson noise. For our experiments, we use $N_{0}=4\times 10^{35}$
which is imperceptible.


# Binary Early-Exit Network for Adaptive Inference on Low-Resource Devices

###### Abstract

Deep neural networks have significantly improved performance on a range of
tasks with the increasing demand for computational resources, leaving
deployment on low-resource devices (with limited memory and battery power)
infeasible. Binary neural networks (BNNs) tackle the issue to an extent with
extreme compression and speed-up gains compared to real-valued models. We
propose a simple but effective method to accelerate inference through unifying
BNNs with an early-exiting strategy. Our approach allows simple instances to
exit early based on a decision threshold and utilizes output layers added to
different intermediate layers to avoid executing the entire binary model. We
extensively evaluate our method on three audio classification tasks and across
four BNNs architectures. Our method demonstrates favorable quality-efficiency
trade-offs while being controllable with an entropy-based threshold specified
by the system user. It also results in better speed-ups (latency less than
6ms) with a single model based on existing BNN architectures without
retraining for different efficiency levels. It also provides a straightforward
way to estimate sample difficulty and better understanding of uncertainty
around certain classes within the dataset.

Index Terms: early-exit, audio recognition, binary neural networks

##  1 Introduction

The rapidly increasing size of deep neural networks, along with advancements
in design and training, have made them perform well on a broad spectrum of
tasks. The large-scale models improve predictive performance but significantly
escalate production costs with slower inference, in particular, severely
limiting the adoption of deep models on resource-constrained edge devices like
wearables with limited battery and computational power. Binary neural networks
(BNNs) have become promising methods for obtaining highly compact and
efficient models in deployment on resource-constrained devices due to extreme
compression and speed-up gains compared to their real-valued counterparts [1,
2]. A complementary technique for accelerating deep models is a dynamic input-
dependent prediction generation, which has recently gained notable traction
and has become widely known as early-exit architecture [3, 4, 5, 6, 7, 8, 9,
10].

The key idea behind early-exit models is that the difficulty of classifying an
example, and thus the required model capacity to do so, varies greatly in
practice. For instance, a speech classifier is likely to be provided with
utterances over a wide range of signal-to-noise ratios. The shallow features
from initial layers are mostly sufficient to confidently distinguish easy
examples of different classes to a certain extent [9, 7, 6]. These models are
capable of trading-off computation and accuracy dynamically on a per-instance
basis to efficiently utilize available resources. An early-exit architecture
adapts its computation to the difficulty of an example by placing auxiliary
decision layers (the “exits”) at different depths of its main architecture. At
inference, an example passes through each exit in a sequential fashion, and a
decision rule (or exit threshold) is typically used to decide whether to use
the current exit’s prediction or to continue.

We design an early-exit model to reduce inference cost and further speed-up
execution of BNNs based on the aforementioned observation. We focus solely on
BNNs due to their greater efficiency and minimal model size that is largely
suitable for devices with limited memory [11]. We address the problem of
early-exiting from the perspective of varying model sizes and sample (or
instance) difficulty. Our approach provides a simple yet effective way to
assess the computational difficulty of making a prediction for a given input
and pair it with an optimal exit at a specific depth within the model to
reduce uncertainty. As a result, not all instances have to pass through the
entire model. The ‘easy’ to classify examples are matched with earlier exits.
In comparison, the ‘harder’ examples are tackled by deeper exits to maintain
good performance and thus save computation on a per-example basis.

Figure 1: Overview of our early-exit binary neural networks. Grey blocks
denote a set of binarized convolutional and other layers, orange and green
squares denote decision exits an classification output layers, respectively.
Our early-exit model enables adaptive inference for each input sample based on
an entropy-based decision threshold to early-exit the input, thus saving
computation.

We apply our method to a broad range of high-performing and efficient binary
networks architectures, which we repurpose for different audio recognition
tasks. To the best of our knowledge, our work is the first attempt at
proposing and studying binary early-exit models for audio recognition.
Specifically, we add classification output layers (or also called exits) at
varying depths, while maintaining the rest of the architecture as is. We
notice that an exit at the $j$th layer is efficient though (seemingly) less
effective as compared to later exit $k$, (where $k>j$). Surprisingly, in some
cases we achieve similar or even better performance than standard single-exit
models, hinting at ‘over-thinking’ phenomena in neural networks in which the
predictions based on earlier layers are correct but become incorrect with
progressively deeper layers, resulting in wasteful computation [12]. We
provide high-level illustration of the approach in Figure 1.

Our approach of unifying binary networks with early exits accelerates
inference substantially and results in saving in terms of computation compared
to single-exit models while adding very few learnable parameters to the model.
Likewise, it does not result in a penalty in terms of training convergence.
Our simple strategy is flexible in practice compared to other techniques, for
instance, methods that require re-training to achieve an optimal trade-off
between speed-up and accuracy. On the contrary, we can train the early-exit
BNNs end-to-end and require only the selection of the appropriate value of
decision threshold to decide on exiting or further processing of the input.
Likewise, the proposed technique is orthogonal to pruning and other strategies
for improving neural network efficiency. It can be easily combined to further
enhance the computational run-time or energy utilization of the BNNs.

##  2 Method

We design an early-exit classification architecture with $E$ exits that
produces a list of outputs ($p_{e_{1}}$, $p_{e_{2}}$, …$p_{e_{E}}$) where each
exit $p_{e_{i}}$ is a probability distribution over classes. Specifically, for
the exits $1<i<=E$, an exit $e_{i}$ is potentially more accurate and expensive
to compute than the previous exit i.e., $e_{i-1}$. Nevertheless, a slightly
less accurate exit can also be used for inference based on a pre-defined
threshold of the output to trade-off between accuracy and computation. At
inference time, the early-exit model can quantify the exit’s confidence on a
per-example basis with an entropy of output probability distribution or
temperature-scaled softmax score $\mu$. When an instance arrives at an exit,
the chosen metric is computed and compared with a pre-defined threshold $S$ to
decide to exit earlier or pass on the sample to the next layer. Intuitively,
the larger values of the threshold $S$ result in a faster inference but less
accurate prediction and vice versa as the earlier exits can be inaccurate to
solve the specified learning task. This strategy is also termed as “early-
exiting,” i.e., for a sample, an output can be produced at any time depending
on the computational budget or accuracy.

We consider binary convolutional networks [1] as backbone models to augment
with early-exit capability for inference on low-resource devices with limited
computational power. We represent the weights and input features with
$\mathcal{W}\in\mathbb{R}^{k\times k\times c\times o}$ and
$\mathcal{X}\in\mathbb{R}^{w\times h\times c}$, with $c$ and $o$ being number
of input and output channels, $k$ denoting kernel height and width, while $w$
and $h$ represent spatial dimension of the input. Generally, both the weights
and activations of the model are binarized, where convolution operation is
performed as $\mathcal{X}*\mathcal{W}$ using bit-wise operations. Previously,
various strategies and network architectures have been proposed to tackle
degradation in performance due to extreme quantization as compared to their
real-valued counterparts. Here, we focus on state-of-the-art binary networks
to demonstrate our approach of early-exiting that include QuickNet [13],
BiRealNet [14], BinaryDenseNet [15], and Meliusnet [16]. The key details of
these architectures are summarized in Table 1. We utilize Larq Compute Engine
for latency benchmarking of binary models. We compute latencies by converting
the models to TFLite format using a device equipped with Snapdragon 855.

Our approach leverages the multilayered structure of models to add output
layers after specific intermediate layers. In all the considered
architectures, we place five exits in total (including the last) such that
early-exit classifiers have adequate capacity. The exits are comprised of a
global pooling layer followed by a binary dense layer with hidden units equal
to the number of outputs. For model training, the loss function of the
$j^{th}$ exit is:
$\mathcal{L}_{i}(\mathcal{D};\theta)=\frac{1}{|\mathcal{D}|}\sum_{(x,y)\in\mathcal{D}}\mathcal{H}(y,f_{j}(x;\theta))$,
where, $\mathcal{D}$ is the training set comprising of $(x,y)$ input-label
pairs, $\theta$ are the set of learnable model parameters, and $\mathcal{H}$
is the standard cross-entropy loss function. We aggregate the losses from
respective output layers to enable earlier layers to act as both useful
feature extractors for deeper layers and as good input to their corresponding
classifiers. Further, this training regime indicates that every exit layer is
trained to perform well on all instances in $\mathcal{D}$. We do not perform
early exiting based on decision-threshold (i.e., entropy) during training but
employs it for inference. The way our approach works at inference time is
shown in Algorithm 1.

Input: Input $x$, Entropy threshold $\delta$

Output: $y$

for  _$i\leftarrow 1$ to $E$_ do

$\hat{y}_{i}$ = $f_{i}(x;\theta)$

$\epsilon$ = entropy($\hat{y}_{i}$)

if  _$\epsilon$ $<$ $\delta$_ then

return $\hat{y}_{i}$

end for

return $\hat{y}_{E}$

Algorithm 1 Early Exit Inference Table 1: Architectural specification of
binary neural networks. We compute latency on a device equipped with
Snapdragon 855 with a TFLite model including audio front-end.

Model | #Parameters | Size (MiB) | Latency (ms) |
---|---|---|---|---
Binary | Float-32 | $\mathcal{E}_{1}$ | $\mathcal{E}_{2}$ | $\mathcal{E}_{3}$ | $\mathcal{E}_{4}$ | $\mathcal{E}_{5}$
QuickNet [13] | 12.7 M | 2.23 | 48.54 | 2.62 | 2.73 | 2.97 | 3.24 | 3.51
BiRealNet [14] | 11.2 M | 2.04 | 42.63 | 3.24 | 3.31 | 3.56 | 3.91 | 4.17
BinaryDenseNet [15] | 4.56 M | 1.85 | 42.63 | 6.71 | 6.14 | 6.19 | 6.24 | 6.31
MeliusNet [16] | 6.44 M | 1.94 | 24.56 | 3.24 | 6.58 | 7.08 | 7.37 | 7.59

##  3 Experiments

Model | Exit 1 | Exit 2 | Exit 3 | Exit 4 | Exit 5
---|---|---|---|---|---
$\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$
QuickNet | 56.8 | 61.2 | 73.4 | 76.1 | 74.8 | 78.7 | 80.7 | 81.0 | 78.3 | 80.7
BiRealNet | 59.1 | 58.0 | 74.6 | 72.9 | 77.2 | 77.7 | 79.7 | 80.4 | 80.2 | 80.3
BinaryDenseNet | 78.9 | 73.1 | 83.8 | 79.1 | 83.8 | 80.5 | 83.8 | 81.1 | 81.7 | 80.5
MeliusNet | 18.0 | 41.2 | 84.3 | 81.7 | 87.9 | 86.3 | 87.5 | 87.4 | 85.9 | 86.3
(a) MSWC (Micro-EN)

Model | Exit 1 | Exit 2 | Exit 3 | Exit 4 | Exit 5
---|---|---|---|---|---
$\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$
QuickNet | 65.7 | 86.3 | 92.8 | 93.3 | 95.2 | 94.7 | 94.6 | 95.0 | 94.4 | 95.1
BiRealNet | 90.7 | 89.5 | 91.9 | 93.1 | 94.7 | 94.2 | 92.9 | 94.0 | 91.7 | 93.9
BinaryDenseNet | 65.6 | 93.2 | 93.1 | 95.2 | 94.4 | 95.9 | 91.1 | 96.0 | 93.6 | 95.9
MeliusNet | 81.9 | 81.5 | 94.7 | 93.8 | 95.6 | 93.4 | 95.2 | 94.9 | 94.9 | 94.9
(b) SpeechCommands

Model | Exit 1 | Exit 2 | Exit 3 | Exit 4 | Exit 5
---|---|---|---|---|---
$\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$ | $\mathcal{E}_{s}$ | $\mathcal{E}_{m}$
QuickNet | 75.7 | 75.8 | 88.8 | 83.0 | 86.2 | 84.3 | 81.3 | 84.9 | 70.9 | 79.0
BiRealNet | 72.3 | 70.8 | 82.9 | 73.9 | 87.3 | 80.1 | 81.6 | 76.6 | 76.2 | 78.1
BinaryDenseNet | 87.6 | 83.5 | 86.3 | 88.6 | 87.8 | 89.9 | 85.2 | 87.3 | 82.2 | 85.0
MeliusNet | 70.5 | 74.0 | 89.4 | 87.3 | 91.0 | 90.8 | 77.6 | 88.6 | 84.5 | 88.2
(c) Voxforge

Table 2: Performance (accuracy) analysis of single-exit models
$\mathcal{E}_{s}$ with the corresponding exit in an early-exit model
$\mathcal{E}_{m}$, where all exits are trained in an end-to-end manner. We
note that the performance of multi-exit classifiers is inline with when models
up to certain intermediate layer are trained independently. Figure 2:
Evaluation of early-exit binary neural networks across a range of tasks and
architectures for varying entropy thresholds. The early-exit models provide an
adaptive exiting strategy for each example and maintain a good balance between
exiting early and increasing uncertainty. Our approach also achieves
performance close to state-of-the-art floating-point models on Voxforge and
SpeechCommands datasets close to $90\%$ and $95\%$ in comparison to $95.4\%$
[17] and $97.9\%$ [18], respectively.

###  3.1 Datasets, Tasks and Implementation Details

We conduct experiments on various audio recognition tasks ranging from spoken
language identification, keyword spotting, to spoken commands recognition. We
choose Speech Commands [19] and Voxforge [20], and MSWC (Micro-EN) [21]
datasets sampled at $16$kHz with each comprising $12$, $6$, and $31$ classes,
respectively. In all cases, we utilize the standard train-test splits provided
with the corresponding datasets. Given an audio input sequence, we randomly
select a one-second of audio segment from an entire audio clip to extract log-
compressed Mel-filterbanks with a window size of $25$ ms, a hop size of $10$
ms, and $N=64$ Mel-spaced frequency bins in the range $60$-$7800$ Hz for
$T=98$ frames, corresponding to $980$ ms. We use these features as input to
binary neural networks, i.e., QuickNet, BiRealNet, BinaryDenseNet-28, and
MeliusNet. Even though these architectures are originally proposed for vision
tasks, the $2$D input structure of Mel-filterbanks enables us to use them
without further adjustment. We use the Larq [11] framework for implementation
of BNNs and utilize Binary Optimizer (Bop) [22] and Adam [23] for training
models with a learning rate of $0.001$ and batch size of $128$ for $100$
epochs. During the evaluation, we use each audio sample without any splitting,
effectively using a batch size of one to handle input of varying sizes.

(a)

(b)

(c)

Figure 3: The number of output samples by exit layer for different entropy
thresholds for Meliusnet model. With increasing entropy threshold, a large
fraction of samples utilize early exits, where fifth block is the last exit in
the model.

(a)

(b)

(c)

Figure 4: Average exit and accuracy trade-offs for different binary models.
The later exits classify examples with higher accuracy but at an increased
computational cost.

###  3.2 Results

We start with analyzing the generalization of each exit for considered neural
architectures and datasets in Table 2. The exits shown with $\mathcal{E}_{s}$
are single exit models where a single output layer after some intermediate
layer is added while $\mathcal{E}_{m}$ represents the same exit but trained
jointly in an end-to-end fashion. We notice that the performance of individual
exits within early-exit models is mostly consistent with the ones trained
independently. Further, the single exit model ($\mathcal{E}_{s}$) can also be
seen as efficient baselines, which offer a trade-off between accuracy and
speed, but unlike with one model like our approach.

In Figure 2, we present our key results achieved with different entropy
thresholds for each task and architecture. The curves report accuracy achieved
while setting a fixed value of entropy $\delta$ with an early-exit model. We
use entropy thresholds of $0.1,0.25,0.5,0.75$, and $1.0$. We observe that even
with high values of $\delta$, the performance stays considerably stable. In
the case of SpeechCommands, for all early-exit BNNs, the accuracy is around
$91\%$ when $\delta=0.5$. Interestingly, on Voxforge, we see a significant
difference in performance across architectures. Specifically, the BiRealNet
accuracy is lower than the rest and degrades more rapidly with increasing
uncertainty. However, MeliusNet performs best with $\delta=0.5$, achieving
around $90\%$ accuracy. On the other hand, we notice that architectures
achieve optimal performance with $0.6<\delta<0.8$. Both the early and late
exits results in lower performance, these results echos the finding of [12],
that “overthinking” can lead to incorrect predictions.

Figure 5: Comparison of Voxforge class labels (x-axis) per exit. Each plot
shows results with a specific early-exit model for a fixed entropy threshold,
i.e., $\delta=0.5$.

We show the fraction of instances exiting from each exit of Meliusnet for a
fixed entropy threshold in Figure 3. The lower values of $\delta$ force the
samples to increasingly utilize later exits. With increasing $\delta$, more
examples exit early, thus saving computation. We notice example difficulty
varies across tasks. For instance, in the case of SpeechCommands, the first
exits generally dominate the inference to classify the majority of examples.
However, on MSWC and Voxforge, all exits are mostly utilized. In particular,
we observe an interesting pattern for Voxforge that instances mostly either
exit early or prefer the last exit with few using the fourth exit. These
results highlight that entropy-based exiting optimally selects suitable exit
while maintaining a good trade-off between predictive quality and efficiency.
In Figure 4, we show the accuracy of binary models with respect to average
exit for various entropy thresholds. The dotted gray line represents the
performance of the standard single exit model, where all instances exit from
the last layer. We observe consistent results with later exits improving
performance at the cost of utilizing more compute, with BinaryDenseNet and
MeliusNet achieving higher accuracy as compared to the rest. Furthermore,
Figure 5 presents the proportion of instances in the Voxforge test set for
each semantic class and models for entropy threshold of $0.5$. We notice that
across models, not all classes utilize all exits. In particular, EN is
recognized with early exits, while DE and RU largely used later exits. We also
observe that in QuickNet and BiRealNet, difficult classes pushed more towards
the last layers, on average using the fourth exit.

##  4 Conclusions

We unify binary neural networks with the early-exiting approach for audio
recognition on devices with low computational resources. To this end, we
present a method to enhance their quality-efficiency trade-offs for efficient
inference. Our conceptually simple and architecture-agnostic approach augments
standard models with early exits at different intermediate layers to make
predictions on easy instances by executing fewer layers. Our experimental
evaluation on three different tasks and four neural architectures demonstrates
the effectiveness of our approach over standard single exit models while
maintaining similar performance at a lower computational budget. Further, our
technique provides a simple strategy based on entropy for controlling the
speed and accuracy of inference with a single model.

## References

  * [1] M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, and Y. Bengio, “Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1,” _arXiv preprint arXiv:1602.02830_ , 2016.
  * [2] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, “Xnor-net: Imagenet classification using binary convolutional neural networks,” in _European conference on computer vision_. Springer, 2016, pp. 525–542.
  * [3] S. Teerapittayanon, B. McDanel, and H.-T. Kung, “Branchynet: Fast inference via early exiting from deep neural networks,” in _2016 23rd International Conference on Pattern Recognition (ICPR)_. IEEE, 2016, pp. 2464–2469.
  * [4] T. Bolukbasi, J. Wang, O. Dekel, and V. Saligrama, “Adaptive neural networks for efficient inference,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 527–536.
  * [5] K. Trapeznikov and V. Saligrama, “Supervised sequential classification under budget constraints,” in _Artificial intelligence and statistics_. PMLR, 2013, pp. 581–589.
  * [6] G. Huang, D. Chen, T. Li, F. Wu, L. Van Der Maaten, and K. Q. Weinberger, “Multi-scale dense networks for resource efficient image classification,” _arXiv preprint arXiv:1703.09844_ , 2017.
  * [7] S. Scardapane, M. Scarpiniti, E. Baccarelli, and A. Uncini, “Why should we add early exits to neural networks?” _Cognitive Computation_ , vol. 12, no. 5, pp. 954–966, 2020.
  * [8] A. Fan, E. Grave, and A. Joulin, “Reducing transformer depth on demand with structured dropout,” _arXiv preprint arXiv:1909.11556_ , 2019.
  * [9] R. Schwartz, G. Stanovsky, S. Swayamdipta, J. Dodge, and N. A. Smith, “The right tool for the job: Matching model and instance complexities,” _arXiv preprint arXiv:2004.07453_ , 2020.
  * [10] S. Laskaridis, A. Kouris, and N. D. Lane, “Adaptive inference through early-exit networks: Design, challenges and directions,” in _Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning_ , 2021, pp. 1–6.
  * [11] L. Geiger and P. Team, “Larq: An open-source library for training binarized neural networks,” _Journal of Open Source Software_ , vol. 5, no. 45, p. 1746, Jan. 2020. [Online]. Available: https://doi.org/10.21105/joss.01746
  * [12] Y. Kaya, S. Hong, and T. Dumitras, “Shallow-deep networks: Understanding and mitigating network overthinking,” in _International conference on machine learning_. PMLR, 2019, pp. 3301–3310.
  * [13] T. Ghosh, “Quicknet: Maximizing efficiency and efficacy in deep architectures,” _arXiv preprint arXiv:1701.02291_ , 2017.
  * [14] Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng, “Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 722–737.
  * [15] J. Bethge, H. Yang, M. Bornstein, and C. Meinel, “Back to simplicity: How to train accurate bnns from scratch?” _arXiv preprint arXiv:1906.08637_ , 2019\.
  * [16] J. Bethge, C. Bartz, H. Yang, Y. Chen, and C. Meinel, “Meliusnet: Can binary neural networks achieve mobilenet-level accuracy?” _arXiv preprint arXiv:2001.05936_ , 2020.
  * [17] S. Shukla, G. Mittal _et al._ , “Spoken language identification using convnets,” in _European Conference on Ambient Intelligence_. Springer, 2019, pp. 252–265.
  * [18] D. Seo, H.-S. Oh, and Y. Jung, “Wav2kws: Transfer learning from speech representations for keyword spotting,” _IEEE Access_ , vol. 9, pp. 80 682–80 691, 2021.
  * [19] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” _arXiv preprint arXiv:1804.03209_ , 2018.
  * [20] K. MacLean, “Voxforge,” _Ken MacLean.[Online]. Available: http://www. voxforge. org/home.[Acedido em 2012]_ , 2018.
  * [21] M. Mazumder, S. Chitlangia, C. Banbury, Y. Kang, J. M. Ciro, K. Achorn, D. Galvez, M. Sabini, P. Mattson, D. Kanter _et al._ , “Multilingual spoken words corpus,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * [22] K. Helwegen, J. Widdicombe, L. Geiger, Z. Liu, K.-T. Cheng, and R. Nusselder, “Latent weights do not exist: Rethinking binarized neural network optimization,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.