# Prior Knowledge Input to Improve LSTM Auto-encoder-based Characterization of
Vehicular Sensing Data

Nima Taherifard, Murat Simsek , Charles Lascelles, and Burak Kantarci N.
Taherifard, M. Simsek and B. Kantarci are with the School of Electrical
Engineering and Computer Science, University of Ottawa, Ottawa, ON, Canada.
E-mails: {ntahe062, murat.simsek<EMAIL_ADDRESS>Lascelles
is with Raven Connected, 441 MacLaren St, Ottawa, ON, K2P 2H3, Canada. Email:
<EMAIL_ADDRESS>
###### Abstract

Precision in event characterization in connected vehicles has become
increasingly important with the responsive connectivity that is available to
the modern vehicles. Event characterization via vehicular sensors are utilized
in safety and autonomous driving applications in vehicles. While
characterization systems have been shown to be capable of predicting the risky
driving patterns, precision of such systems still remains an open issue. The
major issues against the driving event characterization systems need to be
addressed in connected vehicle settings, which are the heavy imbalance and the
event infrequency of the driving data and the existence of the time-series
detection systems that are optimized for vehicular settings. To overcome the
problems, we introduce the application of the prior-knowledge input method to
the characterization systems. Furthermore, we propose a recurrent-based
denoising auto-encoder network to populate the existing data for a more robust
training process. The results of the conducted experiments show that the
introduction of knowledge-based modelling enables the existing systems to
reach significantly higher accuracy and F1-score levels. Ultimately, the
combination of the two methods enables the proposed model to attain 14.7%
accuracy boost over the baseline by achieving an accuracy of 0.96.

###### Index Terms:

Deep learning, knowledge-based modelling, encoder networks, intelligent
transportation, LSTMs, vehicular sensing.

##  I Introduction

Leveraging precise measuring equipment along with the increasing computing
power of the vehicles, the driving event characterization (DEC) systems are
the key components to the road safety of the vehicles in an intelligent
transportation system (ITS) where highly reliable connectivity is available
between the vehicles [1]. Moreover, the response time improvement introduced
by 5G [2] further incentivizes the investigation and refinement of the DEC for
vehicular safety systems.

Typically, DECs are mainly deployed in distributed sensing platforms such as
mobile devices and vehicles on the road and the inertial sensor data is the
main instrument used by the systems. Utilizing the in-vehicle inertial sensor
data allows for more prompt detection with less lag that can be utilized in
the safety applications since inertial data are direct measures of the
physical forces applied to the vehicles [3].

It is proven that specific driving characteristics like sudden lane changes,
acceleration and such are tightly associated as risky driving behaviors [4].
To detect these behaviors, the DEC systems follow the objective of detecting
anomaly behavior or classifying the pre-defined behavior in the data [5].
These events include risky driving patterns such as aggressive lane changing,
aggressive acceleration, harsh brakes, etc. which occur in small time windows
during regular driving patterns and cause the issue of data imbalance.

The most recent developments in artificial intelligence and machine learning
have made the real-time detection of the driving events more feasible and
accurate [6] by catering accurate methods for analysis of time-series data.
More specifically, the recurrent networks which are able to store a brief
history of the past input data at any time that makes them the feasible tools
for time-series data analysis. Recurrent models such as Long short-term memory
(LSTM) architecture allows the machine-learning systems to reserve and
consider longer history of the data in the decision making by the networks.
Furthermore, storing recent data information is critical for pattern
recognition methods employed for driving event characterization systems. LSTM
networks are a refined type of recurrent neural networks which are utilized to
extract temporal features in time-series data [7]. Such networks are the main
enabling factors of the contemporary weather forecasting and language models
which heavily depend on the sequential history of the most recent data [8].

Figure 1: The PKI modulation as module to the classification networks. It
accepts the knowledge of the input signals as additional feature for an
optimized training process.

In this paper, in order to study the effect of knowledge-based modelling, a
previously studied and optimized convolutional LSTM encoder network is chose
for baseline experiments. The network allows the system to extract both
spatial and temporal features of the signals in order to achieve best
performance. Convolutional layers are utilized as spatial feature extraction
module since the events have distinct patterns on each axis of the input data
while LSTM network performs temporal feature extraction to gain knowledge of
the time dependant patterns of the signals. Moreover, a recurrent encoder-
decoder network is proposed to not only denoise the input signals but also
learn the behavior pattern of individual signals for precise reconstruction of
synthetic signals.

The main contribution of this paper is the introduction of the Prior Knowledge
Input (PKI) modelling into machine learning-based recurrent event
characterization models. Knowledge based modelling has been developed to
integrate the existing knowledge to the learning process so further
improvement can be possible through the mapping between existing knowledge and
desired responses. Prior Knowledge Input (PKI) that is one of the knowledge
based methods is considered to obtain the better response (up to 0.96
accuracy) when compared to the previous work [5]. PKI utilizes the existing
knowledge of driving events as supplementary input alongside with the sensed
input signals. As illustrated in Fig. 1, the PKI is modulated to the existing
characterization networks as add-on, which can enable the models to reach the
optimal training conditions.

The rest of the paper is organized as follows. Related work is presented in
Section II whereas Section III presents the methodology. Performance
evaluation and numerical outcome are reported and discussed in Section IV.
Finally, Section V concludes the work and offers future intentions.

##  II Related Work

Numerous studies have been proposed to accelerate the feasibility of the
driving event characterization models. Additionally, neural networks have
surpassed the machine learning and other intelligent methods in terms of
performance. The signal processing domain has also been affected by the
aforementioned fields, therefore many researchers are focusing on deep
learning methods in order to improve the state-of-the-art event
characterization systems. In this section, we first summarize the most recent
driving event characterization models and then review the developments of the
knowledge-based systems.

###  II-A Driving Event Detection Systems

Since vehicular sensory data has become increasingly available, the use of
neural networks which are highly dependant on big data has become feasible for
vehicular applications, including the driving event characterization
applications. The systems that utilize such networks are typically focused on
visual or sensory data inputs from the vehicles. Ultimately, these models are
trained to extract the desired features from the data and identify the
anomalies or the pre-defined behaviors from the features.

A series of studies were conducted on the feature extraction methods to gain
knowledge from the inertial or global positional sensed data. Sun et al. [9]
propose a system to identify irregular driving on the highways based on
accurate positional satellite measurements. A unique calculation method is
studied in [10] to model the velocity, acceleration, and dynamics of the
vehicles. A static threshold value is then applied to the calculated metric in
order to detect driving behaviors. Additionally, more studies are focused on
the combination modelling of the vehicular measured data. In [11], the authors
have examined the correlation between the vehicle velocity and lateral
acceleration. Investigating this correlation, they conclude that the threshold
applied to the lateral acceleration of the vehicle in an event
characterization model has to be decreased as the vehicle gains speed.
Moreover, smartphone reliant driving event characterization systems are also
studied in the literature [12] to utilize the cost-efficient in-built sensors
of the smartphones. In [13], the authors designed a system to detect sudden
changes of acceleration and unsafe vehicle turns in order to categorise the
drivers as aggressive or non-aggressive drivers.

Figure 2: Data collection method. The utilized sensor records acceleration
along the three axes of the vehicles.

Modern day transportation industry is adopting machine-learning systems that
can improve through data-driven approaches [14, 15]. Optimized image
processing methods are utilized in [16] to categorize driving incidents and
make decision on alerting the first-responders to the location. A pre-trained
convolutional neural network is used in [17] on the phase-space reconstructed
vehicle trajectories to evaluate driving behaviors. The study quantitatively
evaluates the abnormal behaviors to then detect the abnormal drivers.
Moreover, the distracting behaviors that lead to unsafe driving are the
subject of various studies. Rao et al. [18] takes the approach of processing
the captured camera data via convolutional neural networks and principal
component analysis (PCA) to whiten the camera feed and classify the whitened
images as distracted and non-distracted driver activities.

Recurrent neural networks have been improved and made practical more recently.
In addition to the progress of various types of recurrent functions, numerous
recurrent-based network architectures have been introduced for innovative
applications. There has been novel solutions to the major issue of recurrent
neural networks, i.e., the vanishing gradient problem which led the networks
to face issues in the training process. Long Short-term Memory (LSTM) and
Gated Recurrent Units (GRUs) models are among most recent solutions to the
issue. Furthermore, there has been numerous studies in order to expand the
applications of the recurrent networks. [19] proposes data compression method
utilizing recurrent networks. Combining generative and discriminative
networks, a denoising auto-encoder is introduced in [20] in which the model
has the ability to take the input features on the fly. An auto-encoder network
with the support of random sampling from the encoder latent space is proposed
with generative objective in [21]. The model mixes a combination of
adversarial and reconstruction losses, but unlike Generative Adversarial
Networks (GAN) discriminators, the authors employed progressive growing of
generator and encoder networks.

###  II-B Knowledge-based Modeling

Knowledge-based modelling is suitable for the driving event classification
systems since it only requires the input-output relations [22]. In a
conventional neural network, a large training dataset is required to meet the
optimal stopping conditions which is not convenient in vehicular applications
where there is imbalance to the data. In [23], machine-learning detection
models are boosted by novel knowledge-based methods and feature selection
mechanisms. Utilizing Prior Knowledge Input (PKI) and Prior Knowledge Input
with Difference (PKID), the authors are able to significantly improve the
average accuracy of the existing machine-leaning methods. The PKI is widely
adopted in neural network-based modeling systems such as adversarial task
detection in mobile crowd-sensing to microwave modelling for computer aided
design [24]. Using the PKI modelling, it is possible to remarkably improve the
driving event characterization model in terms of accuracy and overall
performance. Indeed, further experimental results are needed which are
presented in the following sections.

##  III Methodology

In this section, the prior knowledge-based signal classification model which
is modulated on top of our previous signal generation network is presented in
detail. Machine-learning is modelled to imitate the learning process of the
brain, and its classification accuracy and the training process efficiency can
be improved by modelling the networks utilizing a knowledge-based modulation.
We employ a prior knowledge input method to boost our classification
performance which is presented in Section III-C. A brief introduction of our
dataset collection process and the details of our signal generation model are
revisited in Section III-A and Section III-B, respectively whereas the
parameters are presented in Table I.

TABLE I: Notation description for mathematical formulas. Notation | Description
---|---
$x$ | Signal features
$\hat{x}$ | Synthetic signal features
j | Signal dimension
i | Signal length
C | Number of classes
t | Target category of the signal
$e_{PKI}$ | Error value of PKI model
n | Training iteration
$P_{c}$ | Prediction of the classifier network

###  III-A Driving Event Data Gathering

Both the signal generation network as well as the knowledge-based
classification network utilize raw accelerometer sensor data over varied
recording sessions. In order to collect the original signals, Raven OBD-II
sensor kit 111Raven OBD-II Sensor Kit https://ravenconnected.com/ is mounted
on the dashboard of the vehicles. The sensor orientation is perpendicular to
the latitudinal, altitudinal, and longitudinal axes of the vehicle as
illustrated in Fig. 2.

The collected raw signal data are distributed among five pre-defined risky
driving behaviors and on multiple vehicles with different physical features.
The pre-defined behaviors include: Aggressive Acceleration (AA), Harsh Braking
(HB), Harsh Left Lane Changes (HL), Harsh Right Lane Changes (HR) events in
addition to Regular Driving (RD) events.

Figure 3: Illustration of the baseline convolutional recurrent neural network.
Multiple convolutional and LSTM feature extraction units are utilized before
the classification network.

###  III-B Revisiting Synthetic Signal Generation

In an attempt to overcome the data imbalance and infrequency, we previously
proposed an auto-encoder model [25] which is able to accurately extract the
underlying behavior of the signals and generate precise synthetic signals. The
model consists of two modules, namely, a LSTM recurrent encoder-decoder
network which is leveraged in order to populate the training dataset. The
second module is used for classification of the signals and is trained on the
synthetic dataset.

In order to generate the signals, a windowing mechanism is ran over each
individual raw signal. Using the windowing mechanism, the raw signals are
split into several signal windows of fixed length with overlapping sections.
The overlapping is performed in order to avoid discontinuities in the windowed
signals.

The auto-encoder consists of an encoding network which encodes the input
signals to vectors of lower dimension that are feature-rich and contain less
noise [26]. Moreover, the encoded vectors are used by the decoding network to
reconstruct synthetic signals. The synthetic signal reconstruction is carried
out to populate the events in the dataset and to strengthen the classification
training process by mitigating data imbalance.

To train the recurrent auto-encoder for synthetic signal generation, the
signal windows are infused with randomly generated noise and fed as the input
to the auto-encoder network. The recurrent auto-encoder network concludes a
two-layer LSTM network as the encoder which maps the signal windows into
fixed-size vectors. It is worth mentioning that the conversion of the signals
into lower dimension vectors by the encoder network has the property of signal
denoisification since high dimensional data is often more convenient to learn
in lower dimensions as it contains less noise. The decoder network attempts to
reconstruct the original signals from the vectors. We set the network to
generate 10 times synthetic signal windows for the training process of the
classification module. In the training process, the auto-encoder network is
trained in self-supervised manner with the intention of minimizing the mean
absolute error (1) of the original and generated signals.

$\sum_{j}\sum_{i}\left\|x^{\left(j\right)}-{\hat{x}}^{\left(j\right)}\right\|^{2}$
(1)

In order to demonstrate the improvement introduced by the PKI based modelling
proposed in this work, we trained a feed forward multi-layer perceptron (MLP)
of 3 layers and a convolutional recurrent classification (ConvLSTM) model
which adds the spatio-temporal feature extraction layers to the MLP network.
The ConvLSTM model utilizes multiple feature extraction layers of decreasing
sizes as illustrated in Fig. 3. However, the objective of both classification
networks is to categorize the signals into 5 pre-defined driving event
categories using categorical cross-entropy which is implemented as the softmax
(2) of cross-entropy function (3).

$f\left(x\right)=\frac{e^{x}}{\sum_{C}e^{x}}$ (2)

$CE=-\sum_{C}t\;log(f(x))$ (3)

###  III-C Prior Knowledge Input

To boost the performance of the classification networks on the driving event
signals, we propose knowledge-based schemes. The Prior Knowledge Input (PKI)
method, specifically, is used to incorporate the existing knowledge of the
signals into the characterization process. Doing so allows for a more
efficient and less complex training process which requires less amount of
training data [23] and aids the characterization systems to push past the
limited accuracy of deep learning methods and achieve the potential accuracy
rates [22]. Moreover, any modelling scheme can be utilized in order to
generate the input knowledge when there is no available prior knowledge of the
inputs. Though, neural networks are often the method of choice for the
knowledge-based modelling [27].

The PKI model embeds the experience/knowledge of the signal category into
training process which allows for reduction of model complexity through
augmentation of the model inputs. The training and testing process of the PKI
model is shown in Fig 4. The PKI network is implemented in 2 hidden layers
that utilize $tanh$ activation function with a softmax classification layer to
output the event prediction. The training process can be formulated with (4)
while iterating to lower the margin of the network predictions and the prior
knowledge (5) for $n$ training iterations. The inference process is also
calculated by (6) as illustrated in Fig. 4.

$\left(PKI\right)_{Train}=\arg\min_{x}{\left\|\;...\;\;\;e^{(n)^{T}}_{PKI}\;\;...\;\right\|}$
(4)

$e^{(i)}_{PKI}=C_{events}(x^{(i)})-PKI(x^{(i)},P^{(i)}_{c})$ (5)

$P_{PKI}=(PKI)_{Train}(x_{,}P_{c})$ (6)

Figure 4: Training and testing process of the prior knowledge input method to
improve existing performance of driving event characterization systems.

##  IV Experimental Evaluation

###  IV-A Experimental Settings

All the experiments are performed on the gathered dataset of 70 driving event
sessions populated by 10 times synthetic signals using the auto-encoder
network. To keep the experiments credible, all tests are performed on an
identical testset.

Initially, accelerometer sensor signals with variable duration of 2000 to 3600
milliseconds are captured. The accelerometer recording frequency is set to 25
Hz. In total, 70 driving event sessions are recorded. The distribution of the
collected dataset is demonstrated in Table II. Originally, 12 AA, 13 HB, 16
HL, 15 HR, and 14 RD driving sessions were accumulated in the dataset. The
sessions are split into 572 fixed-size event windows through the sliding
mechanism using 600 millisecond window length and 50% overlap settings as
demonstrated in Fig. 5. Last but foremost, a randomly selected training
dataset of 70% (400) sliced windows is selected and 30% (172) sliced windows
are kept unseen by the systems for testing.

TABLE II: Training signal count distribution before and after slicing over 5 classes. Event Type | Event Sessions | Sliced Windows
---|---|---
HB | 13 | 104
AA | 12 | 108
HL | 16 | 126
HR | 15 | 121
RD | 14 | 113
Figure 5: Abstract visualization of the signal slicing performed before the
training process. The signals are sliced into 6000 ms windows with 0.5 overlap
value. Figure 6: Training process flow of the PKI method. The vector
representation of the noisy raw signals are extracted from the encoder
network. The vectors are utilized by the decoder network for synthetic signal
generation and by the PKI classification network for characterization of the
events.

The synthetic data generation is don on the training data through the auto-
encoder network with 3 LSTM network with decreasing hidden layers, from 1000
to 300 cells. However, each individual input signal is first duplicated into
10 individual signals and each is infused with random Gaussian noise. The
noisy signals are mapped to vectors of size 300 at the final hidden layer of
the encoder network. Subsequently, the decoder network is implemented as a
mirrored LSTM network with the same dimensions to the encoder and attempts to
reproduce the raw signals from the noisy inputs. Furthermore, the decoder
output (i.e. synthetic training data) is used in the training process of the
classifier networks. However, in the inference process, the decoder network is
deactivated and the encoded vectors are directly passed to the classifier
network. The process flow of the system is depicted in Fig. 6.

In order to demonstrate the impact of the PKI module, we chose a fully
connected feed forward classifier network, more specifically a Multi-Layer
Perceptron (MLP), as baseline as well as an optimized convolutional recurrent
classifier network (ConvLSTM) for further experimentation. Moreover, the
classification modules perform a supervised task utilizing categorical cross-
entropy for optimization on each training iteration and aim to lower the
margin of the predicted events and the target categories. Adam optimizer with
training parameter of $\beta_{1}=0.9$ and $\beta_{1}=0.95$ and
$\epsilon=10^{-8}$ is selected for both classifier networks. Lastly, the
networks are trained with learning rate of 0.03 until the stopping is
triggered.

###  IV-B Numerical Results

As mentioned earlier, the MLP and ConvLSTM classification networks are trained
on our driving event dataset with and without the PKI modulation.
Additionally, the networks are trained and tested on raw signals, in the
absence of the reconstructed synthetic data, for a more comprehensive
demonstration of the PKI benefits. Through testing of the models on identical
testset, the impact of the PKI model is demonstrated in Table III.

Collecting the experimental results, the performance improvement of PKI
modulation can be observed across all the tested classification models. The
experiments on each classifier network are performed over 10 separate runs for
statistical presentation of the results which are stated with confidence
levels of 95%. While the baseline MLP classifier gained the most significant
accuracy improvement (by average of over 5%), the PKI modulation proved to
increase our most accurate model (i.e. auto-encoder with ConvLSTM classifier)
to over 96% average accuracy as shown in Fig. 7. Last but not least, as a
benefit of PKI accuracy improvements, the decrease in false positive cases of
the models is reflected in improved precision, recall, and therefore F1-score
values across the models.

Figure 7: Prior Knowledge Input modulation impact comparison. TABLE III: Numerical results. PKI Modulation | Model | Accuracy | Precision | Recall | F1-Score
---|---|---|---|---|---
OFF | MLP | 0.624$\pm$0.014 | 0.27$\pm$0.011 | 0.30$\pm$0.012 | 0.29$\pm$0.012
OFF | ConvLSTM | 0.814$\pm$0.009 | 0.77$\pm$0.010 | 0.79$\pm$0.009 | 0.78$\pm$0.009
OFF | AE + MLP | 0.932$\pm$0.007 | 0.89$\pm$0.008 | 0.90$\pm$0.008 | 0.89$\pm$0.007
OFF | AE + ConvLSTM | 0.940$\pm$0.004 | 0.12$\pm$0.003 | 0.11$\pm$0.002 | 0.11$\pm$0.003
ON | MLP | 0.678$\pm$0.009 | 0.39$\pm$0.010 | 0.44$\pm$0.009 | 0.42$\pm$0.009
ON | ConvLSTM | 0.861$\pm$0.008 | 0.68$\pm$0.008 | 0.62$\pm$0.010 | 0.65$\pm$0.009
ON | AE + MLP | 0.954$\pm$0.006 | 0.87$\pm$0.008 | 0.90$\pm$0.006 | 0.88$\pm$0.007
ON | AE + ConvLSTM | 0.961$\pm$0.003 | 0.87$\pm$0.002 | 0.93$\pm$0.002 | 0.90$\pm$0.002

##  V Conclusion

This paper has presented a novel approach to boost the classification
performance for risky driving events that are recognized from vehicular
sensors. To do so, the integration of a Prior Knowledge Input (PKI) modelling
into the event characterization networks has been proposed to not only improve
the overall classification accuracy but also reduce the false detections. The
PKI model leverages the existing knowledge of the input signals as an
supporting feature in order to lower the complexity of the model and therefore
improve the detection accuracy of the baseline classification networks. The
results prove that the performance of the classification models can be
significantly improved with the introduction of the PKI modelling. Our results
reveal that using the PKI method improves the performance of the baseline MLP
classifier by over 5%. Similarly the ConvLSTM network experiences a 4.7%
improvement when coupled with the prior knowledge module while the performance
of the auto-encoder with ConvLSTM is also improved by an additional 2%.

Future extensions of this work include extending the experiments to run on
larger datasets where data scarcity is not a challenge. Furthermore,
characterization of additional behaviour classes is also included our ongoing
work.

## Acknowledgment

This work was supported in part by the Natural Sciences and Engineering
Research Council of Canada (NSERC) DISCOVERY RGPIN/2017-04032 and Ontario
Centres of Excellence (OCE) TalentEdge Internship Project 32815 in
collaboration with Raven Connected.

## References

  * [1] F. A. Silva, A. Boukerche, T. R. B. Silva, L. B. Ruiz, E. Cerqueira, and A. A. Loureiro, “Vehicular networks: A new challenge for content-delivery-based applications,” ACM Comp Surv, vol. 49/1, pp. 1–29, 2016.
  * [2] C. Campolo, A. Molinaro, A. Iera, and F. Menichella, “5g network slicing for vehicle-to-everything services,” IEEE Wireless Communications, vol. 24, no. 6, pp. 38–45, 2017.
  * [3] J. Ryu and J. C. Gerdes, “Integrating inertial sensors with global positioning system (gps) for vehicle dynamics control,” J. Dyn. Sys., Meas., Control, vol. 126, no. 2, pp. 243–254, 2004.
  * [4] J. M. Scanlon, R. Sherony, and H. C. Gabler, “Models of driver acceleration behavior prior to real-world intersection crashes,” IEEE Trans. on Intel. Transportation Systems, vol. 19/3, pp. 774–786, 2017.
  * [5] N. Taherifard, M. Simsek, C. Lascelles, and B. Kantarci, “Attention-based event characterization for scarce vehicular sensing data,” IEEE Open Journal of Vehicular Technology, vol. 1, pp. 317–330, 2020.
  * [6] Z. Che, S. Purushotham, K. Cho, D. Sontag, and Y. Liu, “Recurrent neural networks for multivariate time series with missing values,” Scientific reports, vol. 8, no. 1, pp. 1–12, 2018.
  * [7] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [8] A. Sagheer and M. Kotb, “Time series forecasting of petroleum production using deep LSTM recurrent networks,” Neurocomputing, vol. 323, pp. 203–213, 2019.
  * [9] R. Sun, W. Y. Ochieng, and S. Feng, “An integrated solution for lane level irregular driving detection on highways,” Transportation Research Part C: Emerging Technologies, vol. 56, pp. 61–79, 2015.
  * [10] A. S. Zeeman and M. J. Booysen, “Combining speed and acceleration to detect reckless driving in the informal public transport industry,” in Intl. IEEE Conf. on Intelligent Transportation Systems, pp. 756–761, 2013.
  * [11] L. Eboli, G. Mazzulla, and G. Pungillo, “Combining speed and acceleration to define car users’ safe or unsafe driving behaviour,” Transportation research part C: emerging technologies, vol. 68, pp. 113–125, 2016.
  * [12] C. Saiprasert, T. Pholprasit, and S. Thajchayapong, “Detection of driving events using sensory data on smartphone,” Intl. J. of intelligent transportation systems research, vol. 15, no. 1, pp. 17–28, 2017.
  * [13] R. Chhabra, S. Verma, and C. R. Krishna, “Detecting aggressive driving behavior using mobile smartphone,” in Intl. Conf. on Communication, Computing and Networking, pp. 513–521, Springer, 2019.
  * [14] M. Soyturk, K. N. Muhammad, M. N. Avcil, B. Kantarci, and J. Matthews, “From vehicular networks to vehicular clouds in smart cities,” in Smart Cities and Homes, pp. 149–171, Elsevier, 2016.
  * [15] A. Boukerche, B. Kantarci, and C. Kaptan, “Towards ensuring the reliability and dependability of vehicular crowd-sensing data in gps-less location tracking,” Perv.& Mobile Comp., vol. 68, p. 101248, 2020.
  * [16] N. Taherifard, M. Simsek, and B. Kantarci, “Bridging connected vehicles with artificial intelligence for smart first responder services,” in IEEE Global Conf. on Signal and Information Processing, pp. 1–5, 2019.
  * [17] X. He, L. Xu, and Z. Zhang, “Driving behaviour characterisation by using phase-space reconstruction and pre-trained convolutional neural network,” IET Intelligent Transport Systems, vol. 13, no. 7, pp. 1173–1180, 2019\.
  * [18] X. Rao, F. Lin, Z. Chen, and J. Zhao, “Distracted driving recognition method based on deep convolutional neural network,” Journal of Ambient Intelligence and Humanized Computing, pp. 1–8, 2019.
  * [19] G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell, “Full resolution image compression with recurrent neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5306–5314, 2017.
  * [20] A. Ashfahani, M. Pratama, E. Lughofer, and Y.-S. Ong, “Devdan: Deep evolving denoising autoencoder,” Neurocomputing, vol. 390, pp. 297–314, 2020.
  * [21] A. Heljakka, A. Solin, and J. Kannala, “Pioneer networks: Progressively growing generative autoencoder,” in Asian Conference on Computer Vision, pp. 22–38, Springer, 2018.
  * [22] M. Simsek and A. Aoad, “Efficient reconfigurable microstrip patch antenna modeling exploiting knowledge based artificial neural networks,” in Simulation-Driven Modeling and Optimization (S. Koziel, L. Leifsson, and X.-S. Yang, eds.), (Cham), pp. 185–206, Springer International Publishing, 2016\.
  * [23] M. Simsek, B. Kantarci, and A. Boukerche, “Knowledge-based machine learning boosting for adversarial task detection in mobile crowdsensing,” in IEEE Symp. on Comp. and Communications, [virtual], July 2020.
  * [24] M. Simsek, Q.-J. Zhang, H. Kabir, Y. Cao, and n. s. Sengor, “The recent developments in microwave design,” Int. J. of Mathematical Modelling and Numerical Optimisation, vol. 2, pp. 213 – 228, 04 2011.
  * [25] N. Taherifard, M. Simsek, C. Lascalles, and B. Kantarci, “Machine learning-driven event characterization under scarce vehicular sensing data,” in IEEE International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), 2020.
  * [26] W. Yu, G. Zeng, P. Luo, F. Zhuang, Q. He, and Z. Shi, “Embedding with autoencoder regularization,” in Joint European Conf. on ML and Knowledge Discovery in Databases, pp. 208–223, Springer, 2013.
  * [27] M. Simsek, “Efficient neural network modeling of reconfigurable microstrip patch antenna through knowledge-based three-step strategy,” International Journal of Numerical Modelling: Electronic Networks, Devices and Fields, vol. 30, no. 3-4, p. e2160, 2017.


# DooDLeNet: Double DeepLab Enhanced Feature Fusion for Thermal-color Semantic
Segmentation

Oriel Frigo
Lucien Martin-Gaffé

AnotherBrain
Paris, France
{oriel, lucien<EMAIL_ADDRESS>Catherine Wacongne

###### Abstract

In this paper we present a new approach for feature fusion between RGB and
LWIR Thermal images for the task of semantic segmentation for driving
perception. We propose DooDLeNet, a double DeepLab architecture with
specialized encoder-decoders for thermal and color modalities and a shared
decoder for final segmentation. We combine two strategies for feature fusion:
confidence weighting and correlation weighting. We report state-of-the-art
mean IoU results on the MF dataset [1].

##  1 Introduction

Thermal images can improve the performance of computer vision algorithms,
particularly during night time or in harsh weather conditions, when visibility
becomes limited. On the other hand, color images provide a richer source of
visual information during day time. Hence it can be highly beneficial for
vision models to combine both color and thermal images into a RGBT (red,
green, blue, thermal) image pipeline.

Multimodality or the combination of different imaging sources becomes crucial
to alleviate perception and ensure robustness under poor visibility
conditions, but the combination of different cameras and sensors comes with
alignment and synchronization challenges. Calibration of thermal images is a
painstaking task where special heated or masked calibration patterns need to
be employed. Alignment of color and thermal images is also a challenging
problem, not only because of common differences in field of view and
resolution between the color and thermal cameras but also because of the huge
differences in image appearance between the two imaging modalities.

In this paper we consider the task of semantic segmentation of paired color
RGB and thermal LWIR (Long Wave Infra-Red) images. Most previous research in
this field has been focused in searching for good fusion strategies between
the color and thermal images. However, previous work on semantic color-thermal
segmentation has barely considered how to fuse the images taking into account
the feature disagreement due to poor alignment.

Inspired by recent advances in semantic matching, here we consider an end-to-
end deep learning strategy which indirectly takes into account the alignment
of RGB and LWIR thermal images through a matching correlation layer.

We show the benefits of the correlation and confidence aware feature fusion
for the task of semantic segmentation on RGB-thermal data.

The main contributions of this paper are the following:

  * •

We propose a new CNN model for color-thermal semantic segmentation. We show
the interest of learning deep features that are specialized for both thermal
images and color images with specialized encoder-decoder paths.

  * •

We show that semantic segmentation performance on color-thermal image pairs is
increased by the aid of two different weighted feature fusion strategies:

    * –

For matching correlation weighting, we explicitly estimate the feature
matching at the last specialized decoder layer by computing feature
correlations.

    * –

For confidence weighting, we estimate the segmentation confidence at both
color and thermal features, and this confidence is used to weight differently
the color and thermal features.

##  2 Literature Review

###  2.1 Semantic segmentation

Autonomous driving requires understanding the scene ahead. The release of
datasets BDD100K [2] and Cityscapes [3] has boosted the interest of using
semantic segmentation as a way to process images for driving applications.
Semantic segmentation consists in assigning a class to every pixel of an
image. Early efforts in deep-learning based semantic segmentation used fully
convolutional network (FCN) [4] which allowed for dense pixel classification
of arbitrary sized images. However, FCN generated pixel labels from high level
feature maps, resulting in relatively imprecise object boundaries.
Subsequently, one of the main challenge of this task was to propose methods
that extracted global context while providing pixel precise information to
recover both the semantic categories present in the image and the pixel level
boundaries of each object. Several approaches were developed to do so.

Modifying the original FCN design, encoder/decoder architectures were
proposed, that compress information to capture the global context before
decompressing it using either deconvolution [5], pooling indexes (SegNet)[6]
or skip connections (U-net) [7] to recover the fine spatial information.

Other methods use pyramids of features [8, 9] extracted by an encoder model
applied to the same image at different resolutions. The features maps can then
be upsampled to match full scale the image size. More recent work proposed to
use atrous convolutions [10, 11] to capture features at different spatial
scales without the need to upsample the feature maps. Combining atrous
convolutions and spatial pyramid pooling, Deeplabv3+ [10] proposes a simple
and effective encoder-decoder FCN architecture.

Another approach consists in introducing attention mechanisms [12, 13, 14] to
combine different levels of feature maps and better capture long-range image
dependencies.

###  2.2 Multimodality and driving perception

The release of the KAIST Multispectral dataset [15] can be seen as an
important step in the direction of combining color and thermal images for
driving perception. The dataset provides physically aligned color and thermal
images with bounding boxes annotations for a few semantic classes representing
humans, such as _person_ , _people_ and _cyclist_.

Besides the release of the KAIST dataset, other works have considered the
problem of multimodality in driving perception. The work of [16] proposed a
cycle consistent GAN which generates synthetic color images from LWIR and
synthetic LWIR from color images while enforcing that object detection remains
accurate in both modalities. The authors obtain the best object detection
accuracy when combining synthetic RGB and true LWIR images.

It should be noted that thermal and color images are often not perfectly
aligned, due to the parallax effect, poor calibration or registration. This
implies that naive input fusion, which consists in simply stacking the color
and thermal images together by forming a 4-channel RGBT data input is rarely
the most effective fusion strategy.

A number of works investigated better strategies to fuse different image
modalities together in one model. Fusenet [17] was one of the first works
dealing indirectly with the multimodal fusion problem, by proposing a double
encoder architecture for the fusion of depth maps and color images.

The work of [1] proposed the MFNet (Multi-spectral Fusion Networks), a method
for semantic segmentation on RGBT data. Their model consists of separate
encoders for RGB and thermal data and fusion at the decoder level. The authors
also published a semantic segmentation dataset on RGBT data, the MF dataset.
The RTFNet method [18] performs semantic segmentation on RGBT data by fusing
RGB and thermal features at multiple network depths, the method was also
evaluated on the MF dataset [1]. Fusion of features extracted at multiple CNN
layers as performed by [18] can be seen a strategy to attenuate feature
disagreements, as the misalignment magnitude is decreased in deeper network
layers.

FuseSeg-161 [19], a multi-modal fusion model has been able to fuse feature
from thermal and RGB images in one end-to-end model. It consists of two
encoders to extract features from input, one for RGB and another for thermal
images with a two-stage fusion (TSF) strategy. In the first stage, the thermal
feature maps are hierarchically added with the RGB feature maps in the RGB
encoder. The fused feature maps are then concatenated with the corresponding
decoder feature maps in the second fusion stage. TSF method achieved over
state-the-art semantic segmentation performance under various lighting
conditions.

More recently, FEANet [20] proposed a new way to extract detail spatial
information with a two-stage Feature-Enhanced Attention Network. The Feature-
Enhanced Attention Module (FEAM) improve excavation feature capacity from both
the channel and spatial views, involving a better preservation of spatial
information. Thus, the model shift more attention to high-resolution features
which outperform other state-of-the-art techniques.

Another approach consists in training a deep neural network that weights
differently thermal and color features. A first work in this direction was
done by [21], where the model learns to estimate a position shift between
pedestrian bounding boxes in color and thermal modalities.

Our work is related to the method proposed by [21] in the sense of performing
a sort of confidence feature weighting. However, while [21] is focused on
bounding box retrieval and weighting, we compute dense confidence and
correlation weights that are better suited for semantic segmentation.

##  3 Proposed method

Figure 1: Summarized illustration of model architecture for weighted fusion
semantic segmentation. The RGB and thermal encoder-decoder paths are
specialized and both provide a rough segmentation used to compute a spatial
confidence score. The confidence and matching correlation is used to weight
the RGB and thermal features before fusing them in the final shared decoder.
Solid arrows represent concatenation-based skip connections, dashed arrows
represent data flow. Weighted concatenated fusion is performed with features
coming from two different encoder depths (layer 2 and layer 4).

Given a dataset of paired color and thermal images with pixel-level target
labels, we seek to perform semantic segmentation on this data by fusing color
and thermal images in a manner that maximizes overall segmentation
performance. Here we propose a mid-level feature fusion approach, which aims
to limit the feature disagreements caused by misalignment between the thermal
and color images.

Differently than the work of [18] which performs fusion of RGB and thermal
features directly at the RGB encoder, we employ completely specialized thermal
and color encoders.

We rely upon the DeepLabV3+ [10] model backbone, which has dilated (atrous)
convolutions, limiting the loss of spatial resolution and better preserving
structural details. The main decoder ingredient is the ASPP (Atrous Spatial
Pyramid Pooling) which obtains multi-scale features obtained by atrous
convolutions with different dilation rates.

We propose a double encoder-decoder variant of DeepLabV3+ where fusion is
performed at two different feature resolutions at the decoder side, with a
shared decoder for the final segmentation. A motivation for our approach comes
from the observation that rich encoder features from pretrained networks such
as ResNet [22] could be adapted as seamless as possible for multiple deep
learning tasks, in order to minimize training time and maximize prediction
performance.

Moreover, we perform a weighted feature fusion which takes into account
feature agreement and feature confidence. Our proposed architecture is
illustrated in Fig. 1. We detail the two weighting strategies in the following
subsections.

###  3.1 Confidence weighted fusion

It can be claimed that direct fusion of RGB and thermal features may be
problematic depending on the time of day, weather and visibility conditions of
the captured scene.

For instance, during day time and favorable weather, RGB features could
provide the best features to perform segmentation, as thermal images are only
one-channel and texture details from objects are hardly noticeable.

On the other hand, during night time and at low light, thermal images may
provide better features to segment hardly visible objects, in particular the
ones with distinguishable changes in temperature compared to the background
such as pedestrians and cars.

If we always give equal importance for features coming from both the
modalities, the model may achieve lower performance in segmenting objects when
one of the modalities provide unreliable features.

To alleviate this issue, we propose a confidence-based weighting before
performing feature fusion, such that RGB and the thermal features receive
difference importance based on their reliability. The decoder and the
segmentation heads of each modality are able to perform a preliminary
segmentation, from which a confidence can be estimated.

Based on the model confidence of the preliminary segmentation, we may be able
to give more weight to the more reliable features.

Figure 2 illustrates how confidence is computed for each imaging modality.

Figure 2: Encoder-decoder path for modality confidence-aware feature weighting
based on DeepLabv3+ architecture. The illustrated procedure is computed for
both color and thermal images. Resulting re-weighted features from color and
thermal are concatenated together as input for the shared decoder.

Formally, we denote the color image as $I_{c}$ and thermal image as $I_{t}$.
Each image modality $I_{m}$, with $m\in\\{c,t\\}$ has a set of features
computed by a specialized encoder $\mathbf{f}_{m}=e_{m}(I_{m})$ and a modality
prediction computed by specialized decoder
$\mathbf{y}_{m}=d_{m}(\mathbf{f}_{m})$.

The confidence weight matrix $C_{m}$ for modality $m$ is given by an
estimation of the confidence on the most probable prediction:

$C_{m_{i}}=\text{max}\left(\frac{\exp(\mathbf{y}_{m_{i}})}{\sum_{j}\exp(\mathbf{y}_{m_{j}})}\right)$
(1)

where $\mathbf{y}_{m_{i}}$ denotes the one-hot logits for modality $m$ at
spatial position $i$.

The confidence map $C_{m}$ is used to re-weight the features. Color and
thermal features receive different weights, based on the confidence of the
specialized color and thermal decoders. Weighting is performed by point-wise
Hadamard product, and resulting weighted color and thermal features are
concatenated together:

$\displaystyle\mathbf{f}^{\prime}_{c}=C_{c}\odot\mathbf{f}_{c}$ (2)
$\displaystyle\mathbf{f}^{\prime}_{t}=C_{t}\odot\mathbf{f}_{t}$ (3)
$\displaystyle\mathbf{f}^{\prime}_{ct}=\mathbf{f}^{\prime}_{c}\oplus\mathbf{f}^{\prime}_{t}.$
(4)

where $\odot$ denotes the Hadamard product and $\oplus$ denotes concatenation.

###  3.2 Matching correlation weighted fusion

Another possible issue of directly fusing color and thermal features is the
spatial misalignment between both images. Figure 3 illustrates a sample image
from MF dataset [1] where feature disagreement can be caused due to image
misalignment.

As it may be hard to achieve perfect pixel by pixel alignment between the
different imaging modalities, we propose to evaluate the alignment through a
matching layer.

Figure 3: Example of misalignment between color and thermal images. It can be
seen that the label for the pedestrian class (overlayed in yellow) is not
perfectly aligned with the pedestrians on the thermal image. Such misalignment
can cause the RGB and the thermal encoder features to disagree. We propose a
correlation feature weighting to alleviate this issue.

Inspired by the work of [23] on deep image alignment, we propose to employ a
correlation layer which accounts to evaluate the matching between features
coming from the two different encoders.

The correlation layer computes pair-wise feature similarities between the
color and thermal decoder and the obtained scores are normalized such that
weak matched features are suppressed by down weighting.

In order to deal with strong appearance differences between color and thermal
features, we estimate semantic matching between the last specialized decoder,
such that correlation is invariant to appearance changes.

Formally, given $\mathbf{y}_{c}=d_{c}(\mathbf{f}_{c})$ corresponding to the
color decoder predictions and $\mathbf{y}_{t}=d_{t}(\mathbf{f}_{t})$
corresponding to the thermal decoder predictions, the correlation weight
matrix $M_{ct}\in\mathbb{R}^{h\times w}$ is obtained by computing modality-
wise prediction correlations (normalized scalar product of a pair of
predictions), followed by a channel compressing convolutional block:

$\displaystyle
M_{ct}=c(\|\sigma(\widebar{\mathbf{y}_{t}}^{T}\widebar{\mathbf{y}_{c}})\|_{2})$
(5)

where $\sigma$ denotes ReLU activation, $\widebar{.}$ denotes spatially
flattening ($y_{m}\in\mathbb{R}^{k\times h\times w}$,
$\widebar{y_{m}}\in\mathbb{R}^{k\times N},N=h\times w$, $k$ being the number
of predicted classes) , $\|.\|_{2}$ denotes $L_{2}$ vector normalization and
$c$ is a convolutional block performing channel compression ($N\times h\times
w$ input dimensions, $h\times w$ output dimensions). Note that $c$ acts in
analogy to a feature regressor estimating parametric alignment [23], but here
we instead estimate alignment-aware spatial weights.

Then, the concatenated confidence-weighted features obtained by Eq. 4 are re-
weighted by the correlation map:

$\mathbf{f}^{\prime\prime}_{ct}=M_{ct}\odot\mathbf{f}^{\prime}_{ct}.$ (6)

Finally, the obtained feature map $\mathbf{f}^{\prime\prime}_{ct}$ is used as
input to the shared decoder head to produce the final semantic segmentation.

##  4 Experiments

In this section we perform semantic segmentation experiments that shows the
advantages of the proposed method.

###  4.1 Dataset

We perform semantic segmentation experiments on the MF dataset released by the
authors of the method MFNet [1]. To the best of our knowledge, this is the
only public dataset with paired color and thermal images and dense annotated
masks in the context of urban street scenes. The dataset has eight hand-
labeled object classes (car, person, bike, curve, car stop, guardrail, color
cone, bump) and one unlabelled background class. This dataset contains 1569
pairs of RGB and thermal images, in which 820 taken at daytime and 749 taken
at nighttime. We follow the dataset splitting scheme proposed in [1], with the
training set consisting of 784 pairs of images. The validation set consists of
392 pairs of images, and the test set contains 392 image pairs.

###  4.2 Quantitative comparison

In this section, we compare our model with previous works on color-thermal
semantic segmentation: FRRN [24], BiSeNet [25], DFN [26], HRNetV2 [27], MFNet
[1], FuseNet [17], DACNN [28], RTFNet(50-152) [18], FuseSeg-161 [19] and
FEANet [20].

For comparison with previous work, we compute the mean IoU (Intersection Over
Union) and the mean accuracy. The IoU (also called Jaccard index) is arguably
the most adapted metric to evaluate semantic segmentation, it measures the
overlap between predicted and ground truth mask, while the accuracy simply
measures the total percentage of correct predictions.

In Table 1, it can be seen that our model improves mean IoU over all classes
in the MF dataset. It improves IoU on three classes and notably on the person
class, which is arguably one of the most important category for driving
perception.

It should be noted that similarly to other datasets for driving perception, MF
dataset is extremely class unbalanced. This implies that the IoU metric is a
better indicator of segmentation performance than accuracy for this dataset.

Table 1: Comparison with previous work on the MF test set for Acc (%) and IoU (%). We denote 3-channel RGB and 4-channel RGB-thermal data respectively as 3c and 4c. Note that mAcc and mIoU are calculated as a mean over all classes, including unlabelled. Bold font highlights the highest score in each column, underlined font highlights the second highest score. We report an improvement of 2% in overall mIoU compared to previous work. Methods | Car | Person | Bike | Curve | Carstop | Guardrail | Cone | Bump | mAcc | mIoU
---|---|---|---|---|---|---|---|---|---|---
Acc |  IoU | Acc |  IoU | Acc |  IoU | Acc |  IoU | Acc |  IoU | Acc |  IoU | Acc |  IoU | Acc |  IoU
FRRN-4c [24] | 81.9 | 74.7 | 66.2 | 60.8 | 62.8 | 50.3 | 41.2 | 35.0 | 12.5 | 11.5 | 0.0 | 0.0 | 37.2 | 34.0 | 35.2 | 34.6 | 48.5 | 44.2
FRRN-3c [24] | 80.0 | 71.2 | 53.0 | 46.1 | 65.1 | 53.0 | 34.0 | 27.1 | 21.6 | 19.1 | 0.0 | 0.0 | 34.7 | 32.5 | 36.2 | 30.5 | 47.1 | 41.8
BiSeNet-4c [25] | 89.7 | 84.1 | 72.0 | 63.2 | 74.1 | 60.1 | 45.1 | 36.7 | 34.2 | 25.3 | 18.2 | 5.0 | 47.4 | 42.2 | 39.8 | 35.9 | 57.7 | 50.0
BiSeNet-3c [25] | 90.0 | 84.5 | 65.0 | 54.3 | 75.0 | 61.4 | 32.1 | 25.7 | 32.3 | 26.2 | 3.2 | 0.9 | 49.6 | 43.3 | 48.1 | 40.5 | 54.9 | 48.2
DFN-4c [26] | 90.0 | 84.4 | 73.2 | 65.0 | 75.5 | 60.9 | 54.0 | 40.4 | 38.9 | 25.7 | 10.2 | 2.7 | 48.3 | 42.5 | 55.8 | 47.4 | 60.5 | 52.0
DFN-3c [26] | 90.7 | 81.4 | 67.7 | 52.8 | 71.5 | 57.5 | 49.2 | 34.9 | 35.1 | 23.8 | 4.1 | 1.4 | 44.2 | 31.0 | 54.6 | 47.5 | 57.3 | 47.5
HRNet2-4c [27] | 92.8 | 87.6 | 79.3 | 71.0 | 78.3 | 63.4 | 59.8 | 42.5 | 25.7 | 19.1 | 18.8 | 0.0 | 56.5 | 49.8 | 63.5 | 44.5 | 63.7 | 53.2
HRNet2-3c [27] | 92.2 | 86.6 | 73.1 | 59.8 | 74.9 | 61.3 | 47.0 | 33.2 | 23.8 | 28.7 | 7.3 | 0.0 | 54.6 | 47.2 | 61.5 | 46.2 | 60.9 | 51.3
MFNet [1] | 77.2 | 65.9 | 67.0 | 58.9 | 53.9 | 42.9 | 36.2 | 29.9 | 19.1 | 9.9 | 0.1 | 8.5 | 30.3 | 25.2 | 30.0 | 27.7 | 45.1 | 39.7
FuseNet [17] | 81.0 | 75.6 | 75.2 | 66.3 | 64.5 | 51.9 | 51.0 | 37.8 | 28.7 | 15.0 | 0.0 | 0.0 | 31.1 | 21.4 | 51.9 | 45.0 | 52.4 | 45.6
DACNN [28] | 85.2 | 77.0 | 61.7 | 53.4 | 76.0 | 56.5 | 40.2 | 30.9 | 9.9 | 29.3 | 22.8 | 6.4 | 32.9 | 30.1 | 36.5 | 32.3 | 55.1 | 46.1
RTFNet-50 [18] | 91.3 | 86.3 | 78.2 | 67.8 | 71.5 | 58.2 | 69.8 | 43.7 | 32.1 | 24.3 | 13.4 | 3.6 | 40.4 | 26.0 | 73.5 | 57.2 | 62.2 | 51.7
RTFNet-152 [18] | 93.0 | 87.4 | 79.3 | 70.3 | 76.8 | 62.7 | 60.7 | 45.3 | 38.5 | 29.8 | 0.0 | 0.0 | 45.5 | 29.1 | 74.7 | 55.7 | 63.1 | 53.2
FuseSeg-161 [19] | 93.1 | 87.9 | 81.4 | 71.7 | 78.5 | 64.6 | 68.4 | 44.8 | 29.1 | 22.7 | 63.7 | 6.4 | 55.8 | 46.9 | 66.4 | 47.9 | 70.6 | 54.5
FEANet [20] | 93.3 | 87.8 | 82.7 | 71.1 | 76.7 | 61.1 | 65.5 | 46.5 | 26.6 | 22.1 | 70.8 | 6.6 | 66.6 | 55.3 | 77.3 | 48.9 | 73.2 | 55.3
Ours | 91.7 | 86.7 | 81.3 | 72.2 | 76.6 | 62.5 | 58.9 | 46.7 | 36.2 | 28.0 | 35.2 | 5.1 | 56.9 | 50.7 | 74.8 | 65.8 | 67.9 | 57.3

###  4.3 Qualitative comparison

In Fig. 4, we show a number of semantic segmentation results obtained with our
method. It can be seen that segmentation of important details are improved
compared to previous work. In the zoomed in parts of columns 4 and 5, it can
be seen that our method segments better the pedestrian legs. In the last two
columns, it can also be seen that our method is the only one able to retrieve
the person and the bike objects, which are hardly visible in the color image
due the small size of the object and a saturated highlight.

Figure 4: Qualitative results where the robustness of our method can be
appreciated for daytime and nighttime scenes of MF test set split. In the last
two columns (original and zoomed in images), it can be seen that our method is
the only segmenting the person and the bike objects.

###  4.4 Ablation Study

In Table 2, we consider a number of variants of our model, all of them also
based on the DeepLabv3+ backbone. We evaluate the individual contributions of
the confidence weighting, the concatenated decoder-level fusion and
correlation weighting.

We also evaluate the model performance in different lighting conditions, only
day, and only night. It can be seen that our proposed model obtains the
highest mIoU in all lighting conditions, compared to its simpler variants.

###  4.5 Learned features visualization

In Fig. 5, we visualize the mean output feature responses of our proposed
model before and after semantic segmentation training on RGB-T data. It can be
seen that for the given image sample, the shallow second encoder layer gives
somewhat similar responses before and after training. On the other hand, the
deeper feature response at deeper layer seems to vary more in its response
after training, by learning to spot at the two pedestrians.

Figure 5: Illustration of features learned by color encoder (left two columns) and thermal encoder (right two columns) at encoder layer 2 and layer 4 (second and third rows) before (pretrained on Imagenet) and after training for semantic segmentation. Note that at the deeper level the feature responses seem less invariant to the imaging modality after training.  Table 2: Ablation study: comparison of mAcc ($\%$) and mIoU ($\%$) on the day-only, night-only and regular test set of MF dataset for different model variants. For a fair comparison, all of the variants are trained with the same optimizer parameters and are based on DeepLabv3+ with Resnet-101 pretrained backbone. It can be noted the advantage of our approach - simpler model variants based on DeepLabv3+ result in lower mIoUs for all lighting conditions. Variants | Daytime | Nighttime | Both
---|---|---|---
mAcc |  | mIoU | mAcc |  | mIoU |  mAcc |  | mIoU
Only RGB | 54.3 |  | 44.4 | 56.5 |  | 48.3 | 61.2 |  | 50.7
Only thermal | 48.2 |  | 39.0 | 64.0 |  | 51.9 | 65.6 |  | 50.1
Input fusion - stacked RGBT | 48.7 |  | 42.8 | 56.0 |  | 51.0 | 57.1 |  | 51.1
Unweighted fusion | 60.7 |  | 49.0 | 61.4 |  | 53.6 | 64.9 |  | 54.7
Only-confidence weighting | 63.3 |  | 49.6 | 61.2 |  | 54.1 | 65.4 |  | 55.2
Ours | 58.8 |  | 50.1 | 64.1 |  | 55.7 | 67.9 |  | 57.3

###  4.6 Implementation and setup details

Our method is implemented using the pytorch library [29] and the torch
segmentation models library [30]. For all experiences with our method, we use
ResNet-101 model [22] pretrained on Imagenet as backbone. We perform
concatenation and confidence weighting with features from layer 2 and layer 4
of DeepLabv3+ encoder. Bilinear interpolation is used to downsample or
upsample confidence map $C_{m}$ and matching map $M_{ct}$ whenever needed to
fit the encoder feature spatial resolutions from layer 2 and layer 4.

For a fair comparison, we perform similar training setup as compared to
previous methods. We adopt mostly the same hyperparameters as employed by the
authors of the RTFNet method [18]. We use random cropping and random flipping
as image augmentation during training.

For optimization, we use SGD with momentum set to 0.9, weight decay set to
0.0005, initial learning rate set to 0.01 and a maximum of 50 training epochs.
We employ exponential decaying learning rate scheduler with gamma set to 0.95.
As commonly done in the field, we minimize the cross entropy loss between the
predicted and the target labels.

##  5 Conclusion

In this paper we have presented a new feature fusion approach for the task of
semantic segmentation of paired color RGB and thermal LWIR images. We proposed
DooDLeNet, a double DeepLab net which fuses multimodal images taking into
account the feature disagreements and feature confidences. We have shown that
this approach improves semantic segmentation performance compared to previous
works, in particular for the correct segmentation of pedestrians. Future work
may be focused on evaluating different model backbones and to refine further
segmentation details.

##  6 Acknowledgements

This research is part of Heliaus project111European Union’s Heliaus project
website: https://heliaus.eu/, which has received funding from the ECSEL Joint
Undertaking (JU) under grant agreement No 826131. The JU receives support from
the European Union’s Horizon 2020 research and innovation programme and
France, Germany, Ireland, Italy222Disclaimer: Funded by the European Union.
Views and opinions expressed are however those of the authors only and do not
necessarily reflect those of the European Union or ECSEL JU. Neither the
European Union nor the granting authority can be held responsible for them..
Completed by local funding from the French region Auvergne Rhône Alpes.

## References

  * [1] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada, “MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes,” IEEE International Conference on Intelligent Robots and Systems, vol. 2017-Septe, pp. 5108–5115, 2017.
  * [2] F. Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, V. Madhavan, and T. Darrell, “BDD100K: A diverse driving dataset for heterogeneous multitask learning,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 2633–2642, Computer Vision Foundation / IEEE, 2020.
  * [3] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [4] E. Shelhamer, J. Long, and T. Darrell, “Fully convolutional networks for semantic segmentation,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 4, pp. 640–651, 2017.
  * [5] H. Noh, S. Hong, and B. Han, “Learning deconvolution network for semantic segmentation,” in Proceedings of the IEEE international conference on computer vision, pp. 1520–1528, 2015.
  * [6] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation.,” CoRR, vol. abs/1511.00561, 2015.
  * [7] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015\.
  * [8] C. Farabet, C. Couprie, L. Najman, and Y. LeCun, “Learning hierarchical features for scene labeling,” IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 8, pp. 1915–1929, 2012.
  * [9] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881–2890, 2017.
  * [10] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in Proceedings of the European Conference on Computer Vision (ECCV), September 2018\.
  * [11] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 4, pp. 834–848, 2017.
  * [12] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, “Dual attention network for scene segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3146–3154, 2019.
  * [13] C. Yu, J. Wang, C. Gao, G. Yu, C. Shen, and N. Sang, “Context prior for scene segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12416–12425, 2020.
  * [14] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [15] S. Hwang, J. Park, N. Kim, Y. Choi, and I. S. Kweon, “Multispectral pedestrian detection: Benchmark dataset and baselines,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [16] R. Abbott, N. M. Robertson, J. M. del Rincón, and B. Connor, “Unsupervised object detection via LWIR/RGB translation,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 407–415, IEEE, 2020\.
  * [17] C. Hazirbas, L. Ma, C. Domokos, and D. Cremers, “Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture,” in Asian conference on computer vision, pp. 213–228, Springer, 2016.
  * [18] Y. Sun, W. Zuo, and M. Liu, “RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes,” IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2576–2583, 2019.
  * [19] Y. Sun, W. Zuo, P. Yun, H. Wang, and M. Liu, “Fuseseg: Semantic segmentation of urban scenes based on rgb and thermal data fusion,” IEEE Transactions on Automation Science and Engineering, vol. PP, pp. 1–12, 06 2020\.
  * [20] F. Deng, H. Feng, M. Liang, H. Wang, Y. Yang, Y. Gao, J. Chen, J. Hu, X. Guo, and T. L. Lam, “Feanet: Feature-enhanced attention network for rgb-thermal real-time semantic segmentation,” CoRR, vol. abs/2110.08988, 2021.
  * [21] L. Zhang, X. Zhu, X. Chen, X. Yang, Z. Lei, and Z. Liu, “Weakly aligned cross-modal learning for multispectral pedestrian detection,” in 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pp. 5126–5136, IEEE, 2019.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770–778, IEEE Computer Society, 2016.
  * [23] I. Rocco, R. Arandjelovic, and J. Sivic, “Convolutional neural network architecture for geometric matching,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 11, pp. 2553–2567, 2019.
  * [24] T. Pohlen, A. Hermans, M. Mathias, and B. Leibe, “Full-resolution residual networks for semantic segmentation in street scenes,” CoRR, vol. abs/1611.08323, 2016.
  * [25] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, “Bisenet: Bilateral segmentation network for real-time semantic segmentation,” CoRR, vol. abs/1808.00897, 2018.
  * [26] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, “Learning a discriminative feature network for semantic segmentation,” CoRR, vol. abs/1804.09337, 2018.
  * [27] K. Sun, Y. Zhao, B. Jiang, T. Cheng, B. Xiao, D. Liu, Y. Mu, X. Wang, W. Liu, and J. Wang, “High-resolution representations for labeling pixels and regions,” CoRR, vol. abs/1904.04514, 2019.
  * [28] W. Wang and U. Neumann, “Depth-aware CNN for RGB-D segmentation,” CoRR, vol. abs/1803.06791, 2018.
  * [29] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32, pp. 8024–8035, Curran Associates, Inc., 2019.
  * [30] P. Yakubovskiy, “Segmentation models.” https://github.com/qubvel/segmentation_models, 2019.


[affiliation=1]Tzu-QuanLin [affiliation=1]Hung-yiLee [affiliation=2]HaoTang

# DAISY: Data Adaptive Self-Supervised Early Exit
for Speech Representation Models

###### Abstract

Self-supervised speech models have shown to be useful for various tasks, but
their large size limits the use in devices with low computing power and
memory. In this work, we explore early exit, an approach for reducing latency
by exiting the forward process of a network early. Most approaches of early
exit need a separate early exit model for each task, with some even requiring
fine-tuning of the entire pretrained model. We introduce Data Adaptive Self-
Supervised Early Exit (DAISY), an approach that decides when to exit based on
the self-supervised loss, eliminating the need for multiple round of training
and fine-tuning. DAISY matches the performance of HuBERT on the MiniSUPERB
benchmark, but with much faster inference times. Our analysis on the
adaptivity of DAISY shows that the model exits early (using fewer layers) on
clean data while exits late (using more layers) on noisy data, dynamically
adjusting the computational cost of inference based on the noise level of each
sample.

###### keywords:

Self-Supervised Learning, Model Compression, Early Exit

##  1 Introduction

Self-supervised speech models [1, 2, 3, 4, 5, 6] have demonstrated impressive
capabilities in feature extraction. The feature extracted by speech SSL models
could generalize well across various downstream datasets [7, 8, 9]. However,
to effectively learn contextualized information from unsupervised data, these
models are typically large [10], leading to substantial computational costs
during inference. This significantly restricts the use of speech SSL models on
low-resource devices.

To solve this issue, many studies have focused on compressing self-supervised
speech models. Prior work has explored knowledge distillation [11, 12, 13] to
train a smaller student model to mimic the behavior of a larger teacher model.
Some use either unstructured [14] or structured [15, 16] pruning to remove
redundant parameters in self-supervised models, and some apply quantization
methods [17] to store the models at lower bitwidths. Early exit [18], unlike
the other methods mentioned, does not reduce the number of parameters in the
model, but instead aims to stop the forward process to make predictions in
earlier layers, thereby reducing inference latency.

In this work, we will focus on the early exit approach. Most studies on early
exit for self-supervised models are for processing texts [19, 20, 21, 22, 23,
24], with only a few applied to speech [25]. Typically, these methods involve
attaching early exit branches to each layer and fine-tuning these branches on
specific downstream dataset. During inference, the decision to exit at a
particular layer is made by assessing the entropy or confidence of the branch
[18]. Existing approaches require training early exit branches for every
downstream tasks, and many require fine-tuning the self-supervised models for
specific downstream tasks as well, which significantly increases the
computational cost. Moreover, the effectiveness of early exit has not been
shown other than ASR [25].

In this work, inspired by the finding that the self-supervised loss aligns
well with downstream performance [26], we propose Data Adaptive Self-
Supervised Early Exit (DAISY), a novel and effective method using the entropy
of the self-supervised loss to make decisions for early exit. Our approach can
be divided into three stages. In the first stage, we attach an early exit
branch to each hidden layer and optimize each branch with a self-supervised
objective (e.g., predicting quantized representation as in HuBERT [4]). In the
second stage, we fix the early exit branches, and train classifiers for
downstream tasks using the weighted sum of features following SUPERB [7].
Different from typical early exit methods that only perform early exit at the
inference phase, we incorporate early exits during training of the downstream
classifier. In the third stage, all models are frozen for inference with early
exit. Note that the self-supervised model is frozen during all three stages.
Compared to previous early exit methods, our approach requires training early
exit branches only once in the first stage, and the early exit branches are
then used for _all_ downstream tasks. Our approach does not require fine-
tuning the self-supervised model either, significantly more efficient in
training than other approaches.

Since our approach no longer depends on the downstream tasks, when and where a
branch decides to exit solely depends on properties of the input data. In
other words, regardless of the task, once the input data is given, the model
will exit at the same layer. What decides when to exit is not the task, but
the characteristics of the data set on which the task is performed. This
prompts us to study the data adaptivity of our approach. Based on our analysis
with different noise levels, models tend to forward to deeper layers on data
with low signal-to-noise ratio. This allows it to achieve a favorable
performance and speed-up trade-off on datasets with varying degrees of noise.

##  2 Related Work

The concept of early exit was first proposed by BranchyNet [18], where they
proposed adding early exit branches to the intermediate layers of the model.
During inference, it is possible to exit early via these branches when labels
can be inferred with high confidence.

DeeBERT [19] introduced the concept of early exit to self-supervised models.
Similar to BranchyNet, DeeBERT attaches branches to intermediate layers of
BERT and fine-tunes these branches for specific downstream tasks. During
inference, it determines whether to perform an early exit by assessing the
confidence of the predictions. CALM [23] proposed to use token-level early
exit to accelerate the process of language model generation. They validate the
feasibility of early exit in generation tasks. However, they still need to
train the early exit branches for different downstream tasks.

HuBERT-EE [25] utilizes early exit to accelerate the inference process of
HuBERT. However, they only validate their approach on ASR, and similar to all
previous early exit methods, their method requires training branches for
different downstream tasks. In addition, they need to fine-tune the entire
HuBERT model first before training early exit branches.

##  3 Methodology

Figure 1: Three stages of DAISY: (a) the training of early exit branches, (b)
the training of downstream models, and (c) early exit at inference time. EEB
is used to denote early exit branches, where the linear classifier and entropy
computation happen. Model parameters are frozen when the boxes are in gray;
model parameters are being trained when the boxes are in orange.

In essence, early exit amounts to learning a function (a branch) $f^{k}$ for
layer $k$ such that, when taking the $k$-th hidden layer $h^{k}$ as input, if
$f^{k}(h^{k})=1$, the forward process stops and the model exits early;
otherwise, if $f^{k}(h^{k})=0$, the forward process continues. In this
section, we will introduce our approach, DAISY, and discuss how each early
exit branch is trained. There are three distinct phases of DAISY, and they are
summarized in Figure 1.

###  3.1 Training early exit branches

Instead of training early exit branches with downstream tasks, we propose to
train them with a self-supervised loss. We choose a simplified variant of the
HuBERT loss, measuring the cross entropy between a quantized frame and the
output of a linear classifier. Formally, for every layer $k$, we train a
linear classifier $W^{k}$ with

$\displaystyle\frac{1}{T}\sum_{t=1}^{T}CE(W^{k}h^{k}_{t},y_{t}),$ (1)

where $h^{k}_{t}$ is the hidden vector at layer $k$ and time $t$, and $y_{t}$
is the target at time $t$. The target $y_{t}$ should ideally be the targets
used for training HuBERT, but the targets are never released. We instead
create targets by running HuBERT and taking the argmax from the last linear
layer. This variant of HuBERT loss have been shown to be as effective as the
original HuBERT loss [26] and is computationally more efficient [27].

After training the early exit branches, given a data sample, we compute the
entropy of the $k$-th branch

$\displaystyle
E^{k}=\frac{1}{T}\sum_{t=1}^{T}\sum_{c=1}^{C}-p^{k}_{t,c}ln(p^{k}_{t,c}),$ (2)

where $p^{k}_{tc}$ is the probability of the class $c$ at layer $k$ and time
$t$ obtained with the linear layer $W^{k}$ and a softmax. The entropy is by no
means the perfect measure of uncertainty or confidence (see [28] and the
citations therein), but it is simple to implement and, as we will see, works
surprising well. The final exit decision is based on a simple threshold $\tau$
on the entropy. Specifically, $f^{k}(h^{k})$ is 1 when $E^{k}<\tau$, and is
otherwise 0.

###  3.2 Training downstream tasks

To give a sense of how the entropy values vary across layers and across
datasets, we show the average entropy of each layer in HuBERT BASE on four
datasets from MiniSUPERB [29] in Figure 2. We observe that the entropy is
smaller for deeper layers, and the entropy as a function of the layer is
surpringly linear.111The linear relationship between the entropy and the layer
is itself an interesting finding and warrants further study.

The linear relationship between the entropy and the layers prompts us to
design the following heuristic for choosing the threshold. Given a dataset, we
compute the average entropy of each layer and measure the maximum and minimum
average entropy $E_{max}$ and $E_{min}$, respectively. The threshold $\tau$ is
decided by a linear scaling of the mean

$\displaystyle\tau$ $\displaystyle=\frac{E_{max}+E_{min}}{2}\rho$ (3)

where $\rho$ is a ratio hyperparameter in $[0,1]$. The ratio $\rho$ is left to
the user to decide; the smaller the $\rho$, the later the model exits and the
more compute is required.

When training downstream models, given a threshold $\tau$, the model decides
whether to exit at layer $k$ by evaluating $E^{k}<\tau$ for each data sample.
If early exit happens at layer $\hat{k}$, we apply Layer Normalization [30] on
all hidden vectors below the $\hat{k}$-th layer,

$\displaystyle\bar{h}^{1:\hat{k}}=LayerNorm(h^{1:\hat{k}}).$ (4)

Following SUPERB [7], the features to the downstream classifier is the
weighted sum of the hidden vectors $\sum_{k=1}^{\hat{k}}w_{k}\bar{h^{k}}$. The
weights and the subsequent linear classifiers are trained for the downstream
task.

Figure 2: The average entropy of each early exit branch on four datasets of
MiniSUPERB [29].

###  3.3 Exit strategies at inference time

Though the hyperparameter $\rho$ needs to be fixed when training downstream
models, it still can be changed during inference depending on how much compute
the user is willing to spend, much like the spirit of anytime inference [31].
In fact, any additional constraints can be added at inference time, and below
we explore three spans, constraining where the model can exit. If during
inference no layers within the span decides to exit early, we simply force the
model to exit at the deepest layer within the defined span.

#### mean

This constraint forces the model to exit at a layer that is the average number
of layers used during training. Formally, if the average exit layer is $\mu$
during downstream training, we only allow the model to exit at layer $k$ if
$\lfloor\mu\rfloor\leq k\leq\lceil\mu\rceil$.

#### threshold

This constraint only allows the model to exit at layers that have sufficiently
been used as exit points during downstream training. Formally, suppose the
exit happens at the $k$-th layer $r_{k}$ of the time during downstream
training. We only allow the model to exit at layer $k$, say, when
$r_{k}>15\%$.

#### min-max

This constraint imposes the minimum and maximum number of layers, and the two
are determined based on the minimum and maximum used during downstream
training. Formally, suppose the minimum exit layer is $k_{\text{min}}$ and the
maximum is $k_{\text{max}}$ during downstream training. We only allow the
model to exit at layer $k$ if $k_{\text{min}}\leq k\leq k_{\text{max}}$.

##  4 Experiments

We train the early exit branches on the full Librispeech 960 hours [32]. We
fix all the other parameters and train the branches with a batch size of 32
and a learning rate of 5e-5. It takes 88000 steps to converge, about 10 hours
on a single 32GB V100 GPU. We do not apply dropout and layer drop [33] during
training. All the other hyperparameters remain the same as HuBERT. We evaluate
our models on MiniSUPERB [29], a lightweight benchmark that efficiently
evaluate self-supervised speech models. There are four downstream tasks in
MiniSUPERB, including automatic speech recognition (ASR), speaker
identification (SID), speech enhancement (SE), and speech separation (SS). We
use learning rate of 1e-4, 1e-1, 1e-4, and 1e-4 respectively for ASR, SID, SE,
and SS.

###  4.1 Downstream performance and speed-up

The downstream performance and the amount of forward time saved on MiniSUPERB
[29] are shown in Table 1. In general, when using a smaller ratio $\rho$
(0.7), DAISY can achieve close or even better performance to HuBERT BASE on
all four downstream tasks, while saving inference time. To highlight, on SID,
it can improve accuracy by 1.21% and save 23.36% of forwarding time. On SE, it
can improve PESQ by 0.02 and save 20.51% of forwarding time. When using a
larger ratio $\rho$ (1.0), DAISY is able to save even more while still
maintaining performance on SID, SE, and SS. To highlight, on SID, it can save
31.5% of the forward time with less than 1% accuracy degradation. On SE, it
can save 29.56% of forwarding time and improve PESQ by 0.02. On SS, it can
save 24.75% of forwarding time and significantly improve SI-SDRi by 0.19.
Compared to the other three tasks, DAISY finds it more challenging to save a
substantial amount of forwarding time on ASR without sacrificing performance.
This might be related to the distribution of layerwise information in self-
supervised speech models, where phonetic information is predominantly stored
in relatively deep layers [34].

Comparing the three exit strategies at inference time, both mean and threshold
methods can achieve a good trade-off between performance and forward time. We
find that the min-max approach exhibits worse performance in a few cases.

As Table 1 shows, users can decide the value of $\rho$ based on the amount of
computational resources they have. When resources are limited, one can choose
a larger $\rho$. In Figure 3, we further demonstrate the results of DAISY
compared to exiting early at the 6th layer. We observe that DAISY is capable
of achieving better performance than early exit at a fixed layer. This shows
the advantage of DAISY to dynamically determine the early exit layer. In
addition, using different $\rho$'s and different exit strategy provides a
trade-off at inference time.

Table 1: Downstream performance and the percentage of saving forward time of
DAISY on four different downstream tasks of MiniSUPERB [29]. ASR denotes
automatic speech recognition, SID denotes speaker identification, SE denotes
speech enhancement, and SS denotes speech separation. *The performance of
HuBERT base 12 layer is copied from SUPERB benchmark [7]

| ASR | SID | SE | SS
---|---|---|---|---
| WER $\downarrow$ | Time Saved $\uparrow$ | ACC $\uparrow$ | Time Saved $\uparrow$ | PESQ / STOI $\uparrow$ | Time Saved $\uparrow$ | SI-SDRi $\uparrow$ | Time Saved $\uparrow$
DAISY | $\rho=1.0$ | mean | 15.45% | 21.63% | 79.55% | 31.51% | 2.60 / 94.0 | 27.23% | 9.54 | 22.78%
threshold | 15.45% | 31.54% | 80.53% | 31.50% | 2.60 / 94.0 | 29.56% | 9.55 | 24.75%
min-max | 15.43% | 18.99% | 72.36% | 28.64% | 2.58 / 93.9 | 29.22% | 9.52 | 20.93%
$\rho=0.7$ | mean | 6.96% | 4.98% | 82.63% | 19.42% | 2.60 / 94.0 | 20.51% | 9.32 | 2.23%
threshold | 6.98% | 7.71% | 82.63% | 23.36% | 2.59 / 94.0 | 25.03% | 9.32 | 2.50%
min-max | 6.96% | 4.74% | 82.26% | 15.19% | 2.59 / 94.0 | 23.72% | 9.32 | 1.25%
*HuBERT Base 12 layer [4] | 6.42% | 0.00% | 81.42% | 0.00% | 2.58 / 93.9 | 0.00% | 9.36 | 0.00%

Figure 3: The comparison of DAISY and the early exit baseline at a fixed
layer. The colored dots represent DAISY, while the black crosses represent
early exit at the 6th layer. For each downstream task, we present the results
of three different $\rho$ values (0.7, 0.76, 1.0) combined with the three exit
strategy at inference time, resulting in a total of 9 dots.  Figure 4: Violin
plot of the early exit probability at each layer when applying different
levels of MUSAN noise [35] to the Librispeech test-clean set. The horizontal
line represents the maximum, average, and minimum of the early exit layer,
respectively.  Figure 5: The word error rate of DAISY on samples with
different level of noises. HuBERT-first-6L represents the results of
statically early exiting at 6th layer.

###  4.2 Noise adaptivity of DAISY

From the results of the previous section, we can already see that DAISY is
capable of dynamically deciding what samples require more layers to process
and what samples do not. However, it is unclear what properties of the data
drive the early exit decision. There might be many properties that determines
when to exit. In this section, we only focus on the noise level of the
recordings, as the noise level can be controlled by adding noise to clean
speech. Specifically, we add different levels of noise to the Librispeech test
clean dataset, including SNR values of 10, 5, and 0. We then measure the early
exit results for each level of noise. The noise recordings were randomly
sampled from the MUSAN dataset [35]. The results are presented in Figure 4.
Overall, DAISY tends to use more layers as the SNR decreases. Our findings is
consistent with Ravuri et al. [36], where they found that the entropy of self-
supervised speech models correlate well with their prediction (in their case,
MOS scores). In addition to general background noise, we also experiment with
adding music and speech, and the results are consistent with adding background
noise.

###  4.3 Applications of noise adaptativity

The above analysis provides some evidence, supporting DAISY's ability to adapt
to noise. In particular, DAISY would have an advantage when the noise level
has a large variance. Unfortunately but also not surprisingly, most of the
publicly available datasets have a relatively homogeneous noise level among
data point. To test the adaptivity of DAISY, we create a dataset with a range
of noise levels. We add MUSAN noise recordings to LibriSpeech 100 hours, dev
clean, and test clean subsets, in which 40% remain clean, 30% of SNR 10, 20%
of SNR 5, and 10% of SNR 0. We then perform ASR on this dataset. The result is
shown in Figure 5. Comparing to early exit at a fixed layer (the 6th), our
approach performs better across all noise levels, and the gap becomes more
pronounced when the level of noise increases. The average WER across all noise
levels is 20.67, 23.44, 23.32, and 26.1 for DAISY-mean, DAISY-threshold,
DAISY-min-max, and HuBERT-first-6L, respectively. This result clearly shows
that DAISY is able to identify the noisy samples, and, based on the noise
level, allocate the appropriate amount of compute for this task.

Since most of the time we cannot precisely know how noisy a dataset is, and
sometimes a dataset may contain data with varying degrees of noise, it becomes
challenging to predetermine the appropriate layer for early exit apriori. In
such cases, DAISY can dynamically adjust the amount of layer needed based on
how noisy each sample is, thereby achieving better performance.

##  5 Conclusion

In this work, we propose DAISY, a novel and effective approach for early exit
based on a self-supervised loss. We use the entropy of the prediction to
decide whether to exit early. Unlike previous early exit approaches, DAISY
does not require training early exit branches for different downstream tasks
and does not require any fine-tuning of the self-supervised model, making the
training process much more efficient. On four downstream tasks of MiniSUPERB,
DAISY achieves comparable or even better performance than HuBERT, while saving
inference time. Lastly, we analyze the noise adaptivity property of DAISY,
showing that noisy samples tend to require more layers to process. We then
show that in a setting where the range of noise level is large, DAISY performs
better comparing to early exit at a fixed layer, even though the amortized
compute is the same. The early exit approach provides a different avenue to
model compression at achieving speed-up at inference time.

##  6 Acknowledgement

We thank the National Center for High-performance Computing (NCHC) of the
National Applied Research Laboratories (NARLabs) in Taiwan for providing
computational and storage resources.

## References

  * [1] Y.-A. Chung, W.-N. Hsu, H. Tang, and J. Glass, ``An unsupervised autoregressive model for speech representation learning,'' in _Interspeech_ , 2019.
  * [2] A. T. Liu, S.-w. Yang, P.-H. Chi, P.-c. Hsu, and H.-y. Lee, ``Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders,'' in _ICASSP_. IEEE, 2020.
  * [3] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, ``wav2vec 2.0: A framework for self-supervised learning of speech representations,'' in _Advances in Neural Information Processing Systems_ , 2020.
  * [4] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, ``HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,'' _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2021.
  * [5] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , ``WavLM: Large-scale self-supervised pre-training for full stack speech processing,'' _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [6] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, ``w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,'' in _ASRU_ , 2021.
  * [7] S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin _et al._ , ``SUPERB: Speech processing universal performance benchmark,'' in _Interspeech_ , 2021.
  * [8] H.-S. Tsai, H.-J. Chang, W.-C. Huang, Z. Huang, K. Lakhotia, S.-w. Yang, S. Dong, A. T. Liu, C.-I. J. Lai, J. Shi _et al._ , ``SUPERB-SG: Enhanced speech processing universal performance benchmark for semantic and generative capabilities,'' _ACL_ , 2022.
  * [9] anonymous, ``SUPERB @ SLT 2022: Challenge on generalization and efficiency of self-supervised speech representation learning,'' in _SLT_ , 2022.
  * [10] Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang _et al._ , ``BigSSL: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,'' _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [11] H.-J. Chang, S.-w. Yang, and H.-y. Lee, ``DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit BERT,'' in _ICASSP_ , 2022.
  * [12] R. Wang, Q. Bai, J. Ao, L. Zhou, Z. Xiong, Z. Wei, Y. Zhang, T. Ko, and H. Li, ``LightHuBERT: Lightweight and configurable speech representation learning with once-for-all hidden-unit BERT,'' in _Interspeech_ , 2022.
  * [13] Y. Lee, K. Jang, J. Goo, Y. Jung, and H. Kim, ``FitHuBERT: Going thinner and deeper for knowledge distillation of speech self-supervised learning,'' in _Interspeech_ , 2022.
  * [14] C.-I. J. Lai, Y. Zhang, A. H. Liu, S. Chang, Y.-L. Liao, Y.-S. Chuang, K. Qian, S. Khurana, D. Cox, and J. Glass, ``PARP: Prune, adjust and re-prune for self-supervised speech recognition,'' in _Advances in Neural Information Processing Systems_ , 2021.
  * [15] Y. Peng, K. Kim, F. Wu, P. Sridhar, and S. Watanabe, ``Structured pruning of self-supervised pre-trained models for speech recognition and understanding,'' in _ICASSP_ , 2023.
  * [16] H. Wang, S. Wang, W.-Q. Zhang, H. Suo, and Y. Wan, ``Task-agnostic structured pruning of speech representation models,'' in _Interspeech_ , 2023.
  * [17] Z. Peng, A. Budhkar, I. Tuil, J. Levy, P. Sobhani, R. Cohen, and J. Nassour, ``Shrinking bigfoot: Reducing wav2vec 2.0 footprint,'' in _Workshop on Simple and Efficient Natural Language Processing_ , 2021.
  * [18] S. Teerapittayanon, B. McDanel, and H.-T. Kung, ``BranchyNet: Fast inference via early exiting from deep neural networks,'' in _ICPR_ , 2016.
  * [19] J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin, ``DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference,'' _ACL_ , 2020.
  * [20] W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei, ``BERT loses patience: Fast and robust inference with early exit,'' in _Advances in Neural Information Processing Systems_ , 2020.
  * [21] J. Xin, R. Tang, Y. Yu, and J. Lin, ``BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression,'' in _Proceedings of the 16th conference of the European chapter of the association for computational linguistics: Main Volume_ , 2021.
  * [22] X. Li, Y. Shao, T. Sun, H. Yan, X. Qiu, and X. Huang, ``Accelerating bert inference for sequence labeling via early-exit,'' _arXiv preprint arXiv:2105.13878_ , 2021.
  * [23] T. Schuster, A. Fisch, J. Gupta, M. Dehghani, D. Bahri, V. Tran, Y. Tay, and D. Metzler, ``Confident Adaptive Language Modeling,'' _NeurIPS_ , 2022.
  * [24] B. Hu, Y. Zhu, J. Li, and S. Tang, ``SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference,'' _IJCAI_ , 2023.
  * [25] J. W. Yoon, B. J. Woo, and N. S. Kim, ``HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognition,'' _arXiv preprint arXiv:2204.06328_ , 2022.
  * [26] T.-Q. Lin, T.-H. Yang, C.-Y. Chang, K.-M. Chen, T.-h. Feng, H.-y. Lee, and H. Tang, ``Compressing Transformer-based self-supervised models for speech processing,'' _arXiv preprint arXiv:2211.09949_ , 2022.
  * [27] G. Yang, Z. Ma, Z. Zheng, Y. Song, Z. Niu, and X. Chen, ``Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learning,'' _ASRU_ , 2023.
  * [28] Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek, ``Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift,'' in _Advances in Neural Information Processing Systems_ , 2019.
  * [29] Y.-H. Wang, H.-Y. Chen, K.-W. Chang, W. Hsu, and H.-y. Lee, ``Minisuperb: Lightweight benchmark for self-supervised speech models,'' _ASRU_ , 2023.
  * [30] J. L. Ba, J. R. Kiros, and G. E. Hinton, ``Layer normalization,'' _arXiv preprint arXiv:1607.06450_ , 2016.
  * [31] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, ``Once-for-all: Train one network and specialize it for efficient deployment,'' in _ICLR_ , 2020.
  * [32] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _ICASSP_ , 2015.
  * [33] A. Fan, E. Grave, and A. Joulin, ``Reducing transformer depth on demand with structured dropout,'' _arXiv preprint arXiv:1909.11556_ , 2019.
  * [34] A. Pasad, J.-C. Chou, and K. Livescu, ``Layer-wise analysis of a self-supervised speech representation model,'' in _ASRU_ , 2021.
  * [35] D. Snyder, G. Chen, and D. Povey, ``Musan: A music, speech, and noise corpus,'' _arXiv preprint arXiv:1510.08484_ , 2015.
  * [36] A. Ravuri, E. Cooper, and J. Yamagishi, ``Uncertainty as a predictor: Leveraging self-supervised learning for zero-shot mos prediction,'' _arXiv preprint arXiv:2312.15616_ , 2023.