# A Quantum Kernel Learning Approach to Acoustic Modeling
for Spoken Command Recognition

###### Abstract

We propose a quantum kernel learning (QKL) framework to address the inherent
data sparsity issues often encountered in training large-scare acoustic models
in low-resource scenarios. We project acoustic features based on classical-to-
quantum feature encoding. Different from existing quantum convolution
techniques, we utilize QKL with features in the quantum space to design
kernel-based classifiers. Experimental results on challenging spoken command
recognition tasks for a few low-resource languages, such as Arabic, Georgian,
Chuvash, and Lithuanian, show that the proposed QKL-based hybrid approach
attains good improvements over existing classical and quantum solutions.

##  1 Introduction

Recently, deep neural network [1, 2] (DNN) models have demonstrated a
competitive performance on many speech processing tasks. Nonetheless, training
a large parameterized DNN using as few as 1,000 utterances usually leads to a
poor speech recognition accuracy. Considering that there exist over $8,000$
spoken languages [3] in the world, it is clear that some of those spoken
languages may not provide enough training materials to properly deploy DNN-
based spoken command recognition (SCR) systems. Meanwhile, we have witnessed a
rapid growth of quantum devices, and quantum machine learning (QML) systems
can now “be deployed in practices” thanks to software simulator (e.g.,
Tensorflow-quantum [4]). QML-based algorithms can also be combined with modern
DNN solutions to accomplish feature extraction or reduce computation
complexity. In the present quantum computing era [5] in which $4$ to $200$
qubits are accessible, an integration of quantum based components (e.g.,
feature extractor) into a classical ML module, such as quantum neural networks
[6, 7, 8] (QNNs) and quantum kernel learning [9] (QKL), is a realistic
solution that can be deployed for real-world applications. QNNs use
parameterized quantum circuits (e.g., tensor network) to optimize latent
features [9] in the Hilbert space [10] for minimizing a loss function. QNNs
can be trained under gradient optimization with variational circuits [11] in
an end-to-end fashion to approximate any non-linear [12] function.

Fig. 1: Computing diagram of quantum kernel learning, where quantum kernel
projection occurs on near-term quantum processing units (QPU) or simulators
(e.g., CPU or TPU).

However, such parameterized variational quantum algorithms are often
associated with problems of “barren plateaus” [13, 14], where training loss
takes an extremely long time to update through gradients. QKL [15, 12] instead
provides an alternative training mechanism to use quantum states for quantum
encoding and projection [16], and then forms a kernel to estimate a hyperplane
to separate training data with its kernel alignment. We will discuss these
quantum operations in detail later. Recent theoretical studies [12, 17] have
also proven that QKL requires less trainable parameters to archive the same
performance of QNNs. These findings could potentially tackle the current
challenges in low-resource SCR. In this paper, we propose a quantum kernel
projection 111A tutorial and guidelines for QKL can be found in
https://www.tensorflow.org/quantum/tutorials/quantum_data. based QML scheme
that uniformly projects acoustic features in a high-dimensional space during
its classical-to-quantum feature encoding phase to establish hybrid quantum-
classical classification models. Figure 1 provides an example showing how to
encode input acoustic features into quantum states and to capture latent
representations through quantum projections in the left block for kernel
metric learning [18, 19, 20] in the right block of Figure 1. Note that quantum
computing processors can be deployed in software-simulated toolkit [4, 21] and
noisy intermediate-scale quantum [22] (NISQ) hardware that are currently
accessible with $1$ to $100$ qubits.

##  2 Related Work

###  2.1 Low-Resources Spoken Command Recognition

Several competitive techniques have been developed for recognizing spoken
commands in various voice assistant applications (e.g., intelligent home
control). On the one hand, DNN-based acoustic models often achieve over $90\%$
accuracies for resource-sufficient languages (e.g., English [23]). On the
other hand, statistical learning (e.g., kernel SVM [24]) has been proven to be
a viable solution for low-resource languages [25], such as Sinhala and Tamil.
Moreover, prototypical network based metric learning [26, 27] shows
competitive performances in few-shot speaker identification tasks [27, 28,
29], where the model [26] learns per-class weights for target keywords, as in
classification objectives and maximizes the distance between non-targeted
samples. In this work, we will show that such an efficient non-parametric
learning approach can be combined with quantum computation for low-resource
spoken command recognition.

###  2.2 Applications of Quantum Machine Learning

Under the current computing constraints of limited qubits, quantum machine
learning (QML) has been mainly deployed with quantum circuits [11] for some
classical image or audio processing problems. The existing QML models are
often hybrid quantum-classical models that use quantum circuits as a
parameterized encoder or decoder. Those quantum circuits project the input
features into a high-dimension space and rotate the quantum state
representations. Some preliminary studies on quantum circuit based convolution
[30] showed that hybrid models could work effectively for English spoken
command recognition. However, the training time of a parameterized circuit
usually relies on the required variational gradient approximation algorithms
[31]. Instead, a quantum kernel technique provides another perspective of
using the characterises of quantum space on non-parametric learning, which
could be beneficial to the few-shot and low-resource training scenarios.

##  3 Quantum Projection for Kernel Learning

###  3.1 Quantum Feature Encoding and Projection

Unlike variational circuit learning [11] that requires the number of model
parameters in the order of the training set size, quantum kernel learning
(QKL) [15] can be adopted for limited training scenarios. QKL builds a quantum
mapping, $\phi$, transforming a classical input vector x into a quantum state
$|\phi(\textbf{x})\rangle$ in a quantum Hilbert space.

Let $Q$ stands for the total number of qubits in the utilized quantum circuit
and $|\delta_{q}\rangle$, $q=1,...,Q$, stands for the quantum state of the
$q^{th}$ qubit with two possible values, $\delta_{q}=0$ or $\delta_{q}=1$ in
Dirac notations [16]. $|0\rangle^{\otimes Q}$ refers to the tensor product of
$|0\rangle$ of $Q$ times, which is equivalent to
$|0\rangle\otimes|0\rangle\otimes\cdot\cdot\cdot\otimes|0\rangle$, as the
initial quantum state. More specifically, we consider a quantum encoding by
using a parametric quantum circuit, with a unitary operator $U(\textbf{x})$ of
$Q$ qubits, such that the input vector x is encoded into an embedded quantum
state for the entire circuit, $|\phi(\textbf{x})\rangle$. In doing so,
$|\phi_{q}(\textbf{x})\rangle$ represents the quantum state of $q^{th}$ qubit
after encoding. Next, we take a quantum measurement of
$|\phi(\textbf{x})\rangle$ based on the Pauli-Z axis for many times [16], and
then averages the measured values to produce a vector output, y, with the same
dimension of the original input vector x. Now we can project the associated
embedded quantum states $|\phi(\textbf{x})\rangle$ back to the original
classical space, such that for each pair ($\textbf{y}_{i},\textbf{y}_{j}$), a
kernel can be redefined as:

$\displaystyle K_{C}(\textbf{y}_{i},\textbf{y}_{j})\rightarrow~{}$
$\displaystyle
K_{Q}(\textbf{x}_{i},\textbf{x}_{j})=\left|\langle\phi(\textbf{x}_{i})|\phi(\textbf{x}_{j})\rangle\right|,$
(1)

where $K_{C}(\textbf{y}_{i},\textbf{y}_{j})$ represents the kernel in a
classical sense while $K_{Q}(\textbf{x}_{i},\textbf{x}_{j})$ denotes quantum
kernels. Finally, we calculate the inner-product of the two quantum feature
maps, $\phi(\textbf{x}_{i})$ and $\phi(\textbf{x}_{j})$, in Eq. (1) as a
distance measure for kernel learning, which are computed by first-order
reduced density matrix $\rho(\textbf{x})$ of classical input vector x with
circuit encoding. The actual measurement [32] between $\phi(\textbf{x}_{i})$
and $\phi(\textbf{x}_{j})$ is formed as a Gaussian kernel projection,
$K_{QG}=\exp\left(-\gamma\sum_{k}\left(\operatorname{Tr}\left[\rho\left(\textbf{x}_{i}\right)\right]-\operatorname{Tr}\left[\rho\left(\textbf{x}_{j}\right)\right]\right)^{2}\right)$,
where $\operatorname{Tr[\cdot]}$ is the trace operator and $\gamma$ is set to
be $1$ in our study.

###  3.2 Metric Learning with Quantum Measurements

We use Gaussian QKL combined with one competitive metric learning solution of
prototypical networks [33, 26]. Our training objective is jointly built upon
generalised end-to-end loss [34] and angular variant prototypical loss used in
[26] for training with low-resource speech data. For model training, we first
collect latent feature embeddings projected from output measurement of quantum
kernel and use prototypical loss to maximize class-wise distance in the latent
space. We train a prediction backbone with kernel SVM referred to the setup in
[26]. We then re-use the quantum projection with kernel learning in Eq. (1)
from the NISQ device for kernel SVM learning as a hybrid model. Similar to the
findings in [26], kernel SVM performs better than DNN serving a backbone
prediction model evaluated in our low-resource command data.

##  4 Experiments and Result Analysis

###  4.1 Experimental Setup and Baselines

We first test the English ($\mathbf{en}$) Google Speech Command [23] corpus,
containing a total of up to $11,165$ training and $6,500$ testing utterances
in $1$-second length. Next, for the low-resource spoken command recognition
task we have four languages: Georgian ($\mathbf{ga}$) with $1458$ utterances
and Chuvash ($\mathbf{cv}$) with $706$ utterances are collected from the
Mozilla Common Voice data set [35] with the same 10 frequent commands
translating from the above set of English words. Meanwhile, Arabic
($\mathbf{ar}$) [36] and Lithuanian [37] ($\mathbf{lt}$) Speech Commands are
referred to the setup in the existing spoken command classification studies.
Arabic includes $1600$ utterances with $6$ smart home control words and 10
spoken digits ($0$ through $9$). For Georgian, Chuvash, and Arabic data, we
use the same setup as in [36] to split these sets into $80$% for training and
$20\%$ for testing with $10$-fold cross-validation. Lithuanian duses $326$ ,
$75$ and $88$ utterances for training, validation and testing, respectively.
In Figure 2 we display the maximum training set sizes for each language used.
Clearly there is a noticeable difference between English and the other four
low-resource languages.

Fig. 2: Training set sizes for 5 languages used here: Arabic, Georgian,
Chuvash, and Lithuanian are considered low-resource when compared to English
(with $\sim 11$k utterances).

####  4.1.1 Data Pre-processing

We follow the data-format used in Google Speech Command [23] and refine all
input utterances of low-resource spoken commands to under $1$ second. We use
the original white noise setup in [23] to simulate a noisy background. Mel-
scaled spectra in $60$ bands and $1024$-point discrete Fourier transform are
extracted from the input speech signals have been extracted its with Librosa
[38] and Keras audio pre-processing layer [39]. For NISQ simulation, We run
TensorFlow quantum on TPU for our experiments with 22.7% reduced decoding time
compared to CPU-based simulation.

####  4.1.2 Baseline Algorithms

We select three competitive spoken command classification models of “pure
DNN”, “QCNN-DNN”, and “kernel metric learning” as baselines. Following the
setup in [30], we first select an attention recurrent neural network [40], as
our “pure DNN” baseline. We next select its extended hybrid quantum-classical
version (denoted as “QCNN-DNN“) proposed in [30], where a quantum convolution
layer has been used to replace some of the neural convolution layers of a pure
DNN model with a boosted performance. We use the best setup of $2\times 2$
convolution filter with $4$ qubits in [30] (increasing the qubits numbers up
to $9$ does not improve performances from our previous study [30]). Finally,
for the ”kernel metric learning” baseline, we use the best setup for speech
command classification proposed in [26] with a backbone of multi-class radial
basis function (RBF) based “kernel SVM” [41] using a typical loss [33] to
optimize its latent embedding as a distance metric.

###  4.2 Classification Result and Analysis

####  4.2.1 A Preliminary Study on Using Kernels

First, we did a quick study to compare QKL with other kernel-based metric
learning with an inner-product cosine distance, and display the results in
Table 1, where we randomly select $1,000$ English speech data for evaluation
on its validation accuracy and clustering distance between each class. As a
result, Gaussian-based quantum projection shows the best accuracy and smallest
distance compared to quantum linear kernel and metric learning [26] baselines.

Table 1: Comparing different quantum kernel learning setups Method | Acc. ($\uparrow$) | Cluster Dist. ($\downarrow$)
---|---|---
Kernel metric learning [26] | 65.3% | 2.12
Linear-QKL | 66.4% | 2.02
Gaussian-QKL | 72.6% | 1.83

####  4.2.2 English Spoken Command Recognition

To investigate the impact of insufficient training data, we train different
classification models introduced in Sec. 4.1 from scratch and report their
average accuracies under $10$-fold cross-validation. As shown in Table 2, DNN-
based baselines outperform the two other baselines and QKL, from $1.7$% to
$4.8$% in large-scale training with $11$k training utterances. However, QKL
demonstrates the best prediction performance after the training data scale
down to 1k and continuously performs as the best compared to all three other
baselines even with a limited training set of $500$ utterances. As a
preliminary finding, DNN-based models show a serve $57.1$% accuracy drop from
$95.2$% with full training (with $11$k utterances) to 38.1% in low-resource
training (with only $500$ samples).

Table 2: Comparing average classification accuracies (in %) of English Speech
Command under different training set sizes.

Number of training Utterances 500 1k 5k 11k Pure DNN [40] 38.1 57.2 83.6 95.2
Kernel metric learning [26] 44.4 61.8 77.4 90.4 QCNN-DNN [30] 41.2 62.9 80.5
93.2 Gaussian-QKL (proposed) 47.7 72.6 84.1 93.5

####  4.2.3 Low-resource Spoken Command Recognition

In Table 3, we display classification accuracies for the four low-resource
languages, $\\{\mathbf{ga}$, $\mathbf{cv}$, $\mathbf{lt}$, $\mathbf{ar}\\}$
discussed in Sec. 4.1. Based on the evaluation results, $\mathbf{cv}$ is the
most challenging language with a low accuracy ($<45\%$) on all models. DNN
(the third row of Table 3) shows poor accuracies of $17.6$% and $46.3$% on the
$\mathbf{cv}$ and $\mathbf{lt}$ when training data are fewer than $1$k
samples. As a hybrid quantum-classical model, QCNN-DNN demonstrates slightly
better performances than its DNN counterparts on the $\mathbf{ga}$,
$\mathbf{cv}$, and $\mathbf{ar}$ test sets. The proposed QKL-based models (the
sixth row of Table 3) perform the best when compared to the other baselines in
all four evaluated low-resource languages.

Table 3: A comparison of average classification accuracies (in %) for four low-resource languages: Georgian ($\mathbf{ga}$), Chuvash ($\mathbf{cv}$), Lithuanian ($\mathbf{lt}$), and Arabic ($\mathbf{ar}$). Language | $\mathbf{ga}$ | $\mathbf{cv}$ | $\mathbf{lt}$ | $\mathbf{ar}$
---|---|---|---|---
# Total classes | $10$ | $10$ | $15$ | $16$
Pure DNN [40] | 56.4 | 17.6 | 46.3 | 66.4
Kernel metric learning [26] | 57.3 | 30.7 | 44.2 | 63.6
QCNN-DNN [30] | 58.5 | 28.9 | 45.9 | 67.2
Gaussian-QKL (proposed) | 75.1 | 41.5 | 57.9 | 70.4

###  4.3 Discussions

####  4.3.1 Performances versus Training Epochs

As the first time to report spoken command classification with quantum kernel-
based learning, we further report its training time behavior as shown in
Figure 3 when compared to existing end-to-end training with the DNN-based
models.

Interestingly, QKL models (in red curves) take more training epochs than DNN
models (in black curves) to attain an accuracy of over $40$%. Convergence of
DNN-based models seems to be faster in both training and testing sets. This
phenomenon could be related to the discussion of “barren plateaus” in previous
QML studies [13, 14]. In summary, both DNN and QKL models seem to be over-
fitted at the training set as shown in the two upper dash-line curves.
However, the test set performance of QKL shows a better than $15$% accuracy
when compared to the test set performance of DNN when comparing the two lower
solid-line curves.

Fig. 3: Average training and test set accuracies at differnt training epochs
evaluating on four low-resource languages.

####  4.3.2 Low-resource SCR with Different Training Set Sizes

Finally, we conduct an additional evaluation of low-resource spoken commands
under different training set sizes ranging from $300$ to $700$, for
$\mathbf{ga}$, $\mathbf{cv}$, and $\mathbf{ar}$. We exclude the $\mathbf{lt}$
data since its maximum training set size is just about $300$. The data in the
three languages have been scaled down to observe the sensitivity of each
trained model. We report average validation accuracies (in the $y$-axis of
Figure 4) weighted by the total number of samples of each of the three
evaluated languages reported in Figure 2. Then , based on Figure 3, we notice
QKL continuously outperforms the other setups with better accuracies. QKL also
shows a more stable performance in terms of convergence rate when compared to
the other algorithms.

Fig. 4: A comparison of average test accuracy of kernel learning for low-
resource SCR in different training set sizes.

##  5 Summary

We propose kernel-based learning for classifying low-resource spoken commands.
Our experimental results suggest that: (i) DNN-based acoustic models are
sensitive to training set sizes, and (ii) quantum kernel learning is effective
in classifying low-resource spoken commands. QKL-based learning also gives
more stable and better validation accuracies when compared to existing kernel
learning and hybrid quantum-DNN models.

## References

  * [1] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath _et al._ , “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” _IEEE Signal processing magazine_ , vol. 29, no. 6, pp. 82–97, 2012.
  * [2] T. Sainath and C. Parada, “Convolutional neural networks for small-footprint keyword spotting,” _in Proc. Interspeech_ , 2015.
  * [3] M. P. Lewis, _Ethnologue: Languages of the world_. SIL international, 2009.
  * [4] M. Broughton, G. Verdon, T. McCourt, A. J. Martinez, J. H. Yoo, S. V. Isakov, P. Massey, R. Halavati, M. Y. Niu, A. Zlokapa _et al._ , “Tensorflow quantum: A software framework for quantum machine learning,” _arXiv preprint arXiv:2003.02989_ , 2020.
  * [5] S. Boixo, S. V. Isakov, V. N. Smelyanskiy, R. Babbush, N. Ding, Z. Jiang, M. J. Bremner, J. M. Martinis, and H. Neven, “Characterizing quantum supremacy in near-term devices,” _Nature Physics_ , vol. 14, no. 6, pp. 595–600, 2018\.
  * [6] A. Abbas, D. Sutter, C. Zoufal, A. Lucchi, A. Figalli, and S. Woerner, “The power of quantum neural networks,” _Nature Computational Science_ , vol. 1, no. 6, pp. 403–409, 2021.
  * [7] N. Killoran, T. R. Bromley, J. M. Arrazola, M. Schuld, N. Quesada, and S. Lloyd, “Continuous-variable quantum neural networks,” _Physical Review Research_ , vol. 1, no. 3, p. 033063, 2019.
  * [8] J. Qi and J. Tejedor, “Classical-to-quantum transfer learning for spoken command recognition based on quantum neural networks,” in _Proc. IEEE ICASSP_ , 2022, pp. 8627–8631.
  * [9] V. Havlíček, A. D. Córcoles, K. Temme, A. W. Harrow, A. Kandala, J. M. Chow, and J. M. Gambetta, “Supervised learning with quantum-enhanced feature spaces,” _Nature_ , vol. 567, no. 7747, pp. 209–212, 2019.
  * [10] G. Cybenko, “Approximation by superpositions of a sigmoidal function,” _Mathematics of control, signals and systems_ , vol. 2, no. 4, pp. 303–314, 1989.
  * [11] K. Mitarai, M. Negoro, M. Kitagawa, and K. Fujii, “Quantum circuit learning,” _Physical Review A_ , vol. 98, no. 3, p. 032309, 2018.
  * [12] M. Schuld, “Supervised quantum machine learning models are kernel methods,” _arXiv preprint arXiv:2101.11020_ , 2021.
  * [13] E. Grant, L. Wossnig, M. Ostaszewski, and M. Benedetti, “An initialization strategy for addressing barren plateaus in parametrized quantum circuits,” _Quantum_ , vol. 3, p. 214, 2019.
  * [14] J. R. McClean, S. Boixo, V. N. Smelyanskiy, R. Babbush, and H. Neven, “Barren plateaus in quantum neural network training landscapes,” _Nature communications_ , vol. 9, no. 1, pp. 1–6, 2018.
  * [15] H.-Y. Huang, M. Broughton, M. Mohseni, R. Babbush, S. Boixo, H. Neven, and J. R. McClean, “Power of data in quantum machine learning,” _Nature communications_ , vol. 12, no. 1, pp. 1–9, 2021.
  * [16] M. A. Nielsen and I. Chuang, _Quantum computation and quantum information_. Cambridge University Press, 2002.
  * [17] M. Schuld and N. Killoran, “Quantum machine learning in feature hilbert spaces,” _Physical review letters_ , vol. 122, no. 4, p. 040504, 2019.
  * [18] K. Q. Weinberger and G. Tesauro, “Metric learning for kernel regression,” in _Artificial intelligence and statistics_. PMLR, 2007, pp. 612–619.
  * [19] J. Li, M. Yuan, and C.-H. Lee, “Approximate test risk bound minimization through soft margin estimation,” _IEEE transactions on audio, speech, and language processing_ , vol. 15, no. 8, pp. 2393–2404, 2007.
  * [20] A. N. Pargellis, E. Fosler-Lussier, A. Potamianos, and C.-H. Lee, “Metrics for measuring domain independence of semantic classes.” 2001, pp. 447–450.
  * [21] V. Bergholm, J. Izaac, M. Schuld, C. Gogolin, M. S. Alam, S. Ahmed, J. M. Arrazola, C. Blank, A. Delgado, S. Jahangiri _et al._ , “Pennylane: Automatic differentiation of hybrid quantum-classical computations,” _arXiv preprint arXiv:1811.04968_ , 2018.
  * [22] J. Preskill, “Quantum computing in the nisq era and beyond,” _Quantum_ , vol. 2, p. 79, 2018.
  * [23] P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” _arXiv preprint arXiv:1804.03209_ , 2018.
  * [24] C. Cortes and V. Vapnik, “Support-vector networks,” _Machine learning_ , vol. 20, no. 3, pp. 273–297, 1995.
  * [25] Y. Karunanayake, U. Thayasivam, and S. Ranathunga, “Transfer learning based free-form speech command classification for low-resource languages,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , 2019, pp. 288–294.
  * [26] J. Huh, M. Lee, H. Heo, S. Mun, and J. S. Chung, “Metric learning for keyword spotting,” in _2021 IEEE Spoken Language Technology Workshop (SLT)_. IEEE, 2021, pp. 133–140.
  * [27] J. S. Chung, J. Huh, S. Mun, M. Lee, H.-S. Heo, S. Choe, C. Ham, S. Jung, B.-J. Lee, and I. Han, “In defence of metric learning for speaker recognition,” _in Proc. Interspeech_ , pp. 2977–2981, 2020.
  * [28] Y. Qian, X. Bianv, Y. Shi, N. Kanda, L. Shen, Z. Xiao, and M. Zeng, “Speech-language pre-training for end-to-end spoken language understanding,” in _Proc. IEEE ICASSP_ , 2021, pp. 7458–7462.
  * [29] Y. Tian and P. J. Gorinski, “Improving end-to-end speech-to-intent classification with reptile,” _in Proc. Interspeech_ , pp. 891–895, 2020\.
  * [30] C.-H. H. Yang, J. Qi, S. Y.-C. Chen, P.-Y. Chen, S. M. Siniscalchi, X. Ma, and C.-H. Lee, “Decentralizing feature extraction with quantum convolutional neural network for automatic speech recognition,” in _Proc. IEEE ICASSP_ , 2021, pp. 6523–6527.
  * [31] M. C. Caro, H.-Y. Huang, M. Cerezo, K. Sharma, A. Sornborger, L. Cincio, and P. J. Coles, “Generalization in quantum machine learning from few training data,” _Nature communications_ , vol. 13, no. 1, pp. 1–11, 2022.
  * [32] T. Hubregtsen, D. Wierichs, E. Gil-Fuster, P.-J. H. Derks, P. K. Faehrmann, and J. J. Meyer, “Training quantum embedding kernels on near-term quantum computers,” _Physical Review A_ , vol. 106, no. 4, p. 042431, 2022.
  * [33] J. Snell, K. Swersky, and R. Zemel, “Prototypical networks for few-shot learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [34] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-end loss for speaker verification,” in _Proc. IEEE ICASSP_ , 2018, pp. 4879–4883.
  * [35] R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in _Proceedings of the 12th Language Resources and Evaluation Conference_ , 2020, pp. 4218–4222.
  * [36] L. Benamer and O. Alkishriwo, “Database for arabic speech commands recognition,” in _Proc. CEST_ , 2020.
  * [37] A. Kolesau and D. Šešok, “Unsupervised pre-training for voice activation,” _Applied Sciences_ , 2020.
  * [38] B. McFee, C. Raffel, D. Liang, D. P. Ellis, M. McVicar, E. Battenberg, and O. Nieto, “librosa: Audio and music signal analysis in python,” in _Proceedings of the 14th python in science conference_ , vol. 8. Citeseer, 2015, pp. 18–25.
  * [39] K. Choi, D. Joo, and J. Kim, “Kapre: On-gpu audio preprocessing layers for a quick implementation of deep neural network models with keras,” _Machine Learning for Music Discovery Workshop, ICML_ , 2017.
  * [40] D. C. De Andrade, S. Leo, M. L. D. S. Viana, and C. Bernkopf, “A neural attention model for speech command recognition,” _arXiv preprint arXiv:1808.08929_ , 2018.
  * [41] J.-P. Vert, K. Tsuda, and B. Schölkopf, “A primer on kernel methods,” _Kernel methods in computational biology_ , vol. 47, pp. 35–70, 2004.


# Enhancing Transformers with Gradient Boosted Decision Trees for NLI Fine-
Tuning

Benjamin Minixhofer1 , Milan Gritta2 and Ignacio Iacobacci2
1Johannes Kepler University, Linz, Austria
2Huawei Noah’s Ark Lab, London, UK
<EMAIL_ADDRESS>
{milan.gritta<EMAIL_ADDRESS>
Work conducted as Research Intern at Huawei Noah’s Ark Lab.

###### Abstract

Transfer learning has become the dominant paradigm for many natural language
processing tasks. In addition to models being pretrained on large datasets,
they can be further trained on intermediate (supervised) tasks that are
similar to the target task. For small Natural Language Inference (NLI)
datasets, language modelling is typically followed by pretraining on a large
(labelled) NLI dataset before fine-tuning with each NLI subtask. In this work,
we explore Gradient Boosted Decision Trees (GBDTs) as an alternative to the
commonly used Multi-Layer Perceptron (MLP) classification head. GBDTs have
desirable properties such as good performance on dense, numerical features and
are effective where the ratio of the number of samples w.r.t the number of
features is low. We then introduce FreeGBDT, a method of fitting a GBDT head
on the features computed during fine-tuning to increase performance without
additional computation by the neural network. We demonstrate the effectiveness
of our method on several NLI datasets using a strong baseline model (RoBERTa-
large with MNLI pretraining). The FreeGBDT shows a consistent improvement over
the MLP classification head.

##  1 Introduction

Recent breakthroughs in transfer learning ranging from semi-supervised
sequence learning Dai and Le (2015) to ULMFiT Howard and Ruder (2018), ELMo
Peters et al. (2018) and BERT Devlin et al. (2019) have brought significant
improvements to many natural language processing (NLP) tasks. Transfer
learning involves pretraining neural networks, often based on the Transformer
Vaswani et al. (2017), on large amounts of text in a self-supervised manner in
order to learn transferable language features useful for many NLP tasks.
Pretraining is followed by fine-tuning the model on the target task.
Pretrained models can also be further trained on intermediate labelled
datasets which are similar to the target task before the final fine-tuning
stage Pruksachatkun et al. (2020). We refer to this as intermediate supervised
pretraining. In this manner, the network learns more meaningful internal
representations of the input text that are better aligned with the target
task. In order to fine-tune the pretrained network, some latent representation
of the text (e.g. the hidden state corresponding to the `[CLS]` token of BERT-
like models) is used as the input to a classification head, usually a randomly
initialised Multi-Layer Perceptron (MLP) Wolf et al. (2019). The output of the
classification head can then be interpreted as probability distribution over
classes. The input to the classification head is referred to as features
throughout the paper. It is a high-dimensional vector that serves as a rich,
distributed representation of the input text.

We investigate whether replacing the commonly used MLP classification head
with a GBDT Friedman (2001) can provide a consistent improvement, using NLI
tasks as our use case. GBDTs are known for strong performance on dense,
numerical features Ke et al. (2019), which includes the hidden states in a
neural network. The number of input features $p$, i.e. the dimension of the
hidden state corresponding to the `[CLS]` token is not necessarily much larger
than the number of samples $n$ (it may even be smaller). GBDTs have proven
effective for tasks where $n<p$ Kong and Yu (2018) and can be more effective
compared to logistic regression if $n\not\gg p$ Couronné et al. (2018).
Therefore, for a language model that was trained on an intermediate supervised
task before fine-tuning, we hypothesise that a GBDT may be able to outperform
an MLP classification head as the hidden states already encode information
relevant to the target task at the start of fine-tuning. The head must learn
to exploit this information exclusively during the fine-tuning stage in which
the training data may consist of only a few samples. Our contributions are as
follows:

  * •

We integrate the GBDT into a near state-of-the-art (SOTA) language model as an
alternative to an MLP classification head and train on the features extracted
from the model after fine-tuning. We refer to it as standard GBDT.

  * •

We introduce a method to train a GBDT on the features computed during fine-
tuning, at no extra computational cost by the neural network, showing a
consistent improvement over the baseline. We refer to it as FreeGBDT.

In the following, we recap different approaches to integrating tree-based
methods with neural networks (Section 2). We introduce our FreeGBDT method in
Section 3. We present our experimental setup in Section 4 and results on
standard NLI benchmarks in Section 5. To conclude, we discuss improvements and
limitations of our method in Section 6.

We release our code111Code will be available at https://github.com/huawei-
noah/free-gbdt., implemented with LightGBM Ke et al. (2017) and Huggingface’s
Transformers Wolf et al. (2019), to the NLP community.

##  2 Related Work

Recent work on transfer learning in NLP has often been based on pretrained
transformers, e.g. BERT Devlin et al. (2019), XLNet Yang et al. (2019), T5
Raffel et al. (2020) and RoBERTa Liu et al. (2019). These models are
pretrained on large datasets using self-supervised learning, typically a
variation of language modelling such as Masked Language Modelling (MLM). MLM
consists of masking some tokens as in the Cloze task Taylor (1953). The
objective of the model is to predict the masked tokens. Recently, approaches
using alternatives to MLM such as Electra Clark et al. (2020) and Marge Lewis
et al. (2020) have also been proposed. Pretraining transformers on large
datasets aims to acquire the semantic and syntactic properties of language,
which can then be used in downstream tasks. The models can additionally be
trained in a supervised manner on larger datasets before being fine-tuned on
the target task.

#### Natural Language Inference

is one of the most canonical tasks in Natural Language Understanding (NLU) Nie
et al. (2020); Bowman et al. (2015). NLI focuses on measuring commonsense
reasoning ability Davis and Marcus (2015) and can be seen as a proxy task that
estimates the amount of transferred knowledge from the self-supervised phase
of training. The task involves providing a premise (also called context) and a
hypothesis that a model has to classify as:

  * •

Entailment. Given the context, the hypothesis is correct.

  * •

Contradiction. Given the context, the hypothesis is incorrect.

  * •

Neutral. The context neither confirms nor disconfirms the hypothesis.

The task can also be formulated as binary classification between entailment
and not_entailment (contradiction or neutral). We focus on NLI as a
challenging and broadly applicable NLP task, with multiple smaller evaluation
datasets being available as well as the large Multi-Genre Natural Language
Inference corpus (Williams et al., 2018, MNLI), which is often used for
effective intermediate pretraining Liu et al. (2019). As such, it provides a
testing ground for the GBDT classification head with intermediate supervised
pretraining.

#### Tree-based methods

Models based on decision trees have a long history of applications to various
machine learning problems Breiman et al. (1984). Ensembling multiple decision
trees via bagging Breiman (1996) or boosting Freund et al. (1999) further
improves their effectiveness and remains a popular method for modelling dense
numerical data Feng et al. (2018). Ensemble methods such as Random Forests
Breiman (2001) and GBDTs combine predictions from many weak learners, which
can result in a more expressive model compared to an MLP. There have been
several approaches to combining neural networks with tree-based models,
approximately divided into two groups.

  1. 1.

Heterogeneous ensembling: The tree-based model and the neural network are
trained independently, then combined via ensembling techniques. Ensembling
refers to any method to combine the predictions of multiple models such as
stacking Wolpert (1992) or an arithmetic mean of the base models’ predictions.

  2. 2.

Direct integration: The tree-based model is jointly optimised with the neural
network.

Heterogeneous ensembling Li et al. (2019) has proven effective for many
applications such as Online Prediction Ke et al. (2019), Learning-to-Rank for
Personal Search Lazri and Ameur (2018) and Credit Scoring Xia et al. (2018).
It is also suitable for multimodal inputs, e.g. text, images and/or sparse
categorical features as some input types are better exploited by a neural
network while others are amenable to tree-based models Ke et al. (2019).

Figure 1: The baseline model architecture. Feature storage is populated during
fine-tuning for the FreeGBDT but after fine-tuning for the standard GBDT
method.

Direct integration makes the tree-based model compatible with back-propagation
thus trainable with the neural network in an end-to-end manner. Examples
include the Tree Ensemble Layer Hazimeh et al. (2020), Deep Neural Decision
Forests Kontschieder et al. (2015) and Deep Neural Decision Trees Yang et al.
(2018). Deep Forests Zhou and Feng (2017) are also related although they aim
to create deep non-differentiable models instead. Other examples include
driving neural network fine-tuning through input perturbation Bruch et al.
(2020), which focuses specifically on using a tree ensemble to fine-tune the
neural network representations.

As pretrained transformer-based models have recently achieved strong
performance on various NLP tasks Devlin et al. (2019), we see an opportunity
to take advantage of their distributed representations by the means of using a
tree-based model as the classification head. Our methods differ from direct
integration in that they are not end-to-end differentiable. The training
procedure is a sequence, i.e. the transformer-based model is fine-tuned first,
then a GBDT is trained with features extracted from the model. We do not
interfere with the model updates during training. Finally, the GDBT replaces
the MLP classification head. Our approach is invariant to the method with
which the neural network is fine-tuned as long as there exists a forward pass
in which the features are computed. Recent methods for neural network fine-
tuning include FreeLB Zhu et al. (2020) and SMART Jiang et al. (2020). FreeLB
is an adversarial method, which perturbs the input during training via
gradient ascent steps to improve robustness. SMART constrains the model
updates during fine-tuning with smoothness-inducing regularisation in order to
reduce overfitting. These approaches could theoretically be combined with both
the standard GBDT and the FreeGBDT.

##  3 Methodology

We introduce the standard GBDT (Algorithm 1) and the FreeGBDT (Algorithm 2),
our new method of using features generated during fine-tuning. Features refers
to the hidden state corresponding to the `[CLS]` token of BERT-like pretrained
models. We use these features as training data for the GBDT and FreeGBDT.

###  3.1 The standard GBDT classification head

In order to train the standard GBDT, we apply the feature extraction procedure
shown in Algorithm 1. Using the fine-tuned neural network, we perform one
additional forward pass over each sample in the training data. We store the
features as training data for the GBDT, denoted ’feature storage’ in Figures 1
and 2. The GBDT can be then used as a substitute for the MLP classification
head.

Figure 2: The GBDT classification head. Algorithm 1 Standard GBDT training
procedure. Features are extracted after fine-tuning.

0: training data $X$, pretrained network $f_{p}$ parametrised by $\theta_{p}$,
classification head $f_{h}$ parametrised by $\theta_{h}$.

for $epoch=1..N_{epochs}$ do

for minibatch $(X_{b},y_{b})\subset X$ do

$cls\leftarrow f_{p}(X_{b},\theta_{p})$

$y_{pred}\leftarrow f_{h}(cls,\theta_{h})$

$loss\leftarrow LossFn(y_{pred},y_{b})$

update $\theta_{h}$ and $\theta_{p}$ via backpropagation of $loss$

end for

end for

$features\leftarrow\textrm{empty list}$

$labels\leftarrow\textrm{empty list}$

for minibatch $(X_{b},y_{b})\subset X$ do

$cls\leftarrow f_{p}(X_{b},\theta_{p})$

extend $features$ with $cls$

extend $labels$ with $y_{b}$

end for

$gbdt\leftarrow train_{gbdt}(features,labels)$

###  3.2 The FreeGBDT classification head

Instead of extracting features once after fine-tuning, the training data for
the proposed FreeGBDT is obtained during fine-tuning. The features computed in
every forward pass of the neural network are stored as training data, shown in
Algorithm 2. As no additional computation by the neural network is required,
this new classification head is called FreeGBDT. Accumulating features in this
manner allows the FreeGBDT to be trained on $N\times E$ samples while the
standard GBDT is trained on $N$ samples where $N$ is the size of the dataset
and $E$ is the number of fine-tuning epochs.

Corpus | Train | Dev | Test | Classes
---|---|---|---|---
ANLI | 162k | 3.2k | 3.2k | 3
CNLI | 6.6k | 800 | 1.6k | 3
RTE | 2.5k | 278 | 3k | 2
CB | 250 | 57 | 250 | 3
QNLI | 104k | 5.4k | 5.4k | 2
Table 1: NLI evaluation datasets. The tasks with 3 classes contain labels:
entailment, neutral and contradiction, tasks with 2: entailment and
not_entailment.

##  4 Experimental Setup

We now describe the featured models, details of training procedures and
evaluation datasets.

###  4.1 Datasets

Algorithm 2 FreeGBDT training procedure. Features are accumulated throughout
fine-tuning.

0: training data $X$, pretrained network $f_{p}$ parametrised by $\theta_{p}$,
classification head $f_{h}$ parametrised by $\theta_{h}$.

$features\leftarrow\textrm{empty list}$

$labels\leftarrow\textrm{empty list}$

for $epoch=1..N_{epochs}$ do

for minibatch $(X_{b},y_{b})\subset X$ do

$cls\leftarrow f_{p}(X_{b},\theta_{p})$

extend $features$ with $cls$

extend $labels$ with $y_{b}$

$y_{pred}\leftarrow f_{h}(cls,\theta_{h})$

$loss\leftarrow LossFn(y_{pred},y_{b})$

update $\theta_{h}$ and $\theta_{p}$ via backpropagation of $loss$

end for

end for

$gbdt\leftarrow train_{gbdt}(features,labels)$

We evaluate our methods on the following NLI datasets, summarised in Table 1.

Method | CB | RTE | CNLI | ANLI | QNLI
---|---|---|---|---|---
MLP head | 93.57 (2.2) | 89.51 (0.8) | 82.49 (0.8) | 57.56 (0.5) | 94.32 (0.1)
standard GBDT | 94.11 (1.7) | 89.33 (0.8) | 80.84 (0.9) | 57.22 (0.5) | 94.29 (0.1)
FreeGBDT | 94.20 (1.7) | 89.69 (0.8) | 82.53 (0.7) | 57.63 (0.5) | 94.30 (0.1)
Table 2: Mean Accuracy (Standard Deviation) on the development sets from 20
runs with different random seeds. A Wilcoxon signed-rank test conducted across
all five datasets confirms significance with $p\approx 0.01$ c.f. Section 5.

  * •

Adversarial NLI (ANLI) Nie et al. (2020). This corpus consists of three rounds
of data collection. In each round, annotators try to break a model trained on
data from previous rounds. We use the concatenation of R1, R2 and R3.

  * •

Counterfactual NLI (CNLI) Kaushik et al. (2020). The CNLI corpus consists of
counterfactually-revised samples of SNLI Bowman et al. (2015). We use the full
dataset i.e. samples with the revised premise and with the revised hypothesis.

  * •

Recognising Textual Entailment (RTE) Wang et al. (2019b). We use the data and
format as used in the GLUE benchmark: a concatenation of RTE1 Dagan et al.
(2006), RTE2 Bar Haim et al. (2006), RTE3 Giampiccolo et al. (2007) and RTE5
Bentivogli et al. (2009), recast as a binary classification task between
entailment and not_entailment.

  * •

CommitmentBank (CB) de Marneffe et al. (2019). We use the subset of the data
as used in SuperGLUE Wang et al. (2019a).

  * •

Question-answering NLI (QNLI) Demszky et al. (2018). This is a converted
version of the Stanford Q&A Dataset Rajpurkar et al. (2016), aiming to
determine whether a given context contains the answer to a question.

###  4.2 Model and Training

We start all experiments from the RoBERTa-large model Liu et al. (2019) with
intermediate pretraining on the Multi-Genre Natural Language Inference (MNLI)
corpus Williams et al. (2018). The MNLI checkpoint is provided by the
fairseq222https://github.com/pytorch/fairseq library Ott et al. (2019). Note
that no task-specific tuning of hyperparameters was performed. Instead, we use
one learning rate cycle Smith (2017) with a maximum learning rate of $1\times
10^{-5}$ for each task to fine-tune RoBERTa for 10 epochs with a batch size of
32. We use the Adam optimiser Kingma and Ba (2015) to optimise the network. In
order to compare the FreeGBDTs with standard GBDTs, we apply Algorithm 1 and
Algorithm 2 during the same fine-tuning session to eliminate randomness from
different model initialisations.

We use LightGBM333https://lightgbm.readthedocs.io to train the GBDT. We do not
manually shuffle the data before training. The individual trees of a GBDT are
learned in a sequence where each tree is fit on the residuals of the previous
trees. One important parameter of the GBDT is thus the number of trees to fit.
This is commonly referred to as boosting rounds.

Parameter | CB | RTE | CNLI | ANLI | QNLI
---|---|---|---|---|---
learning rate | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
max. leaves | 256 | 256 | 256 | 256 | 256
boosting rounds | 10 | 10 | 10 | 40 | 30
Table 3: Hyperparameters of the standard GBDT and the FreeGBDT. All other
hyperparameters are set to the default value as per LightGBM version 2.3.1.

We observe that the optimal number of boosting rounds varies significantly
across tasks, with a tendency towards more boosting rounds for larger
datasets. Thus, we select the number of boosting rounds from the set
$\\{1,10,20,30,40\\}$ for each task. This is the only task-specific
hyperparameter in our experiments. The hyperparameters are identical for the
standard GBDT and the FreeGBDT, shown in Table 3, the model shown in Figure 2.
The time it takes to train the standard GBDT and the FreeGBDT is negligible
compared to the time it takes to fine-tune the RoBERTa model.

Figure 3: Accuracy on RTE and CNLI development sets. Training is paused after
each epoch to compare the GBDT, FreeGBDT and MLP heads. We plot the mean from
20 runs (same hyperparameters but different seeds).

###  4.3 Evaluation

#### Development set

We evaluate our methods using accuracy. Each experiment is repeated 20 times
with different random seeds. We report the mean and standard deviation.

#### Test set

For each task, we train the GBDT with the following boosting rounds:
$\\{1,10,20,30,40\\}$. We select the GBDT with the best score on the
development set. Test scores are obtained with a submission to the SuperGLUE
benchmark444https://super.gluebenchmark.com/ for CB and the GLUE
benchmark555https://gluebenchmark.com/ for RTE and QNLI. We calculate the test
scores on ANLI and CNLI ourselves as the test labels are publicly available.
We report accuracy on the test set for each task except for CB, where we
report the mean of F1 Score and Accuracy, same as the SuperGLUE leaderboard.

##  5 Results and Analysis

We summarise the results on the development sets in Table 2. The FreeGBDT is
compared with a standard GBDT and the MLP classification head. The standard
GBDT achieves a higher score than the MLP head on 1 out of 5 tasks. The
FreeGBDT outperforms the standard GBDT on 5 out of 5 tasks and the MLP head on
4 out of 5 tasks. As recommended for statistical comparison of classifiers
across multiple datasets Demšar (2006) we conduct a Wilcoxon signed-rank test
Wilcoxon (1992) with the accuracy differences between the FreeGBDT and the MLP
across the 20 seeds and 5 datasets. The test confirms that the improvement
from our FreeGBDT method is significant with $p\approx 0.01$.

Method | CB | RTE | CNLI | ANLI | QNLI
---|---|---|---|---|---
MLP head | 92.9 | 87.5 | 83.6 | 57.4 | 94.3
GBDT | 91.3 | 87.5 | 82.1 | 57.2 | 94.3
FreeGBDT | 93.3 | 87.8 | 83.7 | 57.6 | 94.3
Table 4: Results on the test sets.

Results on the test sets are shown in Table 4. The FreeGBDT achieves a small
but consistent improvement over the MLP head on each task except QNLI. This
task is not a conventional NLI task but a question-answering task converted to
an NLI format Demszky et al. (2018). It has been shown that QNLI does not
benefit from MNLI pretraining Liu et al. (2019) hence this result is not
unexpected. Out of the four datasets which do benefit from MNLI pretraining,
the FreeGBDT improves over the MLP head on each one with an average score
difference of +$0.23\%$. As our experiments start from a competitive baseline,
RoBERTa-large with MNLI pretraining, we consider the results important because
(a) to the best of our knowledge, this is the first tree-based method that
achieves near state-of-the-art performance on benchmark NLI tasks and (b) our
method is ’free’ as it requires no additional computations by the model. We
were able to demonstrate that a FreeGBDT head can be successfully integrated
with modern transformers and is a good alternative to the commonly used MLP
classification head.

Method | A1 | A2 | A3 | All
---|---|---|---|---
RoBERTa-large-mnli (ours) | 72.0 | 52.5 | 49.4 | 57.4
RoBERTa-large-mnli + GBDT (ours) | 72.1 | 52.2 | 49.0 | 57.2
SMART Jiang et al. (2020) | 72.4 | 49.8 | 50.3 | 57.1
ALUM Liu et al. (2020) | 72.3 | 52.1 | 48.4 | 57.0
InfoBERT Wang et al. (2021) | 75.0 | 50.5 | 49.8 | 58.3
RoBERTa-large-mnli + FreeGBDT (ours) | 71.9 | 52.7 | 49.7 | 57.6
Table 5: Accuracy across different rounds of the ANLI test set. All denotes a
sample-weighted average. Our FreeGBDT achieves SOTA on the A2 subset. InfoBERT
Wang et al. (2021) is the SOTA on the full test set.

For the Adversarial NLI (ANLI) dataset, we report results which are
competitive with the state-of-the-art shown in Table 5, surpassing both SMART
Jiang et al. (2020) and ALUM Liu et al. (2020). The RoBERTa-large model
pretrained with SNLI, MNLI, FEVER Thorne et al. (2018) and ANLI reported
$53.7\%$ accuracy on the ANLI test set Nie et al. (2020). The state-of-the-art
result of $58.3\%$ accuracy on the ANLI dataset was achieved by InfoBERT Wang
et al. (2021). The FreeGBDT achieves a new state-of-the-art on the A2 subset
of ANLI with $52.7\%$. Interestingly, it does not yield an improvement on the
easier A1 subset but compares favourably to other recent approaches on the
more difficult A2 and A3 subsets of ANLI.

To better understand how performance of the GBDTs evolves during fine-tuning,
we carry out an additional experiment. We pause training after each epoch to
extract features and train a standard GBDT. We compare it with the MLP
classification head and a FreeGBDT trained on the features accumulated up to
the current epoch. The result on the RTE and CNLI datasets is shown in Figure
3. Notably, the FreeGBDT does not improve on the standard GBDT after the first
epoch where the number of instances the GBDT is trained on is equal to the
size of the training dataset for both. As the FreeGBDT starts accumulating
more training data, however, it consistently outperforms the standard GBDT and
eventually, the MLP head.

The state-of-the-art in NLI provides some context for our method of combining
tree-based models with modern neural networks. RoBERTa (large with MNLI
pretraining) reports $89.5\%$ accuracy on the development set of RTE Liu et
al. (2019) and $94.7\%$ accuracy on the development set of QNLI. The same
model obtains an F1 Score / Accuracy of $90.5/95.2$ on the CB test set and an
accuracy of $88.2\%$ on the RTE test set. Note that ensembles of 5 to 7 models
Liu et al. (2019) were used while our test figures achieve similar scores of
$91.3/95.2$ for CB and $87.8\%$ for RTE with a single model. These are not
direct comparisons, however, the figures demonstrate that FreeGBDT can operate
at SOTA levels while matching and exceeding the ’default’ MLP head classifier
accuracy. Across all datasets, the FreeGBDT improves by an average of $0.2\%$
and $0.5\%$ over the MLP head and the standard GBDT head, respectively.

##  6 Discussion

The FreeGBDT improves over the MLP head on each task where intermediate
supervised pretraining on MNLI is effective. The improvement is significant
but not large. This is expected since the input features of the classification
head are already a highly abstract representation of the input. Thus, there is
limited potential for improvement. However, our results show that a tree-based
method is a viable alternative to the commonly used MLP head and can improve
over a baseline chosen to be as competitive as possible. Notably, the FreeGBDT
improves the MLP baseline on the CB dataset by $>0.6\%$ solely by switching to
our tree-based classification head.

Furthermore, the FreeGBDT outperforms a standard GBDT by a large margin in
some cases. For instance, we observe a +$1.5\%$ improvement on the CNLI
dataset. Figure 3 shows the gap forming towards the end of training. We think
this may be due to overfitting to the training data. Recall that the standard
GBDT is trained only on features extracted after fine-tuning. At this point,
the features may exhibit a higher degree of memorisation of the training data.
The FreeGBDT is able to mitigate this problem as it was trained with features
collected throughout training. Let $f(x,\theta_{t})$ denote a mapping from the
input text $x$ to the output space parameterised by $\theta_{t}$ where
$t\in\\{0..T\\}$ and $T$ is the total amount of steps the model is fine-tuned
for. Then, the standard GBDT is trained on features from $f(x,\theta_{T})$,
while the FreeGBDT is trained on features from every $t$ in $\\{0..T\\}$. As
such, it may help to think of the FreeGBDT as a type of regularisation through
data augmentation (from the FreeGBDT’s point of view), having trained on
several perturbed views of each training instance.

Figure 4: Values of a typical dimension from the 1,024 dimensional vector
stored during fine-tuning to train a FreeGBDT (left). The same dimension
extracted after fine-tuning to train a standard GBDT (right).

Figure 4 helps illustrate the regularisation effect by showing the differences
between the FreeGBDT and standard GBDT training data beyond just size. The
figure shows the temporal changes in a typical feature collected during fine-
tuning and the same feature extracted after fine-tuning. We can see that the
distribution gradually changes from earlier epochs but remains similar to the
distribution of the feature at the end of fine-tuning. FreeGBDT is able to
exploit the information at the start of fine-tuning as the features at $t=0$
already encode information highly relevant to the target task hence all
training data is useful. The FreeGBDT head compares favourably to an MLP head,
which is randomly initialized at the start of the fine-tuning stage and must
thus learn to exploit the latent information from a potentially small amount
of training examples. Therefore, we believe intermediate supervised
pretraining is essential for the effectiveness of the FreeGBDT, supported by
the results from preliminary experiments on the BoolQ dataset Clark et al.
(2019) and QNLI, which does not benefit from pretraining on MNLI Liu et al.
(2019) where FreeGBDT matches the accuracy of the MLP head but does not exceed
it. Our experiments also suggest that the potential for improvement from
FreeGBDT depends on the size of the training dataset. The gap between FreeGBDT
and the MLP head in Table 4 is larger for the smaller datasets CB and RTE and
smaller for the larger datasets (ANLI, CNLI, QNLI). This is consistent with
prior work showing that GBDTs are especially effective compared to other
methods if $n\not\gg p$ Couronné et al. (2018) and hints that the FreeGBDT
method might be especially useful for smaller datasets.

##  7 Future Work

One possible avenue for future work is exploring different features to train
the GBDT, e.g. the hidden states from different layers of the pretrained
model. This includes new combinations of top layer representations of the
Transformer to generate richer input features for the classification head.
This could lead to potential improvement by leveraging a less abstract
representation of the input. Given that our method operates on distributed
representations from a pretrained encoder, applications in other domains such
as Computer Vision may be possible, e.g. using features extracted from a
ResNet He et al. (2016) encoder. Furthermore, a GBDT might not be the best
choice for each task hence the use of Random Forests Breiman (2001) or Support
Vector Machines Cortes and Vapnik (1995) may also be evaluated to investigate
the effectiveness of combining Transformer neural networks with traditional
supervised learning methods.

##  8 Conclusion

State-of-the-art transfer learning methods in NLP are typically based on
pretrained transformers and commonly use an MLP classification head to fine-
tune the model on the target task. We have explored GBDTs as an alternative
classification head due to their strong performance on dense, numerical data
and their effectiveness when the ratio of the number of samples w.r.t the
number of features is low. We have shown that tree-based models can be
successfully integrated with transformer-based neural networks and that the
free training data generated during fine-tuning can be leveraged to improve
model performance with our proposed FreeGBDT classification head. Obtaining
consistent improvements over the MLP head on several NLI tasks confirms that
tree-based learners are relevant to state-of-the-art NLP.

## References

  * Bar Haim et al. (2006) Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second PASCAL recognising textual entailment challenge.
  * Bentivogli et al. (2009) Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009.  The fifth PASCAL recognizing textual entailment challenge.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Breiman (1996) Leo Breiman. 1996.  Bagging predictors.  _Machine learning_ , 24(2):123–140.
  * Breiman (2001) Leo Breiman. 2001.  Random forests.  _Machine learning_ , 45(1):5–32.
  * Breiman et al. (1984) Leo Breiman, Jerome Friedman, Charles J Stone, and Richard A Olshen. 1984.  _Classification and regression trees_.  CRC press.
  * Bruch et al. (2020) Sebastian Bruch, Jan Pfeifer, and Mathieu Guillame-Bert. 2020.  Learning representations for axis-aligned decision forests through input perturbation.  _CoRR_ , abs/2007.14761.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _ICLR_.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine learning_ , 20(3):273–297.
  * Couronné et al. (2018) Raphael Couronné, Philipp Probst, and Anne-Laure Boulesteix. 2018.  Random forest versus logistic regression: a large-scale benchmark experiment.  _BMC bioinformatics_ , 19(1):270.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The PASCAL recognising textual entailment challenge.  In _Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment_ , pages 177–190. Springer.
  * Dai and Le (2015) Andrew M. Dai and Quoc V. Le. 2015.  Semi-supervised sequence learning.
  * Davis and Marcus (2015) Ernest Davis and Gary Marcus. 2015.  Commonsense reasoning and commonsense knowledge in artificial intelligence.  _Commun. ACM_ , 58(9):92–103.
  * Demšar (2006) Janez Demšar. 2006.  Statistical comparisons of classifiers over multiple data sets.  _Journal of Machine Learning Research_ , 7(1):1–30.
  * Demszky et al. (2018) Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018.  Transforming question answering datasets into natural language inference datasets.  _arXiv preprint arXiv:1809.02922_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Feng et al. (2018) Ji Feng, Yang Yu, and Zhi-Hua Zhou. 2018.  Multi-layered gradient boosting decision trees.  In _Advances in neural information processing systems_ , pages 3551–3561.
  * Freund et al. (1999) Yoav Freund, Robert Schapire, and Naoki Abe. 1999.  A short introduction to boosting.  _Journal-Japanese Society For Artificial Intelligence_ , 14(771-780):1612.
  * Friedman (2001) Jerome H Friedman. 2001.  Greedy function approximation: a gradient boosting machine.  _Annals of statistics_ , pages 1189–1232.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , pages 1–9. Association for Computational Linguistics.
  * Hazimeh et al. (2020) Hussein Hazimeh, Natalia Ponomareva, Petros Mol, Zhenyu Tan, and Rahul Mazumder. 2020.  The tree ensemble layer: Differentiability meets conditional computation.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018.  Universal language model fine-tuning for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  * Jiang et al. (2020) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020.  SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2177–2190, Online. Association for Computational Linguistics.
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard Hovy, and Zachary C Lipton. 2020.  Learning the difference that makes a difference with counterfactually augmented data.  _International Conference on Learning Representations (ICLR)_.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017.  Lightgbm: A highly efficient gradient boosting decision tree.  In _Advances in neural information processing systems_ , pages 3146–3154.
  * Ke et al. (2019) Guolin Ke, Zhenhui Xu, Jia Zhang, Jiang Bian, and Tie-Yan Liu. 2019.  Deepgbm: A deep learning framework distilled by gbdt for online prediction tasks.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 384–394.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kong and Yu (2018) Yunchuan Kong and Tianwei Yu. 2018.  A deep neural network model using random forest to extract feature representation for gene expression data classification.  _Scientific reports_ , 8(1):1–9.
  * Kontschieder et al. (2015) Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. 2015\.  Deep neural decision forests.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1467–1475.
  * Lazri and Ameur (2018) Mourad Lazri and Soltane Ameur. 2018.  Combination of support vector machine, artificial neural network and random forest for improving the classification of convective and stratiform rain using spectral features of seviri data.  _Atmospheric research_ , 203:118–129.
  * Lewis et al. (2020) Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020.  Pre-training via paraphrasing.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18470–18481. Curran Associates, Inc.
  * Li et al. (2019) Pan Li, Zhen Qin, Xuanhui Wang, and Donald Metzler. 2019.  Combining decision trees and neural networks for learning-to-rank in personal search.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2032–2040.
  * Liu et al. (2020) Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020.  Adversarial training for large neural language models.  _arXiv preprint arXiv:2004.08994_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * de Marneffe et al. (2019) Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019.  The commitmentbank: Investigating projection in naturally occurring discourse.  _Proceedings of Sinn und Bedeutung_ , 23(2):107–124.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of NAACL-HLT 2019: Demonstrations_.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020\.  Intermediate-task transfer learning with pretrained language models: When and why does it work? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5231–5247.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Smith (2017) Leslie N Smith. 2017.  Cyclical learning rates for training neural networks.  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 464–472. IEEE.
  * Taylor (1953) Wilson L Taylor. 1953.  “cloze procedure”: A new tool for measuring readability.  _Journalism quarterly_ , 30(4):415–433.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint 1905.00537_.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In the Proceedings of ICLR.
  * Wang et al. (2021) Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. 2021.  Infobert: Improving robustness of language models from an information theoretic perspective.  In _International Conference on Learning Representations_.
  * Wilcoxon (1992) Frank Wilcoxon. 1992.  Individual comparisons by ranking methods.  In _Breakthroughs in statistics_ , pages 196–202. Springer.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.
  * Wolpert (1992) David H Wolpert. 1992.  Stacked generalization.  _Neural networks_ , 5(2):241–259.
  * Xia et al. (2018) Yufei Xia, Chuanzhe Liu, Bowen Da, and Fangming Xie. 2018.  A novel heterogeneous ensemble credit scoring model based on bstacking approach.  _Expert Systems with Applications_ , 93:182–199.
  * Yang et al. (2018) Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. 2018.  Deep neural decision trees.  _arXiv preprint arXiv:1806.06988_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5753–5763.
  * Zhou and Feng (2017) Zhi-Hua Zhou and Ji Feng. 2017.  Deep forest: Towards an alternative to deep neural networks.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , pages 3553–3559.
  * Zhu et al. (2020) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020.  Freelb: Enhanced adversarial training for natural language understanding.  In _International Conference on Learning Representations_.