# Model Uncertainty–Aware Knowledge Amalgamation for
Pre-Trained Language Models

Lei Li${\dagger}$, Yankai Lin$\S$, Xuancheng Ren${\dagger}$,
Guangxiang Zhao${\dagger}$, Peng Li$\S$, Jie Zhou$\S$, Xu Sun${\dagger}$
${\dagger}$MOE Key Laboratory of Computational Linguistics, School of EECS,
Peking University
$\S$Pattern Recognition Center, WeChat AI, Tencent Inc., China
<EMAIL_ADDRESS>
{renxc, guangxiangzhao<EMAIL_ADDRESS>
{yankailin, patrickpli<EMAIL_ADDRESS>
###### Abstract

As many fine-tuned pre-trained language models (PLMs) with promising
performance are generously released, investigating better ways to reuse these
models is vital as it can greatly reduce the retraining computational cost and
the potential environmental side-effects. In this paper, we explore a novel
model reuse paradigm, Knowledge Amalgamation (KA) for PLMs. Without human
annotations available, KA aims to merge the knowledge from different teacher-
PLMs, each of which specializes in a different classification problem, into a
versatile student model. The achieve this, we design a Model Uncertainty–aware
Knowledge Amalgamation (MUKA) framework, which identifies the potential
adequate teacher using Monte-Carlo Dropout for approximating the golden
supervision to guide the student. Experimental results demonstrate that MUKA
achieves substantial improvements over baselines on benchmark datasets.
Further analysis shows that MUKA can generalize well under several complicate
settings with multiple teacher models, heterogeneous teachers, and even cross-
dataset teachers.

##  1 Introduction

Large-scale pre-trained language models (PLMs), such as BERT (Devlin et al.,
2019), RoBERTa (Liu et al., 2019) and T5 (Raffel et al., 2020) have recently
achieved promising results after fine-tuning on various natural language
processing (NLP) tasks. Many fine-tuned PLMs are generously released for
facilitating researches and deployments. Reusing these PLMs can greatly reduce
the computational cost of retraining the PLM from scratch, thus alleviating
the potential environmental side-effects (Strubell et al., 2019; Schwartz et
al., 2020).

A commonly adopted model reuse paradigm is knowledge distillation (Hinton et
al., 2015; Romero et al., 2015), which transfers learned knowledge from a
teacher model to a student model via matching the outputs between the teacher
model and the student. Though achieving promising results with PLMs (Sun et
al., 2019; Jiao et al., 2020; Sanh et al., 2019), KD restricts the student to
perform the same task as the teacher model, leading to an under-utilization of
released PLMs fine-tuned on different tasks. In this paper, we explore a novel
model reuse paradigm for PLMs, namely _Knowledge Amalgamation_ (KA). Given
multiple fine-tuned teacher-PLMs, each of which is capable of performing
classification over a unique label set, KA aims to train a versatile student
that can make predictions over the union of teacher label sets. Note that we
assume no human annotations are available during KA, as in many scenarios the
labeled data for training the teachers may not be publicly available due to
privacy issues. KA can make full use of the released PLMs, as it relaxes the
assumption of KD and trains a student to cover the classification skills of
all teacher models. Figure 1 illustrates the difference between KA and KD.

Figure 1: Comparison of knowledge distillation (KD) and knowledge amalgamation
(KA) with two teachers. KD assumes that the student performs predictions on
the identical label set with the teacher, while KA trains a student model that
is capable of performing classification over the union label set of teacher
models.

As no annotations are available, the core challenge of KA is to construct
supervision to guide the student. To this end, we propose a principled
framework to achieve KA for PLMs. Assuming that there is an oracle teacher
capable of providing virtual supervision over the union label set, through
theoretical derivation, we find that the key to construct such supervision is
to identify the adequate teacher for each instance. Our investigation shows
that model uncertainty estimated with Monte-Carlo Dropout (Gal and Ghahramani,
2016) can be utilized as a good proxy. We then build our Model
Uncertainty–aware Knowledge Amalgamation (MUKA) framework, where the golden
supervision is approximated by either taking the outputs of the most confident
teacher, or softly integrating different teacher predictions according to the
relative importance of each teacher. Furthermore, an instance re-weighting
mechanism based on the margin of uncertainty scores is introduced, to down-
weight the contribution of instances with confusing supervision signals.
Experimental results show that MUKA can significantly outperform previous
baselines, even achieving comparable results with models trained with labeled
data. Further analysis shows that MUKA can produce supervision close to the
golden distribution and generalize well in challenging settings, including
merging knowledge from multiple teacher models, heterogeneous teachers with
different architectures, or cross-dataset even cross-lingual teacher models.

Figure 2: Overview of the proposed MUKA framework. The label distribution over
the union label set is estimated according to the model uncertainty scores.
The student model is trained via minimizing the KL-divergence between the
hard(red) /soft(blue) estimated supervision and the student output, weighted
by the model uncertainty margin of teachers. Best viewed in color.

The main contributions of this work can be summarized as follows: (1) We
explore a novel model reuse paradigm, knowledge amalgamation for PLMs, making
full use of released PLMs with different label sets. (2) We present MUKA,
which approximates the golden supervision according to model uncertainty and
re-weights the instance contribution based on uncertainty margin. (3)
Extensive results on benchmarks demonstrate that MUKA provides more accurate
supervision, and is generalizable for amalgamating heterogeneous teachers and
cross-dataset teachers.

##  2 Related Work and Background

##### Knowledge Distillation (KD)

transfers the knowledge from a teacher model to a student model. For example,
Hinton et al. (2015) utilize the soft labels produced by the teacher model for
the student to learn. and Romero et al. (2015) align the intermediate
representations between the student and the teacher. Recent studies apply KD
for PLMs by matching the intermediate states (Sun et al., 2019; Sanh et al.,
2019; Wang et al., 2020), enriching the training data (Jiao et al., 2020;
Liang et al., 2021), and learning from multiple teachers (Wu et al., 2021; Li
et al., 2021). However, all these KD studies assume that the student has an
identical label set with the target teacher model(s), restricting better
utilization of PLMs with different label sets.

##### Knowledge Amalgamation (KA)

aims to train a versatile student model with unlabeled data, by amalgamating
the knowledge from multiple teachers. Previous studies mainly focus on reusing
teacher CNNs for image-related tasks, and can be roughly categorized into two
streams. One is matching the student outputs separately to the corresponding
teacher (Vongkulbhisal et al., 2019). The other instead focuses on aligning
the representations of the student and the teachers: Shen et al. (2019) map
the student features to that of the teacher model layer-by-layer and Luo et
al. (2019) conduct the feature alignment in a common space by minimizing the
maximum mean discrepancy. However, the former results in conflicting
supervision for the student as teacher-PLMs tend to be overconfident (Desai
and Durrett, 2020), and the latter paradigm requires complicated optimization
design for generalizing to heterogeneous teachers. To the best of our
knowledge, we are the first to explore KA for PLMs. We propose a principled
framework based on model uncertainty and demonstrate its effectiveness and
generalizability on benchmark datasets.

##### Bayesian Neural Networks (BNN)

replaces a deterministic weight parameter of a model with a prior distribution
of the weight. Instead of optimizing the model weights directly, BNN averages
over all possible weights when conducting inference. Formally, Bayesian
inference aims to estimate the posterior distribution
$p\left(\mathbf{W}\mid\mathcal{D}\right)$ of the model parameters $\mathbf{W}$
given the dataset $\mathcal{D}$. The prediction of Bayesian inference for
input $x$ is conducted by marginalizing over all the possible parameters as:

$p\left(y\mid x,\mathcal{D}\right)=\int
p\left(y\mid\mathbf{\mathbf{W}},x\right)p(\mathbf{\mathbf{W}}\mid\mathcal{D})\differential\mathbf{W}.$
(1)

As the exact integration is intractable for deep neural networks with complex
architectures and high dimension parameters, various variational inference
methods (Graves, 2011; Blundell et al., 2015; Gal and Ghahramani, 2016) have
been proposed to find a surrogate distribution
$q_{\theta}\left(\mathbf{W}\right)$ to approximate the underlying posterior
$p\left(\mathbf{W}\mid\mathcal{D}\right)$, where the Kullback-Leibler (KL)
divergence of these two distributions are minimized. Among these, the Monte-
Carlo dropout method (Gal and Ghahramani, 2016) is widely adopted as it
requires minimal changes to the original model, which takes the Dropout
distribution (Srivastava et al., 2014) as $q_{\theta}({\mathbf{W}})$ to sample
$K$ masked model weights $\left\\{\mathbf{W}_{k}\right\\}_{k=1}^{K}\sim
q_{\theta}\left(\mathbf{W}\right)$. The estimated probability for
classification tasks thus can be calculated as:

$\displaystyle p\left(y\mid x,\mathcal{D}\right)$ $\displaystyle\approx\int
p\left(y\mid\mathbf{\mathbf{W}},x\right)q_{\theta}(\mathbf{W})\differential\mathbf{W}$
(2)
$\displaystyle\approx\frac{1}{K}\sum_{k=1}^{K}p\left(y\mid\mathbf{W}_{k},x\right).$
(3)

##  3 Methodology

In this section, we first give the task formulation for knowledge
amalgamation, followed by the elaboration on the proposed MUKA framework.
Figure 2 gives an overview of our proposal.

###  3.1 Problem Formulation

We explore the problem of knowledge amalgamation for text classification with
PLMs. Formally, given $N$ teacher PLM models
$TS=\left\\{T_{1},\dots,T_{N}\right\\}$, where each teacher $T_{i}$
specializes in a specific classification problem, i.e., a set of classes
$Y_{i}$, a student model $S$ is supposed to be learned with an unlabeled
dataset $\mathcal{D}$, which is able to perform predictions over the
comprehensive class set $Y=\bigcup_{i=1}^{N}Y_{i}$. As previous studies
(Vongkulbhisal et al., 2019; Thadajarassiri et al., 2021) show that merging
teachers with overlapping classes is easy, we thus focus on a more challenging
yet practical setting where the teacher specialties are totally disjoint,
i.e., $Y_{i}\cap Y_{j}=\emptyset,\forall i\neq j$.

###  3.2 Model Uncertainty–Aware Knowledge Amalgamation

As there are no annotated data available, we need to construct supervision for
guiding the student. Assuming that there is an oracle model $\mathcal{T}$,
which is capable of giving a golden label distribution $\mathcal{T}(x)$ for
each instance $x$ over $Y$, we can approximate this supervision, and then
train the student by minimizing the KL-divergence between its prediction and
the estimated distribution:

$\mathcal{L}=\sum_{x\in\mathcal{D}}\text{KL}\left(S\left(x\right)||\mathcal{T}\left(x\right)\right),$
(4)

where $S(x)$ denotes the output distribution of the student for input $x$. To
estimate the golden supervision $\mathcal{T}(x)$, we first derive the
correlation between $\mathcal{T}(x)$ and the prediction $T_{i}(x)$ of teacher
model $T_{i}$. Specifically, as teacher $T_{i}$ specializes in label set
$Y_{i}$, it can only predict $T_{i}(y\mid x)$ for instance $x$ when $y\notin
Y_{-i}=\bigcup_{j=1,j\neq i}^{N}Y_{j}$. Therefore, the correlation between
$T_{i}(y\mid x)$ and global probability $\mathcal{T}(y\mid x)$ over the full
class set can be derived as:

$\displaystyle T_{i}(y\mid x)$ $\displaystyle=\mathcal{T}\left(y\mid x,y\in
Y_{i}\right)$ (5) $\displaystyle=\frac{\mathcal{T}\left(y,y\in Y_{i}\mid
x\right)}{\mathcal{T}\left(y\in Y_{i}\mid x\right)}.$ (6)

The above derivation indicates that we can recover the golden probability
distribution by estimating the denominator, which means how likely the
instance $x$ belongs to the classes in $Y_{i}$. Followingly, we explore
prediction uncertainty as an estimation of this probability.

####  3.2.1 Uncertainty Estimation

Figure 3: Model uncertainty (normalized) distributions evaluated with $1000$
instances randomly sampled from the AG News dataset. The vanilla prediction
entropy distributions of two teacher models overlaps greatly (left), while
Monte-Carlo Dropout produces a more accurate uncertainty approximation for
distinguishing the correct teacher model (right). Best viewed in color.

As the instances associated with classes not in $Y_{i}$ can be treated as the
out-of-distribution data for the teacher model $T_{i}$, the teacher
predictions would be more confusing about these instances than that of in-
distribution instances (Hendrycks and Gimpel, 2017). Therefore, we propose to
approximate the denominator in the Eq. (6) via model uncertainty. A naïve
estimation is taking the entropy of predicted class distribution. However, due
to the over-confident issues of over-parameterized models like PLMs (Guo et
al., 2017; Desai and Durrett, 2020), this simple estimation can be unreliable.
We investigate this by first spliting the instances of the AG News dataset
(Zhang et al., 2015) into two sets with disjoint labels, and then fine-tuning
teacher models on each set separately. For each instance, there is a correct
teacher that is capable of handling it and a wrong teacher that cannot make a
correct prediction for it. We plot the prediction entropy distributions of the
correct teacher and the wrong teacher in the left part of Figure 3. It can be
found that the wrong teacher also produces confident predictions for instances
that are not in its speciality with nearly zero uncertainty scores, misleading
the identification of the adequate teacher.

To remedy this, we propose to add small perturbations to the model parameter
during prediction to find out the correct teacher model. The intuition behind
is that, as the instance is well fitted by the parameter of the correct
teacher model, the adequate teacher thus produces confident results
consistently in the multiple predictions even with small perturbed parameters.
On the contrary, small perturbations on the model weights of the wrong teacher
will lead to a drastic change in the output probabilities, resulting in more
uncertainty predictions on average. Therefore, we can estimate the model
uncertainty more accurately according to the average predictions under
parameter perturbation, which is compatible with Monte-Carlo Dropout (Gal and
Ghahramani, 2016), a Bayesian perspective for obtaining the model uncertainty
score. Specifically, for an instance $x$, we calculate its output probability
distribution over $Y_{i}$ with $T_{i}$ via Monte-Carlo Dropout as:

$\displaystyle p_{i}\left(y\mid x,\mathcal{D}\right)$
$\displaystyle\approx\frac{1}{K}\sum_{k=1}^{K}p_{i}\left(y\mid\mathbf{W}_{k}^{i},x\right)$
(7)
$\displaystyle=\frac{1}{K}\sum_{k=1}^{K}T_{i}\left(x,\mathbf{W}_{k}^{i}\right),$
(8)

where $\mathbf{W}_{k}^{i}$ is the $k$-th masked weights of $T_{i}$ sampled
from the Dropout distribution (Srivastava et al., 2014). The model uncertainty
of teacher model $T_{i}$ thus can be summarized as the entropy of the averaged
probability distribution $p_{i}$:

$u_{i}=H\left(p_{i}\right)=-\sum_{y=1}^{|Y_{i}|}p_{i}^{y}\log p_{i}^{y}.$ (9)

As shown in the right part of Figure 3, the uncertainty distribution estimated
via Monte-Carlo Dropout exhibit a clear difference between the uncertainty
distributions of the correct teacher and the wrong teacher model, indicating
its great potential for guiding the probability combination.

With the accurately estimated teacher uncertainties
$U=\\{u_{1},\dots,u_{N}\\}$ at hand, we design two methods for approximating
the golden supervision to guide the student model:

MUKA-Hard which directly takes the supervision provided by the teacher with
the lowest uncertainty as the golden distribution:

$\mathcal{T}(x)=\text{Pad}\left(T_{i^{*}}\left(x\right)\right),\text{where}\
i^{*}=\mathop{\arg\min}_{i}u_{i}/\log|Y_{i}|,$ (10)

where $\log|Y_{i}|$ is a normalizing factor for comparison between teacher
models with a different number of classes. As $T_{i}^{*}$ only provides the
label relation over the class set $Y_{i^{*}}$, the probabilities of classes
not in $Y_{i^{*}}$ are set to zeros, denoted by the Pad operation. In this
way, we set $\mathcal{T}(y\in Y_{i^{*}}\mid x)=1$, and thus the student can
learn from the teacher model most confident about $x$.

MUKA-Soft which estimates the golden supervision as a weighted sum of teacher
model predictions by taking the relative uncertain level into consideration:

$\displaystyle\mathcal{T}(x)$
$\displaystyle=\sum_{i=1}^{N}w_{i}\text{Pad}\left(T_{i}\left(x\right)\right)$
(11) $\displaystyle w_{i}$
$\displaystyle=\frac{\exp\left(c_{i}/\tau\right)}{\sum_{j=1}^{N}\exp(c_{j}/\tau)}$
(12) $\displaystyle c_{i}$ $\displaystyle=1-u_{i}/\log|Y_{i}|,$ (13)

where $c_{i}$ denotes the confidence score which indicates how likely $x$
belongs to $C_{i}$, and $\tau$ is a hyper-parameter for controlling the
smoothness of the weights. In this way, the teacher with a higher confidence
score contributes more to the estimated golden supervision signal. Besides,
the difference between the confidence scores reflects the inner correlation
between the classes in different label groups, thus providing extra
information for the classes in disjoint label sets.

####  3.2.2 Instance Re-weighting

Furthermore, the uncertainty distribution overlapping in the right part of
Figure 3 indicates that there is still a small portion of instances on which
teacher models achieve similar confidence levels even with Monte-Carlo
Dropout. For these instances, MUKA-Hard may wrongly select the supervision
source, and MUKA-Soft would assign close weights to all teachers, thus
providing a vague supervision signal. Therefore, the estimated supervision can
be misleading for the student model. To remedy this, we devise an instance re-
weighting mechanism by modifying the objective in Eq. (4):

$\displaystyle\mathcal{L}_{\text{MUKA}}$
$\displaystyle=\sum_{x\in\mathcal{D}}v\left(x\right)\text{KL}\left(S\left(x\right)||\mathcal{T}(x)\right)$
(14) $\displaystyle v\left(x\right)$
$\displaystyle=c_{\text{max}}-c_{\text{sec}},$ (15)

where $c_{\text{max}}$ and $c_{\text{sec}}$ denotes the largest and the second
large teacher confidence score for instance $x$, respectively. By minimizing
the instance-level weighted objective, the student is encouraged to focus more
on the pivotal instances with clearer supervision signals, thus achieving a
better learning performance.

##  4 Experiments

Method | Model Size | AG News | THUCNews | Google Snippets | 5Abstracts Group | Average
---|---|---|---|---|---|---
Supervised | 110M | 94.6 $\pm$ 0.00 | 97.8 $\pm$ 0.00 | 89.3 $\pm$ 0.00 | 90.7 $\pm$ 0.00 | 93.10
Teacher 1 | 110M | 49.9 $\pm$ 0.00 | 48.8 $\pm$ 0.00 | 50.2 $\pm$ 0.00 | 42.0 $\pm$ 0.00 | 47.73
Teacher 2 | 110M | 47.5 $\pm$ 0.00 | 49.8 $\pm$ 0.00 | 43.5 $\pm$ 0.00 | 51.5 $\pm$ 0.00 | 48.08
Ensemble | 220M | 59.8 $\pm$ 0.00 | 93.1 $\pm$ 0.00 | 80.4 $\pm$ 0.00 | 62.3 $\pm$ 0.00 | 73.90
Vanilla KD | 110M | 63.1 $\pm$ 0.81 | 94.9 $\pm$ 0.18 | 83.7 $\pm$ 1.30 | 67.0 $\pm$ 1.14 | 77.18
DFA | 110M | 66.4 $\pm$ 2.33 | 94.4 $\pm$ 0.22 | 82.6 $\pm$ 0.18 | 57.7 $\pm$ 4.41 | 75.28
CFL | 110M | 61.4 $\pm$ 1.18 | 95.1 $\pm$ 0.21 | 84.5 $\pm$ 0.45 | 61.6 $\pm$ 0.12 | 75.65
UHC | 110M | 78.8 $\pm$ 1.42 | 92.1 $\pm$ 0.63 | 86.3 $\pm$ 0.39 | 71.4 $\pm$ 0.67 | 82.15
MUKA-Hard (Ours) | 110M | 87.0 $\pm$ 0.40 |  97.2 $\pm$ 0.12 |  88.4 $\pm$ 0.32 | 79.0 $\pm$ 0.82 | 87.90
MUKA-Soft (Ours) | 110M |  87.1 $\pm$ 0.19 |  97.2 $\pm$ 0.08 | 87.9 $\pm$ 0.32 |  79.3 $\pm$ 0.85 | 87.88
Table 1: Comparisons on the the benchmark datasets. The results are
classification accuracy averaged by three seeds, and standard deviations are
reported. Both MUKA variants achieve statistically significant improvements
over the best-performing baselines ($p<0.01$). Best results are shown in bold.

###  4.1 Experimental Settings

##### Datasets

We conduct evaluations on four classic text classification benchmarks,
including three English datasets: AG News (Zhang et al., 2015), Google
Snippets (Phan et al., 2008), 5Abstracts
Group,111https://github.com/qianliu0708/5AbstractsGroup and a Chinese dataset
THUCNews (Sun et al., 2016). We split 5% of data from the training set for
datasets without a validation set to form a validation set for model
selection. The statistics of datasets can be found in Appendix A.

##### Compared Methods

We implement various baselines to evaluate the effectiveness of our proposal,
which can be categorized as follows:

_Simple Baselines_ , which require no additional training, including: (1)
Original Teacher: The teacher models are used independently for prediction. We
set the probabilities of classes out of the teacher speciality to zeros. (2)
Ensemble: The output logits of teachers are directly concatenated for
predictions over the union label set.

_KA Methods_ , which assume internal states of the teacher model are available
and the unlabeled data $\mathcal{D}$ are used to distill a student model,
including: (1) Vanilla KD (Hinton et al., 2015): The student is trained to
mimic the soft targets produced by logits combination of all teacher models,
via minimizing the vanilla KL-divergence objective. (2) DFA (Shen et al.,
2019): DFA designs a layer-wise feature adaptation mechanism for providing
extra guidance based on Vanilla KD. The student aligns its features to the
merged features of multiple teachers layer by layer. (3) CFL (Luo et al.,
2019): CFL first maps the hidden representations of the student and the
teachers into a common space. The student is trained by aligning the mapped
features to that of the teachers, with supplemental supervision from the
logits combination. (4) UHC (Vongkulbhisal et al., 2019): UHC splits the
student logits into subsets corresponding to the class sets of teacher models.
Each subset is trained to mimic the corresponding output of the teacher model.

We also include a supervised learning method, which trains the student model
with labeled data for better understanding the performance.

##### Implementation Details

We implement our framework using the HuggingFace transformers library (Wolf et
al., 2020). The teacher and student models for English datasets and THUCNews
are BERT-base-uncased (Devlin et al., 2019) and BERT-wwm-ext (Cui et al.,
2020), respectively. For each dataset, the classes are randomly split into two
non-overlapping parts, and two teachers are fine-tuned on each set separately
to imitate the actual applications. Detailed class split can be found in
Appendix A. We first fine-tune the teacher models with the split labeled data
for $3$ epochs with a learning rate $2\times 10^{-5}$. The trained teacher
model weights are frozen during the student training process. We set the
forward number $K$ of Monte-Carlo Dropout uncertainty estimation to $16$ and
the dropout rate is set to $0.1$. Temperature $\tau$ in Eq. (12) is set to
$0.2$ according to our hyper-parameter analysis results in Appendix B. The
student model then is learned by optimizing the KL-divergence objective for
$3$ epochs, with a $2\times 10^{-5}$ learning rate and $32$ batch size. The
student is evaluated on the validation set every $100$ step. We select the
best performing checkpoints for final evaluation. The experiments are
replicated with $3$ random seeds and we report the averaged accuracy.

###  4.2 Main Results

The model performance comparison on the four datasets and the corresponding
model size are listed in Table 1. Our findings are: (1) Simple baselines fall
far behind, showing that it is necessary to conduct amalgamation. (2) DFA and
CFL cannot achieve consistent improvements over Vanilla KD, demonstrating the
instability of supervision based on feature alignments. (3) UHC achieves
better average results than Vanilla KD, while performs relatively worse on the
THUCNews dataset, indicating its instability due to the potential supervision
conflicting. (4) Two variants of MUKA both outperform the previous baseline
models on all the datasets, and the average accuracy of MUKA-Hard is achieves
a $5.75$ points gain over the best performing baseline model. On the THUCNews
dataset, while no label information is included during the knowledge
amalgamation, MUKA can obtain a $97.2$ accuracy, which is very close to $97.8$
of the supervised learning method. We attribute the success to that MUKA
provides the student with the estimated golden probability distribution over
the union label set according to model uncertainty, which can effectively
transfer the knowledge and avoid potential supervision contradictory. These
promising results indicate that our MUKA framework can produce better
supervisions for training the student model, thus has great potentials for
reusing PLMs with different label sets.

###  4.3 Ablation Studies

Method | AG News | THUCNews
---|---|---
MUKA-Hard | 87.0 $\pm$ 0.40 | 97.2 $\pm$ 0.12
w/o Monte-Carlo Dropout | 65.1 $\pm$ 1.67 | 97.0 $\pm$ 0.13
w/o Instance Re-weighting | 85.5 $\pm$ 0.51 | 96.9 $\pm$ 0.06
MUKA-Soft | 87.1 $\pm$ 0.19 | 97.2 $\pm$ 0.08
w/o Monte-Carlo Dropout | 74.2 $\pm$ 0.28 | 95.4 $\pm$ 0.20
w/o Instance Re-weighting | 86.7 $\pm$ 0.30 | 96.8 $\pm$ 0.03
Table 2: Ablation analysis of MUKA. The removed modules both lead to
deteriorated performance.

##### Contribution of Components in MUKA

We conduct ablation experiments on two large datasets for stable results, and
the results are shown in Table 2. By replacing the Monte-Carlo Dropout
estimation of model estimation with a single forward estimation, i.e., set
$K=1$ in Eq. (3), we find that the performance is degraded on both datasets.
This result validates our motivation to adopt the Monte-Carlo Dropout method
to better estimate the model uncertainty. Interestingly, we find that Monte-
Carlo Dropout is much more effective on the AG News dataset. To explore this,
we compute the average ECE score (Guo et al., 2017) of two teacher models on
the out-of-distribution samples, where higher ECE scores indicate more severe
overconfident predictions. The teacher models of AG News achieve an average
$45.42$ ECE score, while that of THUCNews is $19.52$. This result verifies the
effectiveness of Monte-Carlo Dropout for accurate model uncertainty estimation
of overconfident teachers. We also find that removing the instance re-
weighting mechanism leads to deteriorated results for both MUKA variants,
demonstrating that paying more attention to instances with clearer supervision
is beneficial for the student.

Figure 4: The KL-divergence to the golden supervision of different methods.
Lower is better.

##### Investigation on Supervision Quality

We probe the quality of estimated label distribution over the union label set.
Specifically, we train a PLM with all labeled data as an oracle teacher, which
thus can provide golden supervision over the union label set. We then
calculate the KL-divergence between the golden probability distribution and
the approximated supervision of different combination methods. The lower KL-
divergence indicates the combined predictions are more correct and thus can
guide the training of student models better. We discard results of MUKA-Hard
as KL-divergence is not defined for distributions with zeros, and divide MUKA-
Soft into two groups according to $v(x)$, i.e. instances with $v(x)\geq 0.5$
and that with $v(x)<0.5$. As shown in Figure 4, we find that the predictions
of instances with greater confidence score margin, i.e., $v(x)\geq 0.5$, are
much closer to the golden distributions than the Vanilla KD. This indicates
that the estimated supervision of instances with a clearer confidence margin
is of higher quality, thus paying more attention to these instances is
effective for amalgamation.

Method | 3 Teachers | 4 Teachers | 5 Teachers
---|---|---|---
{3,3,4} | {2,2,2,4} | {2,2,2,2,2}
Teacher 1 | 29.9 $\pm$ 0.00 | 20.0 $\pm$ 0.00 | 20.0 $\pm$ 0.00
Teacher 2 | 29.8 $\pm$ 0.00 | 19.1 $\pm$ 0.00 | 19.1 $\pm$ 0.00
Teacher 3 | 39.8 $\pm$ 0.00 | 19.9 $\pm$ 0.00 | 19.9 $\pm$ 0.00
Teacher 4 | N / A | 39.8 $\pm$ 0.00 | 20.0 $\pm$ 0.00
Teacher 5 | N / A | N / A | 20.0 $\pm$ 0.00
Ensemble | 91.0 $\pm$ 0.00 | 74.0 $\pm$ 0.00 | 80.6 $\pm$ 0.00
Vanilla KD | 92.5 $\pm$ 1.13 | 76.5 $\pm$ 0.57 | 82.6 $\pm$ 1.64
DFA | 91.1 $\pm$ 1.25 | 79.6 $\pm$ 1.34 | 82.0 $\pm$ 2.04
CFL | 93.9 $\pm$ 1.07 | 77.4 $\pm$ 1.34 | 84.0 $\pm$ 0.24
UHC | 84.4 $\pm$ 0.91 | 71.6 $\pm$ 2.86 | 69.4 $\pm$ 2.73
MUKA-Hard |  94.7 $\pm$ 0.20 | 93.4∗ $\pm$ 0.58 |  90.5∗ $\pm$ 0.32
MUKA-Soft |  94.7 $\pm$ 0.14 |  93.6∗ $\pm$ 0.30 |  90.5∗ $\pm$ 1.13
Table 3: Results of merging multiple teacher models on the THUCNews dataset.
The best results are shown in bold and ∗ denotes the improvement over the best
performing baseline is significant with $p<0.05$.

###  4.4 Results in Challenging Settings

##### KA with Multiple Teachers

As our framework is agnostic to the number of teacher models, we further
conduct experiments with more teacher models. We conduct experiments on the
THUCNews dataset as it has $10$ classes, allowing us to train up to $5$
teacher models specialized in different class sets. The number of classes
split in to $\\{3,3,4\\}$, $\\{2,2,2,4\\}$ and $\\{2,2,2,2,2\\}$ for $3$, $4$
and $5$ teachers, respectively. As shown in Table 3, the proposed MUKA
framework generalizes well to this setting by outperforming previous baselines
with a clear margin. Besides, we find that the previous models perform poorly
under the $4$-teacher scenario. We attribute it to that when some teacher
models have more classes than others, they usually produce a larger range of
logits to make the prediction more distinguishable. Therefore, directly
combing the teacher logits will lead to a biased probability distribution. The
MUKA instead estimates the golden probability distribution according to model
uncertainty scores, thus can produce better supervision even teacher models
exhibit different logit scales.

##### KA with Heterogeneous Teachers

We further consider merging knowledge from teacher models with different
structures. Specifically, we select BERT-base ($12$ layers and $768$ hidden
units) and BERT-large ($24$ layers and $1024$ hidden units) as the teachers,
respectively. The results are listed in Table 4. We find that while a larger
teacher tends to perform better, the student model performs worse on the
THUCNews dataset than learning from two BERT-base teachers, indicating it is
challenging to amalgamate knowledge in this setting. Our MUKA achieves the
best results on the two datasets, showing its effectiveness for heterogeneous
teachers.

Method | _$\text{T}_{1}$ : BERT-base_ | _$\text{T}_{2}$ : BERT-large_
---|---|---
AG News | THUCNews
Teacher 1 | 49.7 $\pm$ 0.00 | 48.8 $\pm$ 0.00
Teacher 2 | 47.2 $\pm$ 0.00 | 49.8 $\pm$ 0.00
Ensemble | 76.6 $\pm$ 0.00 | 79.3 $\pm$ 0.00
Vanilla KD | 79.6 $\pm$ 0.22 | 82.9 $\pm$ 1.36
DFA | 78.2 $\pm$ 0.30 | 84.7 $\pm$ 1.74
CFL | 75.9 $\pm$ 0.63 | 81.9 $\pm$ 1.37
UHC | 78.3 $\pm$ 2.65 | 92.3 $\pm$ 1.08
MUKA-Hard | 78.3 $\pm$ 1.58 |  95.4∗ $\pm$ 0.45
MUKA-Soft |  80.6 $\pm$ 0.17 |  95.4∗ $\pm$ 0.29
Table 4: Results of merging from heterogeneous teacher models with different
architectures. ∗ denotes results are statistically significant with $p<0.05$.

##### KA with Cross-Dataset Teachers

Specifically, we fine-tune teacher models on different datasets separately and
then train a student to perform classification over the union label set of
both datasets. The results of merging knowledge from two English datasets, AG
News and Google-Snippets and even cross-lingual datasets, AG News (in English)
and THUCNews (in Chinese) are listed in Table 5. The multilingual BERT-base
are adopted for the teachers and the student in the cross-lingual setting. Our
MUKA still outperforms previous baseline models in both settings.
Interestingly, we find that the MUKA-Hard is consistently better than MUKA-
Soft in this setting. We speculate the reason is that the correlations between
classes of different datasets are weak, thus modeling the label relation in
these disjoint groups is unnecessary.

Method | _$\text{T}_{1}$ : AG News (en)_ | _$\text{T}_{1}$ : AG News (en)_
---|---|---
_$\text{T}_{2}$ : GS (en)_ | _$\text{T}_{2}$ : THUCNews (zh)_
Teacher 1 | 26.8 $\pm$ 0.00 | 0.50 $\pm$ 0.00
Teacher 2 | 63.0 $\pm$ 0.00 | 54.6 $\pm$ 0.00
Ensemble | 74.3 $\pm$ 0.00 | 54.5 $\pm$ 0.00
Vanilla KD | 75.1 $\pm$ 0.58 | 54.6 $\pm$ 0.05
DFA | 74.7 $\pm$ 0.30 | 54.1 $\pm$ 0.20
CFL | 74.0 $\pm$ 0.50 | 54.3 $\pm$ 0.65
UHC | 72.0 $\pm$ 0.73 | 53.5 $\pm$ 0.17
MUKA-Hard |  76.3 $\pm$ 0.54 |  68.1∗ $\pm$ 0.20
MUKA-Soft | 75.9 $\pm$ 0.47 | 65.6∗ $\pm$ 0.40
Table 5: Results of merging from teacher models with different knowledge
domains and even in a cross-lingual scenario. ∗ denotes results are
statistically significant with $p<0.05$. GS is short for Google Snippets.

###  4.5 Error Analysis

Figure 5: (Left) Label distribution of wrongly predicted instance. Labels in
the same color indicate they belong to the same teacher specialty. (Right) The
confusion matrix of an oracle model trained with all labeled data.

We investigate the failure cases where MUKA selects the wrong teacher model
for a better understanding of MUKA. Specifically, we probe the label
distribution of instances on which MUKA assigns a higher uncertainty score to
the correct teacher on THUCNews. As shown in the left part of Figure 5, MUKA
only makes incorrect selections on a small portion of instances, and
interestingly, the labels of these instances are not uniformly distributed. To
explore this, we plot the label confusion matrix of an oracle model that is
fine-tuned with labeled data with all categories in the right part of Figure
5. On the one hand, we find that there are labels that can even confuse the
oracle model, e.g., instances of _Estate_ are tending to be classified into
_Politics_ and _Finance_ , indicating that the teacher selection error of MUKA
is partially due to the inner class similarity between the class labels. On
the other hand, while the oracle hardly makes wrong predictions on _Tech._ and
_Finance_ instances, the uncertainty estimation fails more frequently on these
two classes than other classes. We leave the investigation on this phenomenon
for future work. Our instance re-weighting mechanism is effective for these
instances with wrongly selected supervision source, as the the average teacher
uncertainty margin $v(x)$ is $0.04$, which greatly reduces the negative impact
for the student.

##  5 Conclusion

In this paper, we explore knowledge amalgamation for pre-trained language
models for promoting better model reuse. We present a principled framework
MUKA, which combines the partial predictions of teachers according to the
corresponding model uncertainty, and dynamically adjusts the contribution of
each instance by paying more attention to instances with clearer supervision.
Experiments on text classification benchmarks demonstrate our MUKA can
substantially outperform strong baselines. Further investigations show that
MUKA is generalizable for challenging settings, including merging knowledge
from multiple teacher models, heterogeneous teachers, or even cross-dataset
teachers.

##  6 Ethical Considerations

Our work faces several ethical challenges. As the released PLMs may exhibit
potential biases against specific groups, e.g., gender or ethnic minorities
(Kurita et al., 2019; Kennedy et al., 2020), these social biases can be
propagated to the merged student model during the knowledge amalgamation
process. Besides, users may collect unlabeled data from the web for conducting
knowledge amalgamation, which possibly contains offensive content and thus may
introducing new biases into the amalgamated model as well.

We offer possible remedies to reduce the concerns. For the biases exhibited in
the teacher models, de-biasing techniques (Zmigrod et al., 2019; Liang et al.,
2020; Schick et al., 2021) can be applied to eliminate the potential biases in
the teachers before using the teachers model for amalgamation. For the
offensive unlabeled data collected from the internet, simple template-based or
human-in-the-loop data cleaning strategies can be adopted to identify and
filter potential biased data. Except for these ad-hoc techniques, developing a
bias-aware knowledge transfer framework that can de-biasing the supervision
for the student model is also an interesting direction, which we are glad to
explore in the future.

## References

  * Blundell et al. (2015) Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015.  Weight uncertainty in neural network.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , volume 37 of _JMLR Workshop and Conference Proceedings_ , pages 1613–1622. JMLR.org.
  * Cui et al. (2020) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020\.  Revisiting pre-trained models for Chinese natural language processing.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 657–668, Online. Association for Computational Linguistics.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 295–302, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016_ , volume 48 of _JMLR Workshop and Conference Proceedings_ , pages 1050–1059. JMLR.org.
  * Graves (2011) Alex Graves. 2011.  Practical variational inference for neural networks.  In _Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain_ , pages 2348–2356.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1321–1330. PMLR.
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _ArXiv preprint_ , abs/1503.02531.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4163–4174, Online. Association for Computational Linguistics.
  * Kennedy et al. (2020) Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. 2020.  Contextualizing hate speech classifiers with post-hoc explanation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5435–5442, Online. Association for Computational Linguistics.
  * Kurita et al. (2019) Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019.  Measuring bias in contextualized word representations.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 166–172, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2021) Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021.  Dynamic knowledge distillation for pre-trained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liang et al. (2021) Kevin J. Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021.  Mixkd: Towards efficient distillation of large-scale language models.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Liang et al. (2020) Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2020.  Towards debiasing sentence representations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5502–5515, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Luo et al. (2019) Sihui Luo, Xinchao Wang, Gongfan Fang, Yao Hu, Dapeng Tao, and Mingli Song. 2019\.  Knowledge amalgamation from heterogeneous networks by common feature learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_ , pages 3087–3093. ijcai.org.
  * Phan et al. (2008) Xuan Hieu Phan, Minh Le Nguyen, and Susumu Horiguchi. 2008.  Learning to classify short and sparse text & web with hidden topics from large-scale data collections.  In _Proceedings of the 17th International Conference on World Wide Web, WWW 2008, Beijing, China, April 21-25, 2008_ , pages 91–100. ACM.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_ , 21:140:1–140:67.
  * Romero et al. (2015) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015.  Fitnets: Hints for thin deep nets.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  In _NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing_.
  * Schick et al. (2021) Timo Schick, Sahana Udupa, and Hinrich Schütze. 2021.  Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp.  _ArXiv preprint_ , abs/2103.00453.
  * Schwartz et al. (2020) Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2020.  Green ai.  _Communications of the ACM_ , 63(12):54–63.
  * Shen et al. (2019) Chengchao Shen, Xinchao Wang, Jie Song, Li Sun, and Mingli Song. 2019.  Amalgamating knowledge towards comprehensive classification.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 3068–3075. AAAI Press.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _JMLR_ , 15(1):1929–1958.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3645–3650, Florence, Italy. Association for Computational Linguistics.
  * Sun et al. (2016) Maosong Sun, Jingyang Li, Zhipeng Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016.  Thuctc: an efficient chinese text classifier.  _GitHub Repository_.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for BERT model compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4323–4332, Hong Kong, China. Association for Computational Linguistics.
  * Thadajarassiri et al. (2021) Jidapa Thadajarassiri, Thomas Hartvigsen, Xiangnan Kong, and Elke A. Rundensteiner. 2021.  Semi-supervised knowledge amalgamation for sequence classification.  In _AAAI_ , pages 9859–9867.
  * Vongkulbhisal et al. (2019) Jayakorn Vongkulbhisal, Phongtharin Vinayavekhin, and Marco Visentini Scarzanella. 2019.  Unifying heterogeneous classifiers with distillation.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 3175–3184. Computer Vision Foundation / IEEE.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021.  One teacher is enough? pre-trained language model distillation from multiple teachers.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4408–4413, Online. Association for Computational Linguistics.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 649–657.
  * Zmigrod et al. (2019) Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, and Ryan Cotterell. 2019.  Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1651–1661, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Datasets Details

Here we provide the details of datasets used in our main paper. The label set
is first sorted according to the name and will be evenly divided into subsets
according to the number of teacher models. Table 6 gives the dataset
statistics and the class number for two teacher models experiments. Table 7
gives the sorted label list on each dataset. For teacher number that cannot
evenly dividing the label sets, the final label set will include the left
labels. For example, when there are $4$ teacher models are needed on THUCNews
dataset, the label set will be split into {_Sports_ , _Enter._}, {_Furniture_
, _Estate_}, {_Education_ , _Fashion_} and {_Politics_ , _Game_ ,_Tech._ ,
_Finance_}.

Dataset | #Class | #Train | #Test | $\\{|Y|\\}$
---|---|---|---|---
AG News | 4 | 120,000 | 7,600 | {2, 2}
5Abstracts Group | 5 | 5,300 | 1,000 | {2, 3}
Google Snippets | 8 | 10,000 | 2,200 | {4, 4}
THUCNews | 10 | 50,000 | 10,000 | {5, 5}
Table 6: Statistics of datasets used in our paper. $\\{|Y|\\}$ denotes the number of classes each teacher model specializes. Dataset | Label Order
---|---
AG News | World, Sports, Business, Sci/Tech
THUCNews | Sports, Entertainment, Furniture,
Estate, Education, Fashion,
Politics, Game, Technology, Finance
Google Snippets | Business, Computers,
Culture-Arts-Entertainment,
Education-Science, Engineering,
Health, Politics-Society, Sports
5Abstracts Group | Business, CSAI, Law, Sociology, Trans
Table 7: Sorted label names of datasets. Label names of THUCNews are
translated into English.

##  Appendix B Hyper-parameter Search for $\tau$

In MUKA-Soft, $\tau$ is a hyper-parameter controlling the smoothness of
teacher supervision weights. We perform a hyper-parameter search experiment
for the optimal $\tau$. We conduct experiments on AG News and THUCNews for
stable results. The values of $\tau$ are picked from
$\\{0.01,0.1,0.2,0.5,1.0,2.0,5.0,10.0\\}$, and the results with different
$\tau$ are shown in Figure 6. We observe that the accuracy drops significantly
when $\tau$ is set to high values, where the teacher weights distribution is
becoming a uniform distribution, while it reaches a peak when $\tau$ is set to
a small value between $0.2$ and $0.5$. It indicates that slightly sharpening
the teacher weights distribution is helpful for the knowledge amalgamation.
Therefore, we adopt $\tau=0.2$ in all the experiments in the main paper.

Figure 6: Varying temperature $\tau$ for MUKA-Soft. The average accuracy of
three seeds are plotted with standard deviation in shade. Best viewed in
color.