# Dynamic Knowledge Distillation for Pre-trained Language Models

Lei Li${\dagger}$, Yankai Lin$\S$, Shuhuai Ren${\dagger}$, Peng Li$\S$, Jie
Zhou$\S$, Xu Sun${\dagger}$
${\dagger}$MOE Key Laboratory of Computational Linguistics, School of EECS,
Peking University
$\S$Pattern Recognition Center, WeChat AI, Tencent Inc., China
{lilei<EMAIL_ADDRESS><EMAIL_ADDRESS>
{yankailin, patrickpli<EMAIL_ADDRESS>
###### Abstract

Knowledge distillation (KD) has been proved effective for compressing large-
scale pre-trained language models. However, existing methods conduct KD
statically, e.g., the student model aligns its output distribution to that of
a selected teacher model on the pre-defined training dataset. In this paper,
we explore whether a dynamic knowledge distillation that empowers the student
to adjust the learning procedure according to its competency, regarding the
student performance and learning efficiency. We explore the dynamical
adjustments on three aspects: teacher model adoption, data selection, and KD
objective adaptation. Experimental results show that (1) proper selection of
teacher model can boost the performance of student model; (2) conducting KD
with 10% informative instances achieves comparable performance while greatly
accelerates the training; (3) the student performance can be boosted by
adjusting the supervision contribution of different alignment objective. We
find dynamic knowledge distillation is promising and provide discussions on
potential future directions towards more efficient KD methods.111Our code is
available at https://github.com/lancopku/DynamicKD

##  1 Introduction

Knowledge distillation (KD) (Hinton et al., 2015) aims to transfer the
knowledge from a large teacher model to a small student model. It has been
widely used (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2019) to
compress large-scale pre-trained language models (PLMs) like BERT (Devlin et
al., 2019) and RoBERTa (Liu et al., 2019) in recent years. By knowledge
distillation, we can obtain a much smaller model with comparable performance,
while greatly reduce the memory usage and accelerate the model inference.

Although simple and effective, existing methods usually conduct the KD
learning procedure statically, e.g., the student model aligns its output
probability distribution to that of a selected teacher model on the entire
pre-defined corpus. In other words, the following three aspects of KD are
specified in advance and remain unchanged during the learning procedure: (1)
the teacher model to learn from (learning target); (2) the data used to query
the teacher (learning material); (3) the objective functions and the
corresponding weights (learning method). However, as the student competency
evolves during the training stage, it is particularly unreasonable to pre-
define these learning settings and keep them unchanged. Conducting KD
statically may lead to (1) unqualified learning from a too large teacher, (2)
repetitive learning on instances that the student has mastered, and (3) sub-
optimal learning on alignments that are unnecessary. This motivates us to
explore an interesting problem: whether a dynamic KD framework considering the
student competency evolution during training can bring benefits, regarding the
student performance and learning efficiency?

Figure 1: The three aspects of dynamic knowledge distillation explored in this
paper. Best viewed in color.

In this paper, we propose a dynamic knowledge distillation (Dynamic KD)
framework, which attempts to empower the student to adjust the learning
procedure according to its competency. Specifically, inspired by the success
of active learning (Settles, 2009), we take the prediction uncertainty, e.g.,
the entropy of the predicted classification probability distribution, as a
proxy of the student competency. We strive to answer the following research
questions: (RQ1) Which teacher is proper to learn as the student evolves?
(RQ2) Which data are actually useful for student models in the whole KD stage?
(RQ3) Does the optimal learning objective change in the KD process? In
particular, we first explore the impact of the teacher size to dynamic
knowledge distillation. Second, we explore whether dynamically choosing
instances that the student is uncertain for KD can lead to a better
performance and training efficiency trade-off. Third, we explore whether the
dynamic adjustment of the supervision from alignment of prediction probability
distributions and the alignment of intermediate representations in the whole
KD stage can improve the performance.

Our experimental results show that: (1) A larger teacher model with more
layers may raise a worse student. We show that selecting the proper teacher
model according to the competency of the student can improve the performance.
(2) We can achieve comparable performance using only 10% informative instances
selected according to the student prediction uncertainty. These instances also
evolve during the training as the student becomes stronger. (3) We can boost
the student performance by dynamically adjusting the supervision from
different alignment objectives of the teacher model.

Our observations demonstrate the limitations of the current static KD
framework. The proposed uncertainty-based dynamic KD framework only makes the
very first attempt, and we are hoping this paper can motivate more future
research towards more efficient and adaptive KD methods.

##  2 Background: Knowledge Distillation

Given a student model $S$ and a teacher model $T$, knowledge distillation aims
to train the student model by aligning the outputs of the student model to
that of the teacher. For example, Hinton et al. (2015) utilize the teacher
model outputs as soft targets for the student to learn. We denote $S(x)$ and
$T(x)$ as the output logit vector of the student and the teacher for input
$x$, respectively. The KD can be conducted by minimizing the Kullback-Leibler
(KL) divergence distance between the student and teacher prediction:

$\mathcal{L}_{KL}=\text{KL}\left(\sigma\left(S\left(x\right)/\tau\right)||\sigma\left(T\left(x\right)/\tau\right)\right),$
(1)

where $\sigma(\cdot)$ denotes the softmax function and $\tau$ is a temperature
hyper-parameter. The student parameters are updated according to the KD loss
and the original classification loss, i.e., the cross-entropy over the ground-
truth label $y$:

$\displaystyle\mathcal{L}_{CE}$
$\displaystyle=-y\log\sigma\left(S\left(x\right)\right),$ (2)
$\displaystyle\mathcal{L}$
$\displaystyle=(1-\lambda_{KL})\mathcal{L}_{CE}+\lambda_{KL}\mathcal{L}_{KL},$
(3)

where $\lambda_{KL}$ is the hyper-parameter controlling the weight of
knowledge distillation objective. Recent explorations also find that
introducing KD objectives of alignments between the intermediate
representations (Romero et al., 2015; Sun et al., 2019) and attention map
(Jiao et al., 2020; Wang et al., 2020) is helpful. Note that conventional KD
framework is static, i.e., the teacher model is selected before KD and the
training is conducted on all training instances indiscriminately according to
the pre-defined objective and the corresponding weights of different
objectives. However, it is unreasonable to conduct the KD learning procedure
statically as the student model evolves during the training. We are curious
whether adaptive adjusting the settings on teacher adoption, dataset selection
and supervision adjustment can bring benefits regarding student performance
and learning efficiency, motivating us to explore a dynamic KD framework.

##  3 Dynamic Knowledge Distillation

In this section, we introduce the dynamic knowledge distillation. The core
idea behind is to empower the student to adjust the learning procedure
according to its current state, and we investigate the three aspects
illustrated in Figure 1.

Method | RTE | IMDB | CoLA | Avg.
---|---|---|---|---
BERT${}_{\textsc{base}}$ | 67.8 | 89.1 | 54.2 | 70.4
BERT${}_{\textsc{large}}$ | 72.6 | 90.4 | 60.1 | 74.4
No KD | 63.7 | 86.3 | 39.0 | 63.0
KD w/ BERT${}_{\textsc{base}}$ | 64.9 | 86.9 | 39.4 | 63.7
KD w/ BERT${}_{\textsc{large}}$ | 64.5 | 86.5 | 38.2 | 63.1
KD w/ Ensemble | 64.9 | 86.7 | 39.9 | 63.8
Uncertainty-Hard | 66.9∗ | 86.3 | 42.7∗ | 65.3
Uncertainty-Soft | 66.4∗ | 87.1∗ | 41.0 | 64.8
Table 1: We find that bigger teacher with better performance raises a worse
student model. Results are average of 3 seeds on the validation set. ∗ denotes
statistically significant improvement over the best performing baseline with
$p<0.05$.

###  3.1 Dynamic Teacher Adoption

The teacher model plays a vital role in KD, as it provides the student soft-
targets for helping the student learn the relation between different classes
(Hinton et al., 2015). However, there are few investigations regarding how to
select a proper teacher for the student in KD for PLMs during the training
dynamically. In the KD of PLMs, it is usually all teacher models are with the
same model architecture, i.e., huge Transformer (Vaswani et al., 2017) model.
Thus, the most informative factor of teacher models is their model size, e.g.,
the layer number of the teacher model and the corresponding hidden size. This
motivates us to take a first step to explore the impact of model size.

####  3.1.1 Bigger Teacher Not Always Raises Better Student

Specifically, we are curious about whether learning from a bigger PLM with
better performance can lead to a better distilled student model. We conduct
probing experiments to distill a 6-layer student BERT model from
BERT${}_{\textsc{base}}$ with 12 layers, and BERT${}_{\textsc{large}}$ with 24
layers, respectively. We conducts the experiment on two datasets, RTE
(Bentivogli et al., 2009) and CoLA (Warstadt et al., 2019), where two teacher
models exhibit clear performance gap, and a sentiment classification benchmark
IMDB (Maas et al., 2011). Detailed experimental setup can be found in Appendix
A.

As shown in Table 1, we surprisingly find that while the
BERT${}_{\textsc{large}}$ teacher clearly outperforms the small
BERT${}_{\textsc{base}}$ teacher model, the student model distilled by the
BERT${}_{\textsc{base}}$ teacher achieves better performance on all three
datasets. This phenomenon is counter-intuitive as a larger teacher is supposed
to provide better supervision signal for the student model. We think that
there are two possible factors regarding the size of teacher model that
leading to the deteriorated performance:

(1) The predicted logits of the teacher model become less soft as the teacher
model becomes larger and more confident about its prediction (Guo et al.,
2017; Desai and Durrett, 2020), which decreases the effect of knowledge
transfer via the soft targets. We find that a smaller $\tau$ also leads to a
decreased performance of the student model, indicating the the less-softened
teacher prediction will decrease the student performance.222Refer to Appendix
B for details.

(2) The capacity gap between the teacher and student model increases as the
teacher becomes larger. The competency of the student model can not match that
of the large teacher model, which weakens the performance of KD.

To explore the combined influence of these factors, we distill student models
with different layers and plot the performance gain compared to directly
training the student model without distillation in Figure 2. It can be found
that by decreasing the student size, the better supervision from teacher model
boosts the performance, while the two counteractive factors dominate as the
gap becomes much larger, decreasing the performance gain. We notice that this
phenomenon is also observed by Mirzadeh et al. (2020) in computer vision tasks
using convolutional networks, showing that it is a widespread issue and needs
more in-depth investigations. Note that BERT${}_{\textsc{base}}$ and
BERT${}_{\textsc{large}}$ also differs from the number of hidden size, the
experiments regarding the hidden size, where the phenomenon also exists and
corresponding results can be found in Appendix C.

Figure 2: Performance gain of distilled student model with various layer
sizes. The teacher model is BERT${}_{\textsc{base}}$ with 12 layers.

####  3.1.2 Uncertainty-based Teacher Adoption

Our preliminary observations demonstrate that selecting a proper teacher model
for KD is significant for the student performance. While the capacity gap is
an inherent problem once the teacher and the student are set, we are curious
about whether dynamically querying the proper teacher according to the student
competency during training can make the full use of teacher models. Without
loss of generality, we conduct KD to train a student model from two teacher
models with different numbers of Transformer layers. We assume that during the
initial training stage, the student can rely more on the small teacher model,
while turns to the large teacher for more accurate supervision when it becomes
stronger. Specifically, we propose to utilize the student prediction
uncertainty as a proxy of the competency, inspired by the successful
applications in active learning (Settles, 2009), and design two uncertain-
based teacher adoption strategies:

Hard Selection: The instances in one batch are sorted according to the student
prediction uncertainty, i.e., the entropy of predicted class distribution.
Then the instances are evenly divided into instances that the student most
uncertain about and instances that model is most confident about. For the
uncertain part, the small teacher is queried for supervision signals, while
the large teacher provides the soft-label for the instances that the student
is confident about.

Soft Selection: The corresponding KD loss weights from two teachers are
adjusted softly at instance-level. Formally, given two teacher model $T_{1}$
(BERT${}_{\textsc{base}}$, in our case) and $T_{2}$
(BERT${}_{\textsc{large}}$), we can re-write the KD objective in the multiple
teacher setting as:

$\mathcal{L}_{KD}=w_{1}\mathcal{L}^{T_{1}}_{KL}+w_{2}\mathcal{L}^{T_{2}}_{KL}$
(4)

where $\mathcal{L}^{T}_{KL}$ denotes matching loss of the output logits of the
student model and the teacher model $t$. The $w_{1}$ and $w_{2}$ controls the
relative contribution of the supervisions from the two teachers. We adaptively
down-weight the supervision from the large teacher when the student are
uncertain about the training instances. The prediction uncertainty is adopted
as a measurement of the student competency for instance $x$:

$u_{x}=\text{Entropy}\left(\sigma\left(S\left(x\right)\right)\right)$ (5)

where $\sigma$ is a normalization function, e.g, softmax function for mapping
the logit vector to probability distribution. The $w_{1}$ and $w_{2}$ are
adjusted as follows:

$\displaystyle w_{1}=\frac{u_{x}}{U},\quad w_{2}=1-\frac{u_{x}}{U}$ (6)

where $U$ is a normalization factor which re-scales the weight to $[0,1]$. In
this way, the student will pay more attention to the small teacher when it is
uncertain about the current instance, while relies on the large teacher when
it is confident about its prediction.

####  3.1.3 Experiments

##### Settings

We conduct experiments to distill a $6$-layer student model from
BERT${}_{\textsc{base}}$ and BERT${}_{\textsc{large}}$, on RTE, CoLA and IMDB,
following settings of probing analysis in Section 3.1.1.

##### Results

The results of the proposed selection strategies are shown in Table 1. We
observe that the hard selection strategy achieves an overall 65.3 accuracy
which outperforms directly learning from the ensemble of two teacher models.
This demonstrates that the proposed strategy is effective by selecting the
proper teacher model to learn. The soft selection strategy also outperforms
the baseline while falls little behind with the hard version. We attribute it
to that the provided supervisions of two teachers are of different softness,
thus may confuse the student model.

###  3.2 Dynamic Data Selection

The second research question we want to explore is which data will be more
beneficial for the performance of the student. As the distillation proceeds,
the student is becoming stronger, thus the repetitive learning on the
instances those the student has mastered on can be eliminated. If there are
such instances that are vital for the learning of the student model, can we
only conduct KD on these instances, for improving the learning efficiency?
Besides, do the vital instances remain static or also evolve with the student
model? These questions motivate us to explore the effect of dynamically
selecting instances.

####  3.2.1 Uncertainty-based Data Selection

We propose to actively select informative instances according to student
prediction uncertainty, inspired by the successful practice of active learning
(Settles, 2009). Formally, given $N$ instances in one batch, for each instance
$x$, the corresponding output class probability distribution over the class
label $y$ of the student model is $P(y\mid
x)=\sigma\left(S\left(x\right)\right)$. We compute an uncertainty score
$u_{x}$ for $x$ using the follow strategies with negligible computational
overhead:

Entropy (Settles, 2009), which measures the uncertainty of the student
prediction distribution:

$u_{x}=-\sum_{y}P\left(y\mid x\right)\log P\left(y\mid x\right).$ (7)

Margin, which is computed as the margin between the first and second most
probable class $y_{1}^{*}$ and $y_{2}^{*}$:

$u_{x}=P\left(y_{1}^{*}\mid x\right)-P\left(y_{2}^{*}\mid x\right).$ (8)

Least-Confidence (LC), which indicates how uncertain the model about the
predicted class $\hat{y}=\arg\max_{y}P(y\mid x)$:

$\displaystyle u_{x}$ $\displaystyle=1-P\left(\hat{y}\mid x\right)$ (9)

We rank the instances in a batch according to its prediction uncertainty, and
only choose the top $N\times r$ instances to query the teacher model, where
$r\in(0,1]$ is the selection ratio controlling the number to query. Note that
in binary classification tasks like IMDB, the selected subsets using the above
strategies are identical. We also design a Random strategy that selects
$N\times r$ instances randomly, to serve as a baseline for evaluating the
effectiveness of selection strategies.

Method | #FLOPs | SST-5 | IMDB | MRPC | MNLI-m / mm
---|---|---|---|---|---
BERT${}_{\textsc{base}}$ (Teacher) | - | 52.0 | 89.1 | 86.8 | 84.0 / 84.4
Vanilla KD | 45.1B | 47.4 | 86.8 | 80.2 | 81.7 / 82.0
Random | 22.6B | 46.8 | 86.4 | 79.7 | 81.4 / 81.6
Uncertainty-Entropy | 28.2B | 46.7 | 86.8 | 79.4 | 81.5 / 82.0
Uncertainty-Margin | 28.2B | 46.6 | 86.8 | 79.4 | 81.4 / 81.9
Uncertainty-LC | 28.2B | 46.5 | 86.8 | 79.4 | 81.4 / 81.9
$\Delta$ | - | \- 0.6 | 0.0 | \- 0.5 | \- 0.2 / 0.0

Table 2: Dynamic data selection results with $r$ set to $0.5$. Results are
averaged of $3$ seeds on the validation set. $\Delta$ denotes the minimal
performance degradation of different selection strategies compares to vanilla
KD.

Dataset | # Train | # Aug Train | # Dev | # Test | # Class
---|---|---|---|---|---
SST-5 | 8.8k | 176k | 1.1k | 2.2k | 5
IMDB | 20k | 400k | 5k | 25k | 2
MNLI | 393k | 786,0k | 20k | 20k | 3
MRPC | 3.7k | 74k | 0.4k | 1.7k | 2
RTE | 2.5k | 50k | 0.3k | 3k | 2
CoLA | 8.5k | 170k | 1k | 1k | 2

Table 3: Statistics of datasets. # Aug Train denotes the number of the augmented training dataset following Jiao et al. (2020).  Method | #FLOPs | SST-5 | IMDB | MRPC | MNLI-m / mm |  Avg. ($\uparrow$) |  $\Delta$ ($\downarrow$)
---|---|---|---|---|---|---|---
BERT${}_{\textsc{base}}$ (Teacher) | - | 53.7 | 88.8 | 87.5 | 83.9 / 83.4 | 79.5 | -
$\text{TinyBERT}^{\dagger}$ | 24.9B | - | - | 86.4 | 82.5 / 81.8 | - | -
TinyBERT | 24.9B | 51.4 | 87.6 | 86.2 | 82.6 / 82.0 | 78.0 | 0.0
Random | 2.49B | 51.1 | 87.0 | 83.3 | 80.8 / 80.5 | 76.5 | 1.5
Uncertainty-Entropy | 4.65B | 51.5 | 87.7 | 86.5 |  81.8 / 81.0 | 77.7 | 0.3
Uncertainty-Margin | 4.65B | 51.6 | 87.7 | 86.5 | 81.6 / 81.1 | 77.7 | 0.3
Uncertainty-LC | 4.65B | 51.2 | 87.7 | 86.5 | 81.4 / 80.8 | 77.5 | 0.5
Table 4: Test results when the selection ratio $r=0.1$ for dynamic data
selection on various tasks. #FLOPs denotes the average computational cost of
KD for each instance. $\dagger$ denotes results from Jiao et al. (2020).

####  3.2.2 Experiments

##### Settings

We conduct the investigation experiments on two sentiment classification
datasets IMDB (Maas et al., 2011) and SST-5 (Socher et al., 2013), and natural
language inference tasks including MRPC (Dolan and Brockett, 2005) and MNLI
(Williams et al., 2018). The statistics of dataset and the implementation
details can be found in Table 3 and D, respectively. We report accuracy as the
performance measurement for all the evaluated tasks.

Besides, we also provide the corresponding computational FLOPs for comparing
the learning efficiency. In more detail, we divide the computational cost $C$
of KD tinto three parts: student forward $F_{s}$, teacher forward $F_{t}$ and
student backward $B_{s}$ for updating parameters. Note that $F_{s}\approx
B_{s}\ll F_{t}$, as the teacher model is usually much larger than the student
model. By actively learning only from $N\times r$ instances that the student
are most uncertain about, the cost is reduced to:

$C^{\prime}=N\times F_{s}+N\times r\times B_{s}+N\times r\times F_{t}.$ (10)

For example, the number of computational FLOPs of a 6-layer student BERT model
is $11.3$B while that of a 12-layer BERT teacher model is $22.5$B (Jiao et
al., 2020). When $r$ is set to $0.1$, the total KD cost is reduced from
$45.1$B to $14.7$B.

##### Results with Original Dataset

The results when $r$ set to $0.5$ are listed in Table 2. Overall, it can be
found that selecting the instances only lead negligible degradation of the
student performance, compared to that of Vanilla KD, showing the effectiveness
of the uncertainty-based strategies. Interestingly, the random strategy
perform closely to the uncertainty-based strategies, which we attribute to
that the underlying informative data space can also be covered by random
selected instances. Besides, we notice that performance drop is smaller on the
tasks with larger training data size. For example, selecting informative
instances with prediction entropy leads to 0.2 accuracy drop on the MNLI
dataset consisting $393$k training instances, while causes 0.8 performance
drop on MRPC with $3.7$k instances. A possible reason is that for the tiny
dataset, the underlying data distribution is not well covered by the training
data, therefore further down-sampling the training data results in a larger
performance gap. To verify this, we turn to the the setting where the original
training dataset is enriched with augmentation techniques.

##### Results with Augmented Dataset

Following TinyBERT (Jiao et al., 2020), we augment the training dataset 20
times with BERT mask language prediction, as it has been prove effective for
distilling a powerful student model. Our assumption is that with the data
augmentation technique, the training set can sufficiently cover the possible
data space, thus selecting the informative instances will not lead to
significant performance drop. Besides, it is of great practical value to
accelerate the KD procedure via reducing the queries to teacher model on the
augmented dataset. For example, it costs about $\$3,709$ for querying all the
instances of the augmented MNLI dataset as mentioned by Krishna et al.
(2019).333The cost is estimated according to Google Cloud natural language
API: https://cloud.google.com/natural-language/pricing. By only querying a
small portion of instances to the teacher model, we can greatly reduce
economic cost and ease the possible environmental side-effects (Strubell et
al., 2019; Schwartz et al., 2019; Xu et al., 2021) that may hinder the
deployments of PLMs on downstream tasks.

Figure 3: We plot the mean accuracy on the validation set of 3 seeds ($\pm$
one standard deviation) under different selection ratios of various
strategies. Orange dashed line denotes the performance of vanilla KD.

The results with TinyBERT-4L as the backbone model and $r=0.1$ are listed in
Table 4. We can observe that uncertainty-based selection strategy can maintain
the superior performance while saving the computational cost, e.g., the FLOPs
is reduced from $24.9$B to $4.65$B with negligible average performance
decrease. In tasks like SST-5 and IMDB, selecting $10\%$ most informative
instances according to student prediction entropy can even outperform the
original TinyBERT using the whole dataset. Among these strategies, the least-
confidence strategy performs relatively poor, as it only takes the maximum
probability into consideration while neglects the full output distribution.

##### Performance under Different Ratios

We vary the selection ratio $r$ to check the results of different strategies
on the augmented SST-5 dataset. The results are shown in Figure 3. Our
observations are: (1) There exists a trade-off between the performance and the
training costs, i.e., increasing the selection ratio generally improves the
performance of student model, while results in bigger training costs. (2) We
can achieve the full performance using about 20% training data. It indicates
that the training data support can be well covered with about $20$% data, thus
learning from these instances can sufficiently train the student model. It
validates our motivation to select informative instances for reducing the
repetitive learning caused by data redundancy. (3) Selection strategies based
on the uncertainty of student prediction can make the better use of limited
query, performing better than the random selection, especially when the query
number is low.

####  3.2.3 Analysis

We further conduct experiments on the augmented SST-5 dataset to gain insights
about the property of selected instances and visualize the distribution of
selected instances for intuitive understanding.

##### Property of Selected Instances

We plot the teacher prediction entropy and the distance from the selected
instances to the corresponding category center. From Figure 4 (left), we
observe for hard instances with high uncertainty that selected by the student
model, the teacher model also regards them as difficult. It indicates that the
instance difficulty is an inherent property of data and uncertainty-based
criterion can discover these hard instances from the whole dataset. Besides,
the teacher’s entropy of selected instances increases as the training
proceeds, showing that the selected instances also evolve during the training
as the student is becoming stronger. The right part in Figure 4 demonstrates
that uncertainty-based selection will pick up the instances that are far away
from the category center than the ones are randomly picked. These instances
are more informative for the student model to learn the decision boundary of
different classes.

##### Visualization of Selected Instances

Figure 4: Training dynamics comparison of the selected instances. The entropy
strategy can distinguish more informative instances from the whole dataset.

(a) Random (b) Margin

Figure 5: The t-SNE visualization of instance representations. Uncertainty-
based strategies select the instances close to the class boundary, which is
useful for the learning of the student model. Best viewed in color.

We further visualize the distribution of instances in the feature space, i.e.,
the representation before the classifier layer, using t-SNE (Maaten and
Hinton, 2008) and highlight the selected instances with the cross marker. We
compare the best performing strategy margin and random selection on SST-5. As
shown in Figure 5, the instances randomly selected are distributed uniformly
in feature space. The margin strategy instead picks the instances close to the
classification boundary. The results demonstrate that the uncertainty-based
selection criterion can help the student model pay more attention to the
instances that are vital for making correct predictions, thus achieving a
comparable performance with a much lower computational cost.

In all, our analysis experiments show that the uncertainty-based selection is
effective for picking instances that are close to the classification boundary.
Besides, the selected instances also evolve as the student model becomes
stronger. By learning from these instances, the computational cost of KD is
greatly reduced with a negligible accuracy drop.

###  3.3 Dynamic Supervision Adjustment

We finally explore the question of the optimal learning objective functions.
Previous studies have shown that integrating the alignments on the
intermediate representation (Romero et al., 2015; Sanh et al., 2019; Sun et
al., 2019) and attention map (Jiao et al., 2020; Wang et al., 2020) between
the student and the teacher model can further boost the performance. We are
interested in whether the dynamic adjustment of the supervision from different
alignment objectives can bring extra benefits. As the first exploration, we
only consider the combination of the KL-divergence distance with the teacher
prediction and the hidden representation alignments:

$\mathcal{L}_{KD}=\lambda_{KL}*\mathcal{L}_{KL}+\lambda_{PT}*\mathcal{L}_{PT}$
(11)

where $\mathcal{L}_{PT}$ is called PaTient loss, which measures the alignment
between the normalized internal representations of the teacher and student
model (Sun et al., 2019):

$\mathcal{L}_{PT}=\sum_{i=1}^{M}\left\|\frac{\mathbf{h}_{i}^{s}}{\left\|\mathbf{h}_{i}^{s}\right\|_{2}}-\frac{\mathbf{h}_{I_{pt}(j)}^{t}}{\left\|\mathbf{h}_{I_{pt}(j)}^{t}\right\|_{2}}\right\|_{2}^{2}$
(12)

where $M$ is the number of student layer, $I_{pt}(i)$ denotes the
corresponding alignment of the teacher layer for the student $i$-th layer,
$\mathbf{h}_{i}^{s}$ and $\mathbf{h}_{i}^{t}$ denote representation of $i$-th
layer of student and teacher model, respectively.

####  3.3.1 Uncertainty-based Supervision Adjustment

Different from previous studies which set the corresponding alignment
objective weights via hyper-parameter search and keep them unchanged during
the training, we propose to adjust the weights according to the student
prediction uncertainty for each instance. The motivation behind is that we
assume it is unnecessary to force the student model to align all the outputs
of the teacher model during the whole training stage. As the training
proceeds, the student is become stronger and it may learn the informative
features different from the teacher model. Therefore, there is no need to
force the student to act exactly with the teacher model, i.e., requiring the
intermediate representations of the student to be identical with the teacher.
Formally, we turn the weight of KD objective into a function of the student
prediction uncertainty
$u(x)=\text{Entropy}\left(\sigma\left(S\left(s\right)\right)\right)$:

$\displaystyle\lambda_{KL}$
$\displaystyle=\lambda^{*}_{KL}(1-\frac{u_{x}}{U}),\quad\lambda_{PT}=\lambda^{*}_{PT}\frac{u_{x}}{U}$
(13)

where $\lambda^{*}_{KL}$ and $\lambda^{*}_{PT}$ are pre-defined weight for
each objective obtained by parameter search and $U$ is the normalization
factor. In this way, the contribution of these two objectives are adjusted
dynamically during the training for each instances. For instances that the
student is confident about, the supervision from the internal representation
alignment is down-weighted. Thus the student is focusing mimicking the final
prediction probability distribution with the teacher based on its own
understanding of the instance. On the contrary, for instances that the student
is confusing, the supervision from teacher model representations can help it
learn the feature of the instance better.

####  3.3.2 Experiments

##### Settings

The student model is set to 6-layer and BERT${}_{\textsc{base}}$ is adopted as
the teacher model. For intermediate layer representation alignment, we adopt
the Skip strategy, i.e., $I_{pt}=\\{2,4,6,8,10\\}$ as it performs best as
described in BERT-PKD. We conduct experiments on the sentiment analysis task
SST-5, and two natural language inference tasks MRPC and RTE. For
$\lambda^{*}_{KL}$ and $\lambda^{*}_{PT}$, we adopt the searched parameters
provided by Sun et al. (2019).

Method | SST-5 | MRPC | RTE | Avg.
---|---|---|---|---
BERT${}_{\textsc{base}}$ (Teacher) | 52.0 | 86.8 | 67.8 | 68.9
Vanilla KD | 47.4 | 80.2 | 64.9 | 64.2
BERT-PKD | 46.6 | 80.8 | 65.1 | 64.2
Uncertainty | 48.1 | 81.5∗ | 66.4∗ | 65.3
Table 5: Results of dynamic adjusting the supervision weights, showing the
uncertainty-based adjustment is effective. ∗ denote results are statistically
significant with $p<0.05$.

##### Results

The results of adaptive adjusting the supervision weights are listed in Table
5. We observe that the proposed uncertainty-based supervision adjustment can
outperform the static version BERT-PKD on all the tasks, showing that the
proper adjustment of the KD objectives is effective for improving the student
performance. We also plot the batch average of the KL loss weight in Figure 6.
As expected, the corresponding weight of the prediction probability alignment
objective is increasing as the student becomes more confident about its
predictions, thus paying more attention to matching the output distribution
with the teacher model. Interestingly, we find that at the initial stage of
training, the KL weight is decreasing. It indicates that the learning by
aligning the intermediate representations can help the student quickly gain
the understanding the task, thus improving the confidence of predictions.

Figure 6: Evolution of the dynamically adjusted weight of KL-divergence loss
weight.

##  4 Discussions

After the preliminary explorations on the three aspects of Dynamic KD, we
observe that it is promising for improving the efficiency and the distilled
student performance. Here we provide potential directions for further
investigations.

(1) From uncertainty-based selection criterion to advanced methods. In this
paper, we utilize student prediction uncertainty as a proxy for selecting
teachers, training instances and supervision objectives. More advanced methods
based on more accurate uncertainty estimations (Gal and Ghahramani, 2016; Zhou
et al., 2020), clues from training dynamics (Toneva et al., 2018), or even a
learnable selector can be developed.

(2) From isolation to integration. As a preliminary study, we only investigate
the three dimensions independently. Future work can adjust these components
simultaneously and investigate the underlying correlation between these three
dimensions for a better efficiency-performance trade-off.

(3) More fine-grained investigations regarding different components in the
Dynamic KD framework: (i) For teacher adoption, exploring whether dynamically
training a student model from more teacher models or teacher models with
different architectures can bring extra benefits; (ii) For data selection, it
will be interesting to investigate whether the informative data is model-
agnostic, and whether dynamically selecting data from different domains can
improve the generalization performance; (iii) For supervision adjustment,
investigations on the effect of combinations of different objectives can be
promising.

##  5 Related Work

Our work relates to recent explorations on applying KD for compressing the
PLMs and active learning.

##### Knowledge Distillation for PLMs

Knowledge distillation (Hinton et al., 2015) aims to transfer the dark
knowledge from a large teacher model to a compact student model, which has
been proved effective for obtaining compact variants of PLMs. Those methods
can be divided into general distillation (Sanh et al., 2019; Turc et al.,
2019; Wang et al., 2020) and task-specific distillation (Sun et al., 2019;
Jiao et al., 2020; Xu et al., 2020; Li et al., 2020a; Liang et al., 2021; Li
et al., 2020b; Wu et al., 2021). The former conducts KD on the general text
corpus while the latter trains the student model on the task-specific
datasets. In this paper, we focus on the latter one as it is more widely
adopted in practice. Compared to existing static KD work, we are the first to
explore the idea of Dynamic KD, making it more flexible, efficient and
effective.

##### Active Learning

(Settles, 2009), where a learning system is allowed to choose the data from
which it learns from, for achieving better performance with fewer labeled
data. Traditional selection strategies include uncertainty-based methods
(Scheffer et al., 2001; Settles, 2009), which select the instances that model
is most uncertain about, query-by-committee (Freund et al., 1997), which
select instances with highest disagreements between a set of classifiers, and
methods based on decision theory (Roy and McCallum, 2001). In this paper,
inspired by the success of active learning, we introduce Dynamic KD that
utilizes the different strategies like prediction entropy as a proxy of
student competency to adaptively adjust the different aspects of KD. Our
explorations show that the uncertainty-based strategies are effective for
improving the efficiency and performance of KD.

##  6 Conclusion

In this paper, we introduce dynamic knowledge distillation, and conduct
exploratory experiments regarding teacher model adoption, data selection and
the supervision adjustment. Our experimental results demonstrate that the
dynamical adjustments on the three aspects according to the student
uncertainty is promising for improving the student performance and learning
efficiency. We provide discussions on the potential directions worth exploring
in the future, and hope this work can motivate studies towards more
environmental-friendly knowledge distillation methods.

## Acknowledgements

We thank all the anonymous reviewers for their constructive comments and
Xuancheng Ren for his valuable suggestions in preparing the manuscript. This
work was supported by a Tencent Research Grant. Xu Sun is the corresponding
author of this paper.

## References

  * Bentivogli et al. (2009) Luisa Bentivogli, Ido Kalman Dagan, Dang Hoa, Danilo Giampiccolo, and Bernardo Magnini. 2009.  The fifth pascal recognizing textual entailment challenge.  In _TAC Workshop_.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  In _EMNLP_ , pages 295–302, Online.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , pages 4171–4186.
  * Dolan and Brockett (2005) William B. Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _Proceedings of the Third International Workshop on Paraphrasing (IWP)_.
  * Freund et al. (1997) Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby. 1997.  Selective sampling using the query by committee algorithm.  _Machine learning_ , 28(2):133–168.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _ICML_ , pages 1050–1059.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _ICML_ , pages 1321–1330.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  TinyBERT: Distilling BERT for natural language understanding.  In _Findings of EMNLP_ , pages 4163–4174.
  * Krishna et al. (2019) Kalpesh Krishna, Gaurav Singh Tomar, Ankur P Parikh, Nicolas Papernot, and Mohit Iyyer. 2019.  Thieves on sesame street! Model extraction of BERT-based APIs.  In _ICLR_.
  * Li et al. (2020a) Jianquan Li, Xiaokang Liu, Honghong Zhao, Ruifeng Xu, Min Yang, and Yaohong Jin. 2020a.  BERT-EMD: Many-to-many layer mapping for BERT compression with earth mover’s distance.  In _EMNLP_ , pages 3009–3018.
  * Li et al. (2020b) Lei Li, Yankai Lin, Shuhuai Ren, Deli Chen, Xuancheng Ren, Peng Li, Jie Zhou, and Xu Sun. 2020b.  Accelerating pre-trained language models via calibrated cascade.  _arXiv preprint arXiv:2012.14682_.
  * Liang et al. (2021) Kevin J. Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. 2021.  MixKD: Towards efficient distillation of large-scale language models.  In _ICLR_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _ICLR_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _ACL_ , pages 142–150.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _JMLR_ , 9:2579–2605.
  * Mirzadeh et al. (2020) Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020.  Improved knowledge distillation via teacher assistant.  In _AAAI_ , pages 5191–5198.
  * Romero et al. (2015) Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. 2015.  FitNets: Hints for thin deep nets.  In _ICLR_.
  * Roy and McCallum (2001) Nicholas Roy and Andrew McCallum. 2001.  Toward optimal active learning through sampling estimation of error reduction.  In _ICML_ , pages 441–448.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  In _NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing_.
  * Scheffer et al. (2001) Tobias Scheffer, Christian Decomain, and Stefan Wrobel. 2001.  Active hidden markov models for information extraction.  In _International Symposium on Intelligent Data Analysis_ , pages 309–318. Springer.
  * Schwartz et al. (2019) Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2019.  Green AI.  _arXiv preprint arXiv:1907.10597_.
  * Settles (2009) Burr Settles. 2009.  Active learning literature survey.  Computer Sciences Technical Report 1648, University of Wisconsin–Madison.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _EMNLP_ , pages 1631–1642.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in NLP.  In _ACL_ , pages 3645–3650.
  * Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019.  Patient knowledge distillation for BERT model compression.  In _EMNLP-IJCNLP_ , pages 4323–4332.
  * Toneva et al. (2018) Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2018.  An empirical study of example forgetting during deep neural network learning.  In _ICLR_.
  * Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Well-read students learn better: The impact of student initialization on knowledge distillation.  _arXiv preprint arXiv:1908.08962_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_ , pages 5998–6008.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  In _NeurIPS_.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _TACL_ , 7:625–641.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _NAACL-HLT_ , pages 1112–1122.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _System Demonstrations, EMNLP_ , pages 38–45.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. 2021.  One teacher is enough? pre-trained language model distillation from multiple teachers.  In _Findings of ACL-IJCNLP 2021_ , pages 4408–4413.
  * Xu et al. (2020) Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020.  BERT-of-Theseus: Compressing BERT by progressive module replacing.  In _EMNLP_ , pages 7859–7869.
  * Xu et al. (2021) Jingjing Xu, Liang Zhao, Junyang Lin, Rundong Gao, Xu Sun, and Hongxia Yang. 2021\.  KNAS: green neural architecture search.  In _ICML_ , pages 11613–11625.
  * Zhou et al. (2020) Yikai Zhou, Baosong Yang, Derek F Wong, Yu Wan, and Lidia S Chao. 2020.  Uncertainty-aware curriculum learning for neural machine translation.  In _ACL_ , pages 6934–6944.

##  Appendix A Teacher Size Exploration Settings

We conduct the knowledge distillation with BERT${}_{\textsc{base}}$ and
BERT${}_{\textsc{large}}$ as teacher models. The student model is set to a
6-layer student BERT. For training the teacher, the teacher model is fine-
tuned using the script provided by Huggingface Transformers library. The fine-
tuning learning rate is $2\text{e-}5$ with a linear warm-up learning rate
schedule for the first $10\%$ training steps. Batch size is $32$, training
epoch is set to $3$, and the max length of input sentence is set to $128$. The
statistics of used datasets are listed in Table 3.

For distilling the student model, the student model is initialized using the
first $6$ layers weights of BERT${}_{\textsc{base}}$. We adopt the KL-
divergence distance as the KD objective. $\lambda_{KL}$ is set to $0.5$ and we
empirically find this setting works well. The same training hyper-parameters
as fine-tuning the teacher model are used for distillation. The performance is
evaluated on the validation set and averaged on $3$ random seeds.

##  Appendix B Impacts of Prediction Smoothness

To examine the influence of less-softened teacher predictions, we conduct
distillation experiments with various temperature $\tau$ using the hyper-
parameters identical with previous experiments, to mimicking the sharpen
impact introduced by the larger teacher size. In more detail, we setup a
student model with $6$ layers as before and select the
BERT${}_{\textsc{base}}$ as the teacher model. The results are illustrated in
Figure 7. We observe on both datasets, the decreased temperature $\tau$ will
lead a performance decrease. It indicates that the less-softened probability
distribution indeed weakens the performance of knowledge distillation.

##  Appendix C Impacts of Teacher Hidden Size

As mentioned in the main paper, we observe that larger teacher may not raise a
student model with better performance. We further conduct experiments
regarding the hidden size of teacher model. Specifically, we setup a student
with $6$ layers with $256$ hidden units. The small teacher and the large
teacher are a BERT model of 12-layer with $256$ hidden units and a BERT of
12-layer with $768$ hidden units, respectively. Out experiments show that on
the CoLA dataset, the student model distilled with the small teacher can
achieve $11.9$ matthews correlation score while that of model distilled by the
large teacher is $8.8$. The result on the IMDB is consistent, i.e., $83.2$
accuracy for student model distilled by the large teacher and $83.4$ accuracy
for the student distilled by the small teacher. These results again verify the
phenomenon that the larger teacher may not always raise a better student
model.

(a) RTE (b) IMDB

Figure 7: Distillation performance with varying temperature $\tau$ on
different datasets.

##  Appendix D Implementation Details

Our implementation is based on PyTorch and Huggingface transformers library.
Model is optimized with AdamW optimizer with linear learning rate warm-up. The
sentence length is set to $64$ for SST-5 and $128$ for the rest datasets. Our
teacher model is BERT${}_{\textsc{base}}$. The model is trained with learning
rate $2\text{e-}5$ and batch size $32$ for $3$ epochs. $\lambda_{KL}$ is set
to $0.5$, with temperature $\tau$ set to $1$.

For experiments using TinyBERT, we select $\text{TinyBERT}_{4}$ v2 as our
backbone model, and conduct the general distillation for $10$ epochs on the
augmented dataset. We further train the model for $3$ epochs on the augmented
dataset and choose learning rates from
$\\{1\text{e-}5,2\text{e-}5,3\text{e-}5\\}$ and batch sizes from $\\{16,32\\}$
based on the performance on the validation set. The performance are evaluated
on the test sets and $\tau$ is set to $1$, following the practice of TinyBERT.


# CALaMo: a Constructionist Assessment of Language Models

Ludovica Pannitto
CIMeC
University of Trento
<EMAIL_ADDRESS>
&Aurélie Herbelot
CIMeC/DISI
University of Trento
<EMAIL_ADDRESS>

###### Abstract

This paper presents a novel framework for evaluating Neural Language Models’
linguistic abilities using a constructionist approach. Not only is the usage-
based model in line with the underlying stochastic philosophy of neural
architectures, but it also allows the linguist to keep meaning as a
determinant factor in the analysis. We outline the framework and present two
possible scenarios for its application.

##  1 Introduction

Over the years, linguists have given a lot of thought to what language is, and
how it can be best formally described. Different approaches with sometimes
contradictory aims have produced an extremely rich array of conceptual tools
to describe linguistic phenomena. Such tools play diverse roles in explaining
the phylogenetic, ontogenetic or historical-cultural facets of language and
are often heavily interlaced with one another. In this research landscape,
computational modelling has largely been used to simulate and investigate
speaker behaviour at various levels of granularity. A specific area within the
computational community is known as (Neural) Language Modelling, which aims at
reproducing linguistic surface structure by means of (pseudo)-probabilistic
models. Neural architectures have played a special role in this subfield of
research, due to their flexibility.

The extreme complexity of theoretical tools found in linguistics gets cut down
by order of magnitudes when it comes to the analysis of language processing
using computational modelling. For instance, when the term language is
mentioned in relation to Artificial Neural Networks, it seems that the word is
often used as a mere synonym of grammar: while it is clear from a broader
theoretical perspective that the two objects do not overlap, the distinction
gets blurred in many computational studies. That is, assumptions which would
be clearly stated in theoretical linguistics (e.g. how grammatical abstraction
fits into the concept of language), are not explicitly discussed by
computational studies: it is often the case that a specific set of choices
concerning the description of language are taken as default. Most current work
also seems to implicitly make a number of assumptions about what kind of
grammar is supposed to emerge from neural language models (henceforth, NLMs),
and this underlying choice is often echoed in the most common evaluation
settings and in the conclusions that are being drawn from such experiments.
Most of these default assumptions are inherited from the nativist Chomskian
tradition and the Universal Grammar (UG) framework (Chomsky, 1986; Smith and
Allott, 2016), which has pervaded a lot of the computational work on grammar,
and continues to do so in the recent literature on neural models.

Ironically, the nativist assumptions that permeate the mainstream
computational methodology are at odds with the very nature of the models
created by the field. Neural models are essentially based on pattern learning
and are completely agnostic about the nature of the data they are made to
process. The idea that language can be abstracted from a general purpose
statistical mechanism is more akin to usage-based (henceforth, UB) approaches
Barlow and Kemmer (2000); Goldberg (2003); Tomasello (2003), and NLMs would
provide a much more natural testbed for that theoretical strand. In the
cognitive and UB accounts, the exploitation of predictability during language
development (and again we refer to development at all the three tiers of
philogeny, ontogeny and cultural evolution) is the root of a number of
fundamental mechanisms such as schematization, entrenchment and distributional
analysis (Lewkowicz et al., 2018). In the light of these processes, language,
seen as a structured inventory of constructions, gets build through
generations (Cornish et al., 2017) and throughout a speaker’s lifetime: shared
linguistic material among utterances, such as morphological markers for
instance, enable the identification of particular patterns or constructions as
units bearing meaning (Croft, 2001).

The perceived gap between the nativist and non-nativist traditions with
respect to computational modelling probably stems from historical factors. The
Chomskian school and its formal approach offered a definition of language
that, in the past, could easily be interpreted and implemented by emergent
computational approaches. But there is no reason for this bias to perdure. In
this paper, we argue in favour of a usage-based framework to analyse language
acquisition in ANNs. We first point out the aspects of nativist theories that
have so far influenced the evaluation of NLMs (§2). We then introduce a
framework for a quantitative and qualitative analysis of NLMs linguistic
abilities within the constructionist perspective (§3). We finally show some
preliminary analyses performed with the proposed formalization (§4).

##  2 Nativist vs. non-nativist approaches to language acquisition

All theories of language use and development recognize that at the root of
human linguistic ability is the capacity to handle symbolic structures. But
they disagree on the specific content of speakers’ linguistic knowledge, the
mode of acquisition of such content, and the extent to which linguistic
productivity is affected by this stored knowledge (Bannard et al., 2009a).
Theories diverge with respect to three aspects: input, stability and
systematicity. The perspective taken on each of these aspects has consequences
for the conclusions drawn from NLMs’ responses to the evaluation setting. In
the following, we consider each aspect in turn and specifically highlight how
the evaluation of computational models becomes biased due to a lack of
explicitness in relating experimental and theoretical aspects of the research
question.

##### Input.

One of the main arguments introduced by nativist frameworks is the poverty of
the stimulus: the input children are exposed to is underdetermined and does
not explain acquisitional generalizations observed in learners (Crain and
Pietroski, 2001). Such theories assume that children navigate a hypotheses
space defined by innate constraints (Eisenbeiß, 2009). Constructionist
approaches, instead, posit that language emerges from the input through
domain-general mechanisms: this implies that the input is shaped and skewed in
a specific way in order to enhance learnability (Boyd and Goldberg, 2009). A
well established line of research has shown how children are proficient
statistical learners (Gómez and Gerken, 2000; Romberg and Saffran, 2010;
Christiansen, 2019). The emergence of language-like structure from purely
linear signal has also been shown in recent experiments such as Cornish et al.
(2017), which demonstrated how important aspects of the sequential structure
of language may derive from adaptations to the cognitive limitations of human
learners and users (Christiansen and Chater, 2016b). The crucial difference
between the nativist and the non-nativist approach here is how strict the
relation between the received input and the acquired linguistic structure is:
if we commit to a view in which the input only serves as a trigger of an
almost pre-determined cognitive structure, we are naturally driving our
attention far from the features of the input and primarily to the features of
the structure. On the other hand, deriving the linguistic structure from the
input structure itself requires investigating the two aspects together. So
far, most studies on NLMs have disregarded the effect of the input on
experimental results Pannitto and Herbelot (2022).

##### Stability.

The continuity assumption was first introduced by Pinker (1984) in order to
reconcile aspects of developmental language with the generative framework. It
posits that the differences between adult and children linguistic structures
is negligible and merely due to performance factors. In contrast, what we can
refer to as the developmental hypothesis claims that the mechanisms underlying
acquisition remain the same throughout a life-long acquisition process, but
the structures and abstractions they generate evolve over time. UB models also
put emphasis on the linear and time-dependent nature of the linguistic signal
(Christiansen and Chater, 2016b; Cornish et al., 2017). According to the UB
account, generalizations appear gradually, as productivity emerges from item-
specific knowledge (Bannard et al., 2009b).

Another aspect of stability is inter-speaker differences. UG posits that all
speakers eventually converge to the same grammar (Lidz and Williams, 2009;
Crain et al., 2009). Individual differences have however been found in almost
every area of grammar, depending on a variety of factors including
environmental ones (Street and Dąbrowska, 2010). The ‘sameness’ assumption
pervades the computational linguistics literature, where evaluation is
performed according to a single ‘gold standard’ per task. For traditional
tasks such as sentiment analysis or word similarity ratings, the annotations
of human subjects are averaged, and the system is evaluated against the
average. For language modeling, model perplexity is computed with respect to
the statistical features of a large corpus, which aggregates the writing
styles and linguistic habits of thousands of speakers. While this state of
affairs has started to be criticised by various researchers, it remains for
now the status quo. When considering language development as a speaker-
dependent process, strongly affected by the nature of the input, an evaluation
based on an ‘average speaker’ becomes truly unsatisfactory. We cannot assume
the existence of a ground truth, and must rely on softer evaluation measures:
it is clear that the linguistic behaviours of different speakers must overlap
sufficiently to allow for communication, but that we also want to observe in
the output of the network the kind of variability that is seen in humans.

##### Systematicity.

The ability to understand and generate an unbounded number of novel sentences,
using finite means, is considered one of the hallmarks of our language
faculty. The boundaries of this systematicity remain however largely unclear:
provided that we agree on what the finite means at our disposal are, not all
the possibilities are actually realised by speakers and not all realised
possibilities share the same cognitive or linguistic status.

One way to look at systematicity is that of compositionality, for which the
most widely known version is probably due to Katz and Fodor (1963), that port
Chomsky’s innateness theory to semantics: a set of rules or constraints is
needed in order to systematically build the meaning of sentences by
integrating meaning of words. Even the Montagovian formal approach to
compositionality (Montague, 1970) relies on Chomskian-derived ideas of a
stable lexicon that stores meanings, and the existence of a set of precise
interpretation rules that allow for those meanings to be mixed and modulated
through the filter of syntax. The core of both visions is still very much
syntax-centered (to which semantics has to be isomorphic) and very little
space is left for indeterminacy, negotiation between speakers and other
aspects related to the interactive and communicative nature of language
(different individuals can retain in fact quite different concepts associated
to the same lexical label for instance, Labov, 1973). In a nutshell, if we see
systematicity from the standpoint of compositionality, the quasi-regularities
of linguistic structure represents a major hurdle to surpass.

Quasi-regularity is instead the engine of productivity, as in the ability of
speakers to use all the available linguistic means to cue the intended
meaning. Just like compositionality, productivity deals with the domain in
which a grammatical pattern can be employed in a linguistic context without
losing interpretability, and it deals with what is actually possible in the
language and where to draw the boundaries of acceptability. The shift has not
just been syntactic: in the formal representation of these two aspects of
systematicity in semantics, for instance, composition-oriented (Katz and
Fodor, 1963) or productivity-oriented (Fillmore, 1976) theories have
conceptualized the idea of selectional constraints differently.

Knowledge on systematicity is in both cases considered as implicit knowledge
that the speaker has about their language. Nativist approaches have however
primarily dealt with compositionality, and so are NLMs often evaluated: given
grammar rules and lexicon, what are the computational mechanisms that allow
them to combine? UB theories, on the other hand, have primarily been dealing
with productivity: how far can meaning boundaries be forced? What are the
mechanisms that allow for linguistic creativity? This of course entails, in
the UB community, a relation to surface properties of the input as well: Croft
and Alan Cruse (2004), for instance, note how the maximally schematic
constructions, such as sbj verb obj, are also the most productive ones, and
that this has a relation to their frequency too, both as a type and for each
of their instantiations.

##  3 CALaMo

In our proposed methodology, CALaMo (Constructionist Assessment of Language
Models), we incorporate the UB perspective across all three aspects: input,
stability and systematicity.

As far as input is concerned, CALaMo differs from standard approaches by
considering input data an important factor in determining the shape of the
learner’s grammatical knowledge. In traditional scenarios, the input only
serves as a triggering factor and its features play little to no role in the
analysis. From a UB perspective, instead, the relation between the abstract
grammatical structure of the input and the acquired grammar, which then
constrains the production of the learner, is strict.

Regarding stability, depending on the view that is taken on the continuity
hypothesis, we can see NLM’s grammatical competence either as a binary or as a
gradient property. In the first case, we test whether the network is able or
not to handle some linguistic phenomenon, while in the second case, as
advocated by CALaMo, we are interested in seeing how and why some linguistic
aspect becomes more and more salient to the network during training.

The compositionality vs. productivity perspectives, finally, entail a
different organization of linguistic knowledge: the mainstream
compositionality perspective tends to set meaning aside, and treat the lexicon
as an organized repository of meanings (it makes sense therefore to test NLM’s
capabilities on semantically nonsensical sentences or to extend the known
rules to completely unknown lexical items). In the productivity perspective,
instead, meaning is intrinsically part of the process and is treated as a
systematic aspect of grammar, too.

###  3.1 Acquiring language

When talking about NLMs and their linguistic capabilities, the issue of
language acquisition ($A$) is often formalized as how much language $\Lambda$
can be learned by the (artificial) speaker, given a certain level of
computational complexity $C$ by being exposed to a certain type of data $I$:

$A:C\times I\mapsto\Lambda$ (1)

All the components of the equation have been central to the linguistic debate.
However, starting from this basic formalization, we identify two major focus
points that we specifically address in our framework. Firstly, the above
formula describes acquisition as instantaneous, but it is actually better
described as a process $A=(a_{0},a_{1},\cdots a_{N})$ (§3.2). From a cognitive
perspective the process is fully continuous, while in the artificial scenario,
input data is often fed in ‘batches’. We can however imagine that, if we had
the ability to increase the number of steps at will (i.e., make $N$ larger
while keeping constant the amount of data), we could formalize steps small
enough to make the two processes comparable.

Secondly, language is often seen as something that the learner has acquired
and gained knowledge of. We want to bring back in the framework the role of
the linguist-observer, that builds an abstraction over the linguistic behavior
of the speaker (§3.3). As the actual knowledge acquired by the speaker is
undetectable and only explainable metalinguistically, in a way that is not
viable with neural networks (i.e., we cannot ask NLMs what they know about
linguistic regularities), we must take into account the fact that we are
always analyzing both the linguistic input received by the speaker and the
output produced as an effect of the acquisition process through analytical
categories that are created and used by the linguist-observer. In other words,
$\Lambda$ is not a property of the speaker, but rather a function operated by
the linguist-observer. It does not evolve per se during the acquisition
process, but rather it helps us detect and characterize the evolution of the
speaker’s abilities.

###  3.2 The process of acquisition

All the elements of Equation 1 ideally change throughout time as the
acquisition process unfolds.

The input $I$ to which the learner is exposed, in a real-life scenario,
changes continuously. We can therefore define
$I=(\iota_{0},\iota_{1},\cdots,\iota_{N})$, where $\iota_{i}$ is the
collection of input data to which the learner has been exposed to in-between
$a_{i}$ and $a_{i+1}$. Again ideally, with $N$ large enough, each $\iota_{i}$
could even correspond to a single sentence. The computational complexity also
co-evolves with the acquisition function, as linguistic knowledge gets
incorporated into it. In the human case, the initial state is unobservable and
in the artificial scenario it is often not interesting as initialization of
neural models is random. At step $i$, instead, the computational mechanism
that has incorporated knowledge up to step $i-1$ is exposed to $\iota_{i}$.
For these reasons, we define $C=(c_{\emptyset},c_{0},\cdots,c_{N-1})$. As an
effect, $\Lambda$ identifies different subsets
$\lambda_{0},\lambda_{1},\cdots,\lambda_{N}$ throughout the acquisition
process, namely $\Lambda=\bigcup\limits_{i=0}^{N}\lambda_{i}$

Each step of the broader process $A$ can be therefore defined as:

$\begin{cases}a_{0}:\iota_{0}\times c_{\emptyset}\mapsto\lambda_{0}\\\
a_{i}:\iota_{i}\times c_{i-1}\mapsto\lambda_{i}\end{cases}$ (2)

###  3.3 How do we observe learned language?

The notion of language that we introduced incorporates that of grammar, namely
the analytical categories that we superimpose on the linguistic stream in
order to analyze it and its unfolding over time. We do not test language as a
cognitive state of the speaker: we intend it instead a set of categories that
the observer (i.e., the linguist) considers relevant to the description of the
linguistic stream produced by the (artificial) speaker. There exists,
therefore, a striking difference between the linguistic stream (either the
input perceived or the output produced by the speaker) and its representation
through the lens provided by language.

If we wanted to be more precise with the notation, we should acknowledge the
fact that language, i.e. $\Lambda$, as we mean it is actually a function by
itself, that takes as input some linguistic stream (some observable data) and
returns a representation of it. We could therefore rewrite the definition of
$a_{i}$ as $a_{i}:\iota_{i}\times c_{i-1}\mapsto\Lambda(o_{i})$ where $o_{i}$
is the linguistic stream produced by the speaker as a result of acquisition
step $a_{i}$.

As we are interested in the categories that are acquired by the speaker and
deployed during language comprehension and production, defining
$\lambda_{o_{i}}=\Lambda(o_{i})$ allows us to apply the same transformation on
the input $\iota_{i}$ to which the speaker is exposed, thus obtaining
$\lambda_{\iota_{i}}$ that is comparable to $\lambda_{o_{i}}$ in terms of
linguistic categories.

Sticking to the constructionist perspective while trying to make the fewest
possible assumptions on the actual content of linguistic knowledge, we
hypothesize language as made up of a network of structures that are supposed
to approximate constructions. As constructions are form-meaning pairs, the
notion of grammar incorporates that of a meaning space spanning beyond the
lexical level. This can be easily implemented by extending the notion of
vector space models that has been extensively explored and used in
distributional semantics (Lenci, 2008; Erk, 2012; Lenci, 2018). This
represents a major difference with nativist approaches and the standard
evaluation framework: meaning cannot be factored out of grammar effects and
the acquisitional framework must account for its role in the process. If we
had to formalize the content of any $\lambda_{i}$, therefore, we could expand
it as $\lambda_{i}=\\{(\kappa,\vec{\kappa})\mid\kappa\text{ is a construction
wrt. some linguistic stream}\\}$

Unpacking this, we are saying that each obtained constructicon $\lambda_{i}$
is a network of structures. These can be more or less lexicalized, with their
abstractness being a proxy for linking the structures in the network as we
will explain in the next paragraph. Each construction is associated with a
distributional vector (Figure 1), which represent its meaning.

Figure 1: Let’s assume that $\Lambda$ contains both contructions DET NOUN and
the dog, with the latter being a lexicalized instance of the former. At
different steps during acquisition, the two constructions can assume different
meanings and be therefore associated with different distributional vectors. A
distributional vector condenses in fact information about co-occurrences
between linguistic items in a given piece of text. In the figure, we see that
a cluster of vectors gather around DET NOUN in the constructicon built from
the input data (leftmost panel). This means that a variety of lexicalized
instances exist for the construction DET NOUN. During learning, the
constructicons built from generated output show different distributions for
the construction DET NOUN. In the central panel, the cosine distance between
DET NOUN and the dog is 0, meaning that their distributional contexts (i.e.,
their co-occurrences) perfectly overlap. In the rightmost panel instead, the
distance between the two vectors has increased as another lexicalized instance
(i.e., the cat) is being produced. In this scenario, the contexts where DET
NOUN appears do not perfectly overlap with those where the dog appears.

###  3.4 Desiderata: the structure induced by $\Lambda$

We defined $\Lambda$ as a function that takes as input a linguistic stream
$\tau$ and returns a constructicon $\lambda_{\tau}$: a structured repository
of form-meaning pairs. In order to define and explore the internal structure
of the constructicon, we introduce a few auxiliary functions and definitions:
(i): having meaning defined as a distributional space allows for distance
computation $d(\kappa_{i},\kappa_{j})$ with
$d:\Lambda\times\Lambda\mapsto[0,1]$. $d(\cdot,\cdot)$ is a metric function
that computes the distance between two meaning vectors. Usually,
$d(\kappa_{i},\kappa_{j})=1-cos(\vec{\kappa_{i}},\vec{\kappa_{j}})$, where
$cos(\vec{\kappa_{i}},\vec{\kappa_{j}})$ is the cosine similarity between the
two vectors associated to $\kappa_{i}$ and $\kappa_{j}$;
(ii): constructions bearing different abstraction levels are linked in the
network. In order to navigate the network we introduce the function
$c(\kappa_{i},\kappa_{j})$ with $c:\Lambda\times\Lambda\mapsto\\{0,1\\}$ being
a boolean function that computes whether two constructions constitute an
abstraction chain. For instance, $\kappa_{i}=\text{nsubj, GIVE, iobj, dobj}$
and $\kappa_{j}=\text{nsubj, root, iobj, dobj}$ form a chain with $\kappa_{i}$
being a partially lexicalized (hence, less abstract) instance of $\kappa_{j}$.

###  3.5 Use scenarios

####  3.5.1 Individual acquisition over time

The framework can be used to observe how the acquisition process unfolds over
time. We can in fact set a number of steps $n$ and observe: (i) how the shape
of grammar changes over the course of learning, comparing the various steps,
as in: $\Lambda(o_{1})\sim\Lambda_{(}o_{2})\sim\cdots\sim\Lambda_{(}o_{n})$,
(ii) how the grammar of the input can be compared to that acquired by the
speaker, as in: $\Lambda(I)\sim\Lambda(o_{n})$. Given a subset
$K\subseteq\Lambda(I)$111Actually, we have to make sure that
$K\subseteq\Lambda(I)\cap\lambda_{0}\cap\lambda_{1}\cap\cdots\cap\lambda_{n}$
of interesting constructions, we can observe their behaviour over the learning
process.

A popular constructionist hypothesis (Goldberg, 2006), for example, states
that the meaning of a construction (e.g., the ditransitive pattern Subj V Obj
Obj2), and therefore its productivity, emerges from the association with
specific lexical items in the input received by the learner (e.g., give in the
case of the ditransitive): part of the lexical meaning remains attached to the
meaning of the syntactic pattern, and therefore its distributional properties
with it. Let’s assume that the speaker has acquired some construction $\kappa$
(e.g., the ditransitive construction). Once they’re able to use it in a
productive and creative way (i.e., in a more varied contexts than the give
contexts the construction is strongly associated with in the input), we can
use the proposed framework to check whether the distributional meaning of two
constructions $\kappa_{i},\kappa_{j}\in\Lambda(I)$ with
$c(\kappa_{i},\kappa_{j})=1$ (i.e., with $\kappa_{i}$ being a less abstract
instance of $\kappa_{j}$) influences the learnability of $\kappa_{j}$ as an
independent construction.

The notion of abstraction chain introduced before helps us testing this
hypothesis as we can check the behaviour of the chain
$(\kappa_{i},\kappa_{j})$ at each timestep. We can denote
$\kappa_{i}^{\lambda_{k}}$ the construction $\kappa_{i}\in\lambda_{k}$ and
similarly $\kappa_{j}^{\lambda_{k}}$ the construction
$\kappa_{j}\in\lambda_{k}$, through distributional analysis we can capture how
the contexts in which $\kappa_{i}$ and $\kappa_{j}$ vary, and whether this
variation is associated with grammatical generalization. We expect, in fact,
$d(\kappa_{i},\kappa_{j})$ to increase during acquisition:

$d(\kappa_{i}^{\lambda_{a}},\kappa_{j}^{\lambda_{a}})\leq
d(\kappa_{i}^{\lambda_{b}},\kappa_{j}^{\lambda_{b}})\;\;\forall\>a,b\mid a\leq
b$ (3)

If $\kappa_{j}$ is produced in contexts that do not perfectly overlap with
those where $\kappa_{i}$ is produced, this indicates that the speaker has
gained a productive use of construction $\kappa_{j}$, which is recognized as
an independent construction from $\kappa_{i}$. If conversely their distance
decreases during acquisition, we might deduce that the speaker has recognized
$\kappa_{j}$ as unnecessary by restricting its application cases to those of
$\kappa_{i}$.

####  3.5.2 Language as the expression of a population of speakers

We are often interested in defining grammar in terms of what can be considered
shared linguistic knowledge among the speakers. A core aspect of construction
grammar is in fact conceiving language primarily as a social and external
phenomenon, as opposed to nativist theories that focus on its inner nature. By
means of the framework, we can analyze grammar as an abstraction over the
linguistic productions of a population of $P$ speakers
$\Pi=(\sigma_{1},\sigma_{2},\cdots,\sigma_{P})$. We can define the grammatical
conventions deployed by the community $\Pi$ as
$\Lambda_{\Pi}=(\lambda_{\sigma_{1}},\lambda_{\sigma_{2}},\cdots,\lambda_{\sigma_{P}})$.
This allows for modelling variation between the acquisition process of the
different speakers. Speaker $\sigma_{i}$ might be exposed to a unique series
of input material $\iota_{0}^{\sigma_{i}},\cdots,\iota_{N}^{\sigma_{i}}$ that
does not necessarily coincide with that of speaker $\sigma_{j}$.

In this setting, we can for instance investigate what is learned no-matter-
the-input, and what is instead specific or idiosyncratic for each speaker. We
can define:

$G_{\geq p}=\left\\{\kappa\mid\sum\limits_{i=0}^{P}X(\kappa,\sigma_{i})\geq
p\right\\}$ (4)

as the set of constructions that we can observe in the linguistic productions
of $p$ or more speakers. With:

$X(\kappa_{i},\sigma_{j})=\begin{cases}1&\text{if
$\kappa\in\Lambda^{\sigma_{j}}$}\\\ 0&\text{otherwise}\end{cases}$ (5)

being an auxiliary function that evaluates to $1$ if the construction $\kappa$
appears in the production of speaker $\sigma_{j}$ and $0$ otherwise (this just
helps us count how many speakers use construction $\kappa$ productively).
$G_{P}$ would for instance be the set of constructions shared by all speakers
in a population, and could be therefore identified as the set of core
constructions in $\Lambda^{\Pi}$. When, instead, $p\ll P$, we are observing
constructions that are not shared by a significant amount of speakers in the
population, and their use can therefore depend on specific input instances or
tendencies in subgroups of speakers. Following the same logic we can of course
also just define $G_{(\sigma_{i},\sigma_{j})}$ as the constructions that are
common to the two speakers $\sigma_{i}$ and $\sigma_{j}$. By means of $G$, we
can define $\widetilde{\Lambda_{G}}$ as an approximation of the function
$\Lambda$, which only uses the categories that are retained in $G$.
$\widetilde{\Lambda_{G_{\geq P}}}$ would for instance be a function that
considers only linguistic knowledge shared by the entire population $\Pi$,
while $\widetilde{\Lambda_{\sigma_{i}}}$ would be restricted to the
constructicon $\lambda_{\sigma_{i}}$. Considering speakers $\sigma_{i}$ and
$\sigma_{j}$, with their respective produced linguistic outputs
$O_{\sigma_{i}}$ and $O_{\sigma_{j}}$, we can produce and compare
$\widetilde{\Lambda_{G_{\sigma_{i}}}}(O_{\sigma_{j}})$ and
$\widetilde{\Lambda_{G_{\sigma_{j}}}}(O_{\sigma_{i}})$: respectively, what
speaker $\sigma_{i}$ is able to retrieve from $O_{\sigma_{j}}$ and what
speaker $\sigma_{j}$ is able to retrieve from $O_{\sigma_{i}}$.

The fact that speakers use the same constructions $\kappa$ to build their
linguistic productions does not of course ensure that the corresponding
meanings $\vec{\kappa}$ coincide.222This makes sure that $G_{\sigma_{i}}$ does
not coincide with $\lambda_{\sigma_{i}}$ Different speakers, depending on the
input they have been exposed to, and to the partial randomness attributed to
computational mechanisms, could associate different meaning spaces to the same
construction. Given two speakers $\sigma_{i}$ and $\sigma_{j}$, and a sentence
$s$, we could therefore compare the portions of $\lambda_{\sigma_{i}}$ and
$\lambda_{\sigma_{j}}$ meaning spaces that are activated to linguistically
(de)compose the sentence $s$.

##  4 Exploratory experiments

In order to explore the potential applications of the framework described in
§3, we built a simple instance using the CHILDES corpus (MacWhinney, 2000) as
input data $I$ and a vanilla character-based LSTM (Hochreiter and Schmidhuber,
1997) as computational mechanisms $C$. With this simple setting, we explored
two aspects: (i) we tested whether distributional similarities in
$\lambda_{I}$ would influence the acquisition of constructicons
$\lambda_{1},\cdots,\lambda_{n}$, and (ii) we tried to describe grammar as it
emerges from a population of speakers. Constructions were approximated through
catenae (Osborne et al., 2012): subtrees extracted from a dependency parsing
syntactic representation (see Figure 2).

[theme = simple] [column sep=0.3em, row sep=.1ex] NOUN & VERB & DET & ADJ &
NOUN
Mary & had & a & little & lamb
(a) & (b) & (c) & (d) & (e)
[edge unit distance = 2ex]2ROOT 21nsubj 25dobj 53det 54nmod

Figure 2: The dependency representation of the sentence Mary had a little
lamb, annotated with morpho-syntactic and syntactic information. In this
structure, we can identify the following catenae: a, b, c, d, e, ab, abce,
abde, abcde, abe, bce, bde, be, ce, de, cde. Other possibilities would have
been strings (e.g., a, ab, abc, … b, bc, …e) or constituents (i.e., a, abcde,
c, d, cde).

###  4.1 Abstracting grammar over training

We first replicate an analysis presented in Pannitto and Herbelot (2020),
where a character-based LSTM was trained on CHILDES corpus. The authors fixed
7 steps during the LSTM’s acquisition process, each after 5 epochs of
training. In our formalization, this equates to 7 constructicons $\lambda_{1}$
to $\lambda_{7}$. The distributional space for each $\lambda_{i}$ is obtained
by counting co-occurrences between constructions within the same sentence. We
can then consider abstraction chains $(\kappa_{i},\kappa_{j})$ in $I$ (i.e.,
in $\Lambda(\text{CHILDES})$ and computed
$d(\kappa^{\lambda_{7}}_{i},\kappa^{\lambda_{7}}_{j})-d(\kappa^{\lambda_{1}}_{i},\kappa^{\lambda_{1}}_{j})$
for each abstraction chain, namely the difference in cosine similarity between
step 7 and step 1. Grouping all chains by $\kappa_{i}$ and $\kappa_{j}$, it is
possible to compute the average distributional shift as shown in Table 1
(i.e., for each $\kappa_{i}$ to its more abstract instances and for each
$\kappa_{j}$ to its more concrete instances).

$\kappa_{i}$ | shift | cosine |  $\kappa_{j}$ | shift | cosine
---|---|---|---|---|---
@nsubj @root so | 0.18 | 0.43 |  more @root | 0.2 | 0.21
@nsubj only @root | 0.18 | 0.41 |  _AUX know @obj | 0.19 | 0.66
what @root @obj | 0.18 | 0.39 |  @advmod tell | 0.17 | 0.64
what @advmod _VERB | 0.16 | 0.19 |  @aux know @obj | 0.16 | 0.71
only @root | 0.16 | 0.38 |  @advmod can _VERB | 0.15 | 0.76
more @root | 0.16 | 0.23 |  know @obj | 0.15 | 0.62
@root it @xcomp | 0.15 | 0.61 |  a _NOUN | 0.13 | 0.52
@det minute | 0.15 | 0.25 |  might @root | 0.13 | 0.70
_PRON only @root | 0.15 | 0.53 |  _PRON @root n’t | 0.12 | 0.53
_VERB _DET minute | 0.15 | 0.33 |  @root that _VERB | 0.12 | 0.65
_PRON @root so | 0.14 | 0.54 |  _VERB ’ll @ccomp | 0.12 | 0.71
_DET minute | 0.134 | 0.33 |  _VERB me @obl | 0.12 | 0.76
Table 1: Constructions with highest average shifts.

Three bins are considered, based on average distributional shift: the
hypothesis is that constructions that underwent the highest shifts during
training are those showing intermediate levels of similarities in the input
distributional space. Indeed, chains with very high input similarities are
unlikely to exhibit abstraction: according to constructionist intuition, their
distributional similarity means that the construction that is part of the
Constructicon is the least schematic one, and there is no need for the more
schematic (and therefore, abstract) category to be created. Low similarity
pairs, on the other hand, may simply contain unrelated constructions. The
three groups show different distributions333A Kruskall-Wallis one-way ANOVA
was performed and resulted in significant values. as shown in Figure 3.

Figure 3: Distribution of average cosine similarities for the three groups of
$kappa_{j}$, showing low, intermediate and high average shifts respectively.

###  4.2 A population of artificial speakers

Following on Pannitto and Herbelot (2020) experiments, we consider a
population of 10 speakers modeled with 10 vanilla character-based LSTMs
trained on random samples of the CHILDES corpus (each containing 1 million
words).

With this setting, we try to identify the locus of variation among different
speakers, under the assumption that some ‘core’ constructions must be shared
by all individuals, while others are less important to successful
communication.

We restrict the analysis to the constructions to which all 10 speakers have
been exposed to through their input (11051 constructions) and create $G_{10}$
as the set of core constructions and $G_{\leq 5}$ as the set of periphery
constructions, i.e. the ones shared by half of the speakers or less. Being
trained on random samples taken from the same distribution, the speakers share
most of the constructions (9086 out of 11051). However, we expect these
numbers to change significantly when the input language varies along more
refined sociolinguistic axes.

We also checked, for all speakers, whether the constructions of the core group
and the constructions of the periphery group had significantly different
frequencies in the input given to each speaker. As shown in Figure 4, the
difference between the three groups are significant despite not appearing as
striking as one would expect.

Figure 4: Difference in input frequency between the three groups of
constructions: core as the ones shared by all speakers, periphery as the ones
shared by half of the speakers or less, and other as the remaining ones.

Lastly, we explored the input through $\widetilde{\Lambda_{G_{10}}}$ and
$\widetilde{\Lambda_{G_{\leq 5}}}$, as shown in Table 2: both representations
(the one obtained through core constructions and the one obtained through
periphery constructions) highlight meaningful patterns in the sentence, but
only the former can be considered a grammatical representation shared by the
population.

corpus | Core | Periphery
---|---|---
does | AUX | -
n’t | n’t | -
that | that | PRON
seem | @root | VERB
kind | - | ADV
of | - | -
silly | ADV | ADV
Table 2: A sentence (left column) as it would appear if we restricted to only
core (middle column) or periphery (right column) constructions.

##  5 Concluding remarks

The nature of linguistic representations is a core issue in linguistic
theories of language development. We feel this aspect has been overlooked in
the NLMs literature and propose an approach that brings back theoretical
insights into the picture. We commit here to the UB constructionist framework,
not as an ideal model of human language acquisition, but rather as a set of
tools and categories that suffice to explain NLMs’ generated language.

Since learning a language largely overlaps with learning to process the input,
there must be a relation between processing biases relating to certain types
of constructions and the distribution of those constructions in the linguistic
input (Christiansen and Chater, 2016a). As experience grounds linguistic
knowledge, distributional properties become a key aspect to determine the
content of linguistic representations. In this framework, language is not
considered as an autonomous cognitive system. Rather, the acquisition of
grammar is regarded as any other conceptualization process and knowledge of
language emerges from use.

To conclude, the observation of NLMs linguistic abilities would benefit from a
constructionist approach. The evaluation can take place at multiple levels and
includes properties of the situation described by the linguistic signal, but
also properties of the linguistic signal itself. The UB framework may in fact
provide useful categories to analyze the statistical properties of artificial
language learners, and most importantly allows us to examine the semantic and
the syntactic layers in parallel, both in the input received by the learner
and in the stochastic output it generates.

## References

  * Bannard et al. (2009a) Colin Bannard, Elena Lieven, and Michael Tomasello. 2009a.  Modeling children’s early grammatical knowledge.  _Proceedings of the National Academy of Sciences_ , 106(41):17284–17289.
  * Bannard et al. (2009b) Colin Bannard, Elena Lieven, and Michael Tomasello. 2009b.  Modeling children’s early grammatical knowledge.  _Proceedings of the National Academy of Sciences of the United States of America_ , 106(41):17284–17289.
  * Barlow and Kemmer (2000) Michael Barlow and Suzanne Kemmer. 2000.  Usage-based models.  _Language. Stanford_.
  * Boyd and Goldberg (2009) Jeremy K Boyd and Adele E Goldberg. 2009.  Input effects within a constructionist framework.  _Modern Language Journal_ , 93(3):418–429.
  * Chomsky (1986) Noam Chomsky. 1986.  _Knowledge of language: Its nature, origin, and use_.  Greenwood Publishing Group.
  * Christiansen (2019) Morten H Christiansen. 2019.  Implicit statistical learning: A tale of two literatures.  _Topics in Cognitive Science_ , 11(3):468–481.
  * Christiansen and Chater (2016a) Morten H Christiansen and Nick Chater. 2016a.  _Creating language: Integrating evolution, acquisition, and processing_.  MIT Press.
  * Christiansen and Chater (2016b) Morten H Christiansen and Nick Chater. 2016b.  The Now-or-Never bottleneck: A fundamental constraint on language.  _The Behavioral and brain sciences_ , 39:e62.
  * Cornish et al. (2017) Hannah Cornish, Rick Dale, Simon Kirby, and Morten H Christiansen. 2017.  Sequence memory constraints give rise to language-like structure through iterated learning.  _PloS one_ , 12(1).
  * Crain and Pietroski (2001) Stephen Crain and Paul Pietroski. 2001.  Nature, nurture and universal grammar.  _Linguistics and philosophy_ , 24(2):139–186.
  * Crain et al. (2009) Stephen Crain, Rosalind Thornton, and Keiko Murasugi. 2009.  Capturing the evasive passive.  _Language acquisition_ , 16(2):123–133.
  * Croft (2001) William Croft. 2001.  _Radical Construction Grammar: Syntactic Theory in Typological Perspective_.  Oxford University Press.
  * Croft and Alan Cruse (2004) William Croft and D Alan Cruse. 2004.  _Cognitive Linguistics_.  Cambridge University Press.
  * Eisenbeiß (2009) Sonja Eisenbeiß. 2009.  Generative approaches to language learning.  47(2):273–310.
  * Erk (2012) Katrin Erk. 2012.  Vector space models of word meaning and phrase meaning: A survey.  _Language and Linguistics Compass_ , 6(10):635–653.
  * Fillmore (1976) Fillmore. 1976.  Frame semantics and the nature of language.  _Annals of the New York Academy of Sciences_.
  * Goldberg (2003) Adele E Goldberg. 2003.  Constructions: A new theoretical approach to language.  _Trends in cognitive sciences_ , 7(5):219–224.
  * Goldberg (2006) Adele E Goldberg. 2006.  _Constructions at work: The nature of generalization in language_.  Oxford University Press on Demand.
  * Gómez and Gerken (2000) Rebecca L Gómez and LouAnn Gerken. 2000.  Infant artificial language learning and language acquisition.  _Trends in cognitive sciences_ , 4(5):178–186.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Katz and Fodor (1963) Jerrold J Katz and Jerry A Fodor. 1963.  The structure of a semantic theory.  _Language_ , 39(2):170–210.
  * Labov (1973) William Labov. 1973.  The boundaries of words and their meanings.  _New ways of analyzing variation in English_.
  * Lenci (2008) Alessandro Lenci. 2008.  Distributional semantics in linguistic and cognitive research.  _Italian journal of linguistics_ , 20(1):1–31.
  * Lenci (2018) Alessandro Lenci. 2018.  Distributional models of word meaning.  _Annual review of Linguistics_ , 4:151–171.
  * Lewkowicz et al. (2018) David J Lewkowicz, Mark A Schmuckler, and Diane M J Mangalindan. 2018.  Learning of hierarchical serial patterns emerges in infancy.  _Developmental psychobiology_ , 60(3):243–255.
  * Lidz and Williams (2009) Jeffrey Lidz and Alexander Williams. 2009.  Constructions on holiday.  20(1):177–189.
  * MacWhinney (2000) Brian MacWhinney. 2000.  _The CHILDES Project: Tools for analyzing talk. Third Edition._ Lawrence Erlbaum Associates.
  * Montague (1970) Montague. 1970.  English as a formal language.  _Linguaggi nella Societa e nella Tecnica_.
  * Osborne et al. (2012) Timothy Osborne, Michael Putnam, and Thomas Groß. 2012.  Catenae: Introducing a novel unit of syntactic analysis.  _Syntax_ , 15(4):354–396.
  * Pannitto and Herbelot (2020) Ludovica Pannitto and Aurélie Herbelot. 2020.  Recurrent babbling: evaluating the acquisition of grammar from limited input data.  In _Proceedings of the 24th Conference on Computational Natural Language Learning_ , pages 165–176.
  * Pannitto and Herbelot (2022) Ludovica Pannitto and Aurelie Herbelot. 2022.  Can recurrent neural networks validate usage-based theories of grammar acquisition?  _Frontiers in Psychology_ , 13.
  * Pinker (1984) Steven Pinker. 1984.  _Language learnability and language development_.  Harvard University Press, Cambridge, MA.
  * Romberg and Saffran (2010) Alexa R Romberg and Jenny R Saffran. 2010.  Statistical learning and language acquisition.  _Wiley Interdisciplinary Reviews: Cognitive Science_ , 1(6):906–914.
  * Smith and Allott (2016) Neil Smith and Nicholas Allott. 2016.  _Chomsky: Ideas and ideals_.  Cambridge University Press.
  * Street and Dąbrowska (2010) James A Street and Ewa Dąbrowska. 2010.  More individual differences in language attainment: How much do adult native speakers of english know about passives and quantifiers?  _Lingua. International review of general linguistics. Revue internationale de linguistique generale_ , 120(8):2080–2094.
  * Tomasello (2003) Michael Tomasello. 2003.  _Constructing a language: A usage-based theory of language acquisition_.  Harvard University Press.