# A Dual-Path Model With Adaptive Attention For Vehicle Re-Identification

Pirazh Khorramshahi1, Amit Kumar1, Neehar Peri1, Sai Saketh Rambhatla1, Jun-
Cheng Chen2
and Rama Chellappa1
1Center for Automation Research, UMIACS, University of Maryland, College Park
2Research Center for Information Technology Innovation, Academia Sinica
{pirazhkh, akumar14, peri, rssaketh<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

In recent years, attention models have been extensively used for person and
vehicle re-identification. Most re-identification methods are designed to
focus attention on key-point locations. However, depending on the orientation,
the contribution of each key-point varies. In this paper, we present a novel
dual-path adaptive attention model for vehicle re-identification (AAVER). The
global appearance path captures macroscopic vehicle features while the
orientation conditioned part appearance path learns to capture localized
discriminative features by focusing attention on the most informative key-
points. Through extensive experimentation, we show that the proposed AAVER
method is able to accurately re-identify vehicles in unconstrained scenarios,
yielding state of the art results on the challenging dataset VeRi-776. As a
byproduct, the proposed system is also able to accurately predict vehicle key-
points and shows an improvement of more than $7\%$ over state of the art. The
code for key-point estimation model is available at
https://github.com/Pirazh/Vehicle_Key_Point_Orientation_Estimation

##  1 Introduction

Vehicle re-identification (re-id) refers to the task of retrieving all images
of a particular vehicle identity in a large gallery set, composed of vehicle
images taken from varying orientations, cameras, time and locations.
Accurately re-identifying vehicles from images and videos, is of great
interest in surveillance and intelligence applications. In contrast to vehicle
recognition which aims to identify the make and model of the vehicle, vehicle
re-id is concerned with identifying specific vehicle instances. This task is
extremely challenging as vehicles with different identities can be of the same
make, model and color, and thus it is challenging for a Deep Convolutional
Neural Network (DCNN) to make accurate predictions. In this paper, we present
a novel algorithm driven by adaptive attention for re-identifying vehicles
from still images without using information from other sources such as time
and location.

The similar task of person re-id aims at re-identifying humans appearing in
different cameras. While visual appearance models work reasonably well for
person re-id, the same techniques fail to differentiate vehicles due to the
lack of highly discriminating features. Person re-id models are not heavily
reliant on facial features as they also learn discriminating features based on
clothing and accessories. However, vehicle re-id poses a new set of
challenges. Different vehicle identities can have similar colors and shapes
especially those coming from the same manufacturer with a particular model,
trim and year. Subtle cues such as different wheel patterns and custom logos
might be unavailable in the global appearance features. Therefore, it is
important that vehicle re-id model learns to focus on different parts of the
vehicles while making a decision. Previous works in person re-id such as [25]
have used attention models with human key-points as regions of attention and
have shown significant improvement in performance. Similarly, methods such as
[23] have used vehicle key-points to learn attention maps for each of the $20$
key-points defined by [23]. The system proposed by Wang _et al._[23] grouped
key-points into four groups corresponding to front, rear, left and right.

(a) Front

(b) Left

(c) Right

(d) Rear

Figure 1: Heatmaps grouped as suggested in [23]. Attention to all subgroup of
key-points leads to erroneous results. Although, only the rear of the car is
visible, contributions from frontal key-points are non-zero.

However, not all key-points provide discriminating information and their
respective contributions depend on the orientation of the vehicle. For
instance, in Figure 1(a) we observe that the key-points from the front of the
car incorrectly influence the attention of the model as the front of the car
is not visible. Hence, paying attention to all the key-points, as suggested in
[23], can lead to erroneous results. The proposed method tackles the problem
of false attention by adaptively choosing the key-points to focus on, based on
the orientation of the vehicle hence, providing complementary information to
global appearance features. In this work, terms with same connotation, path,
stream and branch, have been used interchangeably.

In the proposed method, the first stream is a DCNN trained to extract
discriminative global appearance features for each vehicle identity. However,
this stream often fails to extract subtle features necessary to distinguish
similar vehicles. Therefore, a second path composed of orientation conditioned
key-point selection and localized feature extraction modules is used in
parallel to supplement the features from the first path. By using orientation
as a conditioning factor for adaptive key-point selection, the model learns to
focus on the most informative parts of the vehicle. Additionally, we develop a
fully convolutional two-stage key-point detection model inspired by the works
of Kumar _et al._[9] and Bulat _et al._[2] for facial key-point detection and
human pose estimation respectively.

The detailed architectures of each module in the proposed method are discussed
in section 3. Through extensive experimentation, we show that the proposed
Adaptive Attention model for Vehicle Re-identification (AAVER) approach
improves the re-id accuracy on challenging datasets such as VeRi-776 [11, 12]
and VehicleID [10]. In addition, the proposed vehicle key-point detection
model, improves the accuracy by more than $7\%$ over state of the art.

##  2 Related Work

In this section, we briefly review recent relevant works in the field of
vehicle classification and re-identification. Learning a discriminating
representation, requires a large-scale annotated data for training, especially
for recent DCNN approaches. Yang _et al._[27] released a large-scale car
dataset (CompCars) for fine-grained vehicle model classification which
consists of 1,687 car models and 214,345 images. The VehicleID dataset by Liu
_et al._ [10] consists of 200,000 images of about 26,000 vehicles. In
addition, Liu _et al._[12, 13] published a high-quality multi-view vehicle re-
id (VeRi-776) dataset. Yan _et al._[26] released two high-quality and well-
annotated vehicle datasets, namely VD1 and VD2, with diverse annotated
attributes, containing 1,097,649 and 807,260 vehicle images captured in
different cities.

Moreover, besides datasets for training, Tang _et al._[22] claimed traditional
hand-crafted features are complementary to deep features and thus fused both
features to realize an improved representation. Instead, Cui _et al._[4] fused
features from various DCNNs trained with different objectives. Furthermore,
Liu _et al._[12, 13] used multi-modal features, including visual features,
license plate, camera location, and other contextual information, in a coarse-
to-fine vehicle retrieval framework. To augment the training data for robust
training, [24] used a generative adversarial network to synthesize vehicle
images with diverse orientation and appearance variations. [31] learns a
viewpoint-aware representation for vehicle re-id through adverserial learning
and a viewpoint-aware attention model.

Besides global features, Liu _et al._ [14] extracted discriminative local
features from a series of local regions of a vehicle by a region-aware deep
model. Different from these approaches, the proposed method leverages
orientation to adaptively select the regions of attentions.

Another effective strategy to learn the discriminative representation is
metric learning. Zhang _et al._[29] proposed an improved triplet loss which
performs joint optimization with an auxiliary classification loss as a
regularizer in order to characterize intra-sample variances. Bai _et al._[1]
introduced Group-Sensitive triplet embeding to better model the intra-class
variance. Shen _et al._ [20] also proposed to improve the matching performance
by making use of spatio-temporal information; they developed a Siamese-CNN
with path LSTM model which generates the corresponding candidate visual-
spatio-temporal paths of an actual vehicle image by a chain-based Markov
random field (MRF) model with a deeply learned potential function. In
contrast, the proposed method uses the $L_{2}$ softmax [19] loss function as
it has shown impressive performance for the task of face verification and
trains faster compared to triplet loss-based methods such as [29] without the
hassle of sampling hard triplets.

##  3 Adaptive Attention Vehicle Re-identification (AAVER)

Figure 2: Adaptive Attention Vehicle Re-identification (AAVER) Model Pipeline.
The input vehicle image is processed in parallel along two paths: In the first
path, the global appearance features ($f_{g}$) are extracted. The second path
is responsible for detecting vehicle key-points and predicting its
orientation, after which localized features ($f_{l}$) are extracted based on
adaptive key-point selection. Subsequently, the two feature vectors $f_{g}$
and $f_{l}$ are fused with a shallow multi-layer perceptron.

The entire pipeline of the proposed method AAVER is composed of three main
modules: Global Feature Extraction, Vehicle Key-Point and Viewpoint
Estimation, and Adaptive Key-Point Selection and Feature Extraction which is
followed by a re-ranking based post-processing. Figure 2 shows the
diagrammatic overview of our method.

In AAVER, the global feature extraction module is responsible for extracting
the macroscopic features ($f_{g}$) of the vehicles. By looking at the entire
vehicle, this model tries to maximally separate the identities in the feature
space. However, this model may fail to take into account subtle differences
between similar cars, most extremely the ones that are of the same make, model
and color. Therefore, the features generated by this module are supplemented
with features ($f_{l}$) from the localized feature extraction module. This can
be achieved by the adaptive attention strategy using the proposed key-point
and orientation estimation network.

In order to estimate the vehicle key-points, we draw inspiration from
literature on facial key-point detection and human pose estimation. Inspired
by [9, 2] we employ a two-stage model to predict the vehicle’s orientation and
landmarks in a coarse to fine manner; the coarse heatmaps predicted by a DCNN
are refined using a shallower network.

Finally, we use the proposed adaptive key-point selection module to select a
subset of most informative key-points and pool features from early layers of
the global feature extraction module to extract localized features around the
selected key-points. The features obtained from the two paths of AAVER are
then merged using a multi-layer perceptron. The entire model can be trained
end-to-end using any differentiable loss function. In our work, we use the
$L_{2}$ softmax loss as proposed in [19]. During inference, we use the
features from penultimate fully connected layer as the representation of a
given vehicle. Additionally, we also perform re-ranking[30] as a post
processing step.

Each module is described in detail in the following sub sections. Pytorch deep
learning framework [17] has been used in all of the experiments.

Figure 3: Vehicle key-point and orientation estimator network. VGG $16$
network outputs $21$ coarse heatmaps corresponding to the 20 vehicle landmarks
and the background (Response maps on the left). A two-stack hourglass network
refines $20$ key-points heatmaps (response maps on the right) excluding
background channel and predicts the vehicle’s orientation. Figure 4: The
confusion matrix of the vehicle orientation estimation network

###  3.1 Global Feature Extraction

For extracting the global appearance features, we employ ResNet-50 and
ResNet-101 [6] as backbone networks and also adopt them as our baseline
models. We initialized the weights of these models using the weights from the
models pre-trained on the CompCars dataset. A $2048$-dimensional features
vector from the last convolutional layer of ResNet is then fed to a shallow
multi-layer perceptron. This network is trained using the $L_{2}$ softmax loss
function which constrains the feature vectors extracted by the network to lie
on a hyper-sphere of radius $\alpha$. This enables the network to embed
features of identical vehicles together while pushing apart the features from
different vehicles. It is mathematically expressed as:

$\centering\mathcal{L}_{S}=-\log\frac{\exp^{(}\mathbf{W}_{y}^{T}(\frac{\alpha\mathbf{x}}{\|\mathbf{x}\|_{2}})+b_{y})}{\sum_{j=1}^{N}\exp^{(}\mathbf{W}_{j}^{T}(\frac{\alpha\mathbf{x}}{\|\mathbf{x}\|_{2}})+b_{j})}\@add@centering$
(1)

where $\mathbf{x}$ is the feature vector corresponding to class label $y$,
$\mathbf{W}_{j}$ is the weight and $b_{j}$ is the bias corresponding to class
j, $\alpha$ is a positive trainable scalar parameter, and $N$ is the number of
classes respectively.

###  3.2 Vehicle Key-Point and Orientation Estimation

In this work, a two-stage model is proposed for key-point estimation. In the
first stage, a VGG-16[21] -based fully convolutional network is employed to do
a coarse estimation of the location of $N_{1}$ ($N_{1}=21=$ 20 key-points plus
background) heatmaps of size $H\times W$ ($56\times 56$). This network is
trained using a per-pixel multi-class cross entropy loss defined as follows:

$\centering\mathcal{L}_{1}=\frac{-1}{H\times
W}\sum_{i=1}^{H}\sum_{j=1}^{W}\log(\frac{\exp(\mathbf{x}_{i,j}({t^{*}}_{i,j}))}{\sum_{k=1}^{N_{1}}\exp(\mathbf{x}_{i,j}(k))})\@add@centering$
(2)

where $\mathbf{x}_{i,j}$ is the vector corresponding to pixel location $i$ and
$j$ across all output channels and $t^{*}_{i,j}$ is the ground-truth class
label for that pixel location. After training the first stage, the weights of
this network are frozen for training of the subsequent stage. The left side of
Figure 3 depicts the the output of the first stage for a sample vehicle image.

Although the responses of the first stage can be used for the prediction of
visible key-point locations, there might be erroneous activations in the
heatmaps that correspond to invisible key-points. Consequently, we use the
second stage that takes in the sub-sampled version of the input image and the
coarse estimates of key-points to refine the results. The refinement network
follows the hourglass architecture introduced in [16] which is commonly used
for refining heatmaps and reducing artifacts due to invisible key-points. In
the second stage, coarse heatmaps estimated from the first stage, are refined
through a two-stack hourglass network with skip connections. Along with
refining the estimated key-points, the orientation of the vehicle is also
predicted through a parallel branch composed of two fully connected layers
designed to classify the orientation into eight classes as defined in[23].
This multi-task learning helps the refinement network to make accurate
predictions of the visible key-points while reducing the response of invisible
key-points. Figure 3 shows the overall schematic flow of the two-stage
network.

To train the heatmap refinement and orientation branches we use Mean Square
Error (MSE) and cross entropy loss respectively. Equation 3 represents the
loss function used for the second stage. It is worth mentioning that in the
second stage we are only interested in foreground heatmaps, hence, we exclude
the refinement of the background channel.

$\centering\mathcal{L}_{2}=\mathcal{L}_{H}+\lambda*\mathcal{L}_{O}\@add@centering$
(3)

where $\mathcal{L}_{H}$ is the heatmap regression loss:

$\mathcal{L}_{H}=\sum_{k=1}^{N_{2}}\sum_{i=1}^{H}\sum_{j=1}^{W}{\mid
h_{k}(i,j)-{h^{*}}_{k}(i,j)\mid}^{2}$ (4)

and $\mathcal{L}_{O}$ is the orientation classification loss:

$\mathcal{L}_{O}=-\log(\frac{\exp(\mathbf{p}(p^{*}))}{\sum_{i=1}^{N_{p}}\exp(\mathbf{p}(i))}).$
(5)

In Equation (4), $N_{2}=N_{1}-1$, $h_{k}(i,j)$ and $h^{*}_{k}(i,j)$ are
predicted and ground-truth heatmaps in stage 2 for the $k^{th}$ key-point at
locations $i$ and $j$ respectively. $\mathbf{p}$, $p^{*}$ and $N_{p}$ in
Equation (5) constitute the predicted orientation vector, the corresponding
ground-truth orientation and number of classes respectively. Finally,
$\lambda$ in Equation (3) is a weight to balance the losses used in model
optimization. In our experiments, $\lambda$ is set to $10$ obtained after
cross-validation. In the right hand side of Figure 3 which shows the the
output of the second stage, it can be observed that the initial coarse
estimates of key-points have been refined.

###  3.3 Adaptive Key-Point Selection and Feature Extraction

Subtle differences in similar vehicles mostly occur close to vehicle
landmarks, _e.g._ same car make and models of same color might be
distinguishable through their window stickers, rims, indicator lights on the
side mirrors, etc. This can be achieved by focusing the attention on parts of
the image that encompasses these distinctions. To this end, regions of
interest within the image are identified based on the orientation of the
vehicle; after which features from the shallower layer of the global
appearance model are pooled. As suggested in [28] these pooled features
contain contextual rather than abstract information. Later, deep blocks (Res3,
Res4 and Res5) of another ResNet model are used to extract supplementary
features corresponding to the regions of interest.

In [23], vehicle’s orientation is annotated into eight different classes,
_i.e._ rear, left ,left front, left rear, right, right front and right rear;
however, there is no absolute boundary between two adjacent orientations. For
instance, for the case of right and right front, the network gets confused
between the two classes when trained for orientation prediction; this can be
observed in Figure 4 which shows the confusion matrix for the eight-class
classification problem. To overcome this issue, we designed a key-point
selector module that takes the predicted orientation likelihood vector and
adaptively selects the key-points based on the likelihoods.

In order to achieve this, we constructed eight groups shown in Table 1
corresponding to each of the eight orientations of a vehicle and its two
adjacent orientations. During inference, the likelihood of each orientation
group is calculated and the one with the highest probability is picked. Also,
experimentally it was observed that for each orientation group at least seven
key-points are always visible. Consequently, given the orientation group with
the highest probability we select the seven heatmaps shown in Table 1
corresponding to the respective orientation group. These orientation groups
are named based on their center orientation _e.g._ the group that contains
left front, front and right front is named front.

Table 1: Seven Prominent key-points in each orientation group

Orientation Group |  Visible Key-Points
---|---
Front |  [11, 12, 7, 8, 9, 13, 14]
Rear |  [18, 16, 15, 19, 17, 11, 12]
Left |  [8, 1, 11, 14, 15, 2, 17]
Left Front |  [9, 14, 6, 8, 11, 1, 15]
Left Rear |  [2, 17, 15, 11, 14, 19, 1]
Right |  [7, 3, 12, 13, 16, 4, 18]
Right Front |  [9, 13, 5, 7, 12, 3, 16]
Right Rear |  [3, 4, 12, 16, 18, 19, 13]

After obtaining the seven heatmaps, for each map, a Gaussian kernel with
$\sigma=2$ is placed in the location of the map’s peak, _i.e._ the key-point
location. This is done in order to emphasize the importance of the surrounding
areas around the key-points as they may have discriminative information.

Following the adaptive heatmap selection and dilation by the Gaussian kernel,
is the localized feature extraction ($f_{l}$) by Res3, Res4 and Res5 blocks of
the parallel ResNet model. The input to this sub-network is the concatenation
of the seven dilated heatmaps of shape $7\times 56\times 56$ and the pooled
global features of shape $256\times 56\times 56$. Finally, the localized
features $f_{l}$ is concatenated with the global appearance features $f_{g}$
and passed through a multi-layer perceptron followed by $L_{2}$ softmax loss
function (refer to Figure 2). Given that features are normalized, we use
cosine similarity to calculate the similarity score between image pairs.

###  3.4 Post Processing Step: Re-Ranking

In general, Re-ID can be regarded as a retrieval problem. Given a probe
vehicle, we want to search in the gallery for images containing the same
vehicle in a cross-camera mode. After an initial ranking list is obtained, a
good practice consists of adding a re-ranking step, with the expectation that
the relevant images will receive higher ranks. Such re-ranking steps have been
mostly studied in generic instance retrievals such as [18], [3], [7] and [30].
The main advantage of many re-ranking methods is that they can be implemented
without requiring additional training samples, and also can be applied to any
initial ranking list.

Significant amount of research in person re-id goes into re-ranking strategies
and vehicle re-id is lacking in that aspect. Most of the state of the art
methods for vehicle re-id do not perform re-ranking on their initial ranking
list. We use the re-ranking strategy proposed by Zhong et al. [30] in our
work.

##  4 Experiments

Here we first present the two large-scale datasets used for the vehicle re-
identification task and their evaluation protocols, after which we describe
the implementation details of the proposed method.

###  4.1 Datasets

To the best our knowledge, there are mainly two large scale vehicle datasets
that are publicly available and are designed for the task of vehicle re-
identification: VeRi-776 [11],[12] and VehicleID [10].

VeRi-776 dataset consists of 49,357 images of $776$ distinct vehicles that
were captured with 20 non-overlapping cameras in variety of orientations and
lighting conditions. Out of these images, 37,778 (576 identities) and 11,579
(200 identities) have been assigned to training and testing respectively. For
the query set, 1,678 images have been selected from the testing set. The
evaluation protocol for this dataset is as follows: for each probe image in
the query set the corresponding identity and the camera ID from which the
image is captured is gathered. The gallery is constructed by selecting all the
images in the testing set except the ones that share the same identity and
camera ID as the probe. Evaluation metrics adopted for this dataset are mean
Average Precision (mAP), Cumulative Match Curve (CMC) for top 1 (CMC@1) and
top 5 (CMC@5) matches.

VehicleID is another large-scale dataset used for vehicle retrieval task and
is composed of $221,567$ images from $26,328$ unique vehicles. Half of the
identities, _i.e._ $13,164$, are reserved for training while the other half
are dedicated for evaluation. There are $6$ test splits for gallery sizes of
$800$, $1600$, $2400$, $3200$, $6000$ and $13,164$. In the recent works [23,
20] the first three splits have been used. The proposed evaluation protocol
for each split in VehicleID dataset is to randomly select an image for each of
the identities to form the gallery of respective size and use the rest of the
images for query. This procedure is repeated ten times and the averaged
metrics, CMC@1 and CMC@5, are reported.

###  4.2 Implementation Details

In our implementation, all the input images were resized to $(224,224)$ and
normalized by the ImageNet dataset [5] mean and standard deviation. Also, in
all of our experiments we used batch training with size of $150$ and Adam
optimizer [8] with the learning rate of $1e-4$.

Initially, we fine-tuned our baseline models (see section 3.1) on VeRi-776 and
VehicleID datasets separately, for $20$ epochs. Then, we initialized the key-
point and orientation estimation network with ImageNet pre-trained weights.
The first stage of this network was trained for $40$ epochs; afterwards the
second stage was trained for $40$ epochs as well.

Next, we trained the orientation conditioned feature extraction branch for
each of VeRi-776 and VehicleID datasets for 20 epochs. Finally, we select the
network’s output of the penultimate layer as the feature vector corresponding
to the input vehicle image.

##  5 Experimental Evaluations

We first present the evaluation results of our vehicle key-point and
orientation estimation model followed by the evaluation of the proposed method
AAVER on both VeRi-776 and VehicleID datasets.

###  5.1 Vehicle Key-Point and Orientation Estimation Evaluation

In order to evaluate the performance of the proposed two-stage key-point
detection model, we use the Mean Square Error (MSE) in terms of pixels for the
location of visible key-points in $56\times 56$ maps over the test set of
VeRi-776 key-point dataset. Table 2 shows the MSE of our model after first and
second stages. Moreover, we measured the accuracy of the model for viewpoint
classification. It can be observed that the refinement stage reduces the key-
point localization error by $20\%$ compared to the first stage.

To the best of our knowledge, [23] is the only work on the VeRi-776 key-point
and orientation estimation dataset. [23] used the averaged distance between
estimated and ground-truth locations of all visible key-points for evaluation.
If the distance is less than a threshold value ($r_{0}$ in terms of pixels in
$48\times 48$ map), the estimation is considered to be correct. We follow the
same protocol to compare the precision with [23] and Table 2 shows the result
of this comparison.

Table 2: Accuracy evaluation and comparison of the vehicle landmark and orientation estimation network | Stage 1 | Stage 2
---|---|---
Key-point localization MSE (pixel) | 1.95 | 1.56
Orientation Accuracy | - | 84.44%
Key-Point Precision Comparison
Model | $r_{0}=3$ | $r_{0}=5$
OIFE [23] | 88.8% | 92.05%
Ours | 95.30% | 97.11%

###  5.2 Evaluation Results on VeRi-776

Table 3 summarizes the results of the global appearance model (baseline) and
the proposed AAVER model with adaptive attention. Note that in both ResNet-50
and ResNet-101 -based architectures, there is a significant improvement in mAP
and CMC@1 scores after incorporating adaptive attention. This indicates that
conditioning on the orientation of the vehicle and selecting corresponding
key-points enables the network to focus more on parts that contains minute
differences in similar cars. This claim is further studied in section 5.5.
Unsurprisingly, we also observe that ResNet-101 shows better performance
compared to ResNet-50 under similar settings.

Table 3: Performance comparison between baseline and the proposed method on
VeRi-776 dataset

Model | mAP | CMC@1 | CMC@5
---|---|---|---
Baseline | ResNet-50 | 52.88 | 83.49 | 92.31
ResNet-101 | 55.75 | 84.74 | 94.34
AAVER | ResNet-50 | 58.52 | 88.68 | 94.10
ResNet-101 | 61.18 | 88.97 | 94.70

Figure 5 plots the probe image and the top three returns of each baseline and
the proposed model. It can be observed that AAVER significantly improves the
performance over the baseline.

|  |  |  |
---|---|---
|  |

(a) Probe Image

(b) Rank $1$

(c) Rank $2$

(d) Rank $3$

(e) Rank $1$

(f) Rank $2$

(g) Rank $3$

Figure 5: Top three returned results of the baseline model (sub-figures b-d)
versus the AAVER model (sub-figures e-g) on VeRi-776 dataset

###  5.3 Evaluation Results on VehicleID

Images in this dataset have less variations in viewpoint, _i.e._ mostly front
and rear, compared to VeRi-776 dataset. For this dataset, the evaluation
metrics are only CMC@1 and CMC@5 as there is only one true match in the
gallery for each probe image. Table 4 presents the re-identification results
of baseline and the proposed models over test splits. As compared to baseline
models, a significant increase in performance is observed when features from
adaptive attention-based path are fused with global appearance features.

Table 4: Performance comparison between baseline and proposed method on
VehicleID dataset

|  | Baseline Model | AAVER Model
---|---|---|---
| Split | ResNet-50 | ResNet-101 | ResNet-50 | ResNet-101
CMC@1 | 800 | 67.27 | 70.03 | 72.47 | 74.69
1600 | 62.03 | 65.26 | 66.85 | 68.62
2400 | 55.12 | 59.04 | 60.23 | 63.54
CMC@5 | 800 | 89.05 | 89.81 | 93.22 | 93.82
1600 | 84.31 | 84.96 | 89.39 | 89.95
2400 | 80.04 | 80.60 | 84.85 | 85.64

Figure 6 shows an examples of a query from VehicleID dataset and the top three
results returned by both global and adaptive attention model.

|  |  |  |
---|---|---
|  |

(a) Probe Image

(b) Rank $1$

(c) Rank $2$

(d) Rank $3$

(e) Rank$1$

(f) Rank $2$

(g) Rank $3$

Figure 6: Top three returned results of the baseline model (sub-figures b-d)
versus the AAVER model (sub-figures e-g) on VehicleID dataset

###  5.4 Comparison with State of the Art Methods

In this section, we compare AAVER model with ResNet-101 backbone against the
recent state of the art methods. The results of this comparison are presented
in Table 5.

Table 5: Comparison with recent methods and state of the arts

| Dataset
---|---
| VeRi-776 | VehicleID
| Test size = 800 | Test size = 1600 | Test size = 2400
Method | mAP | CMC@1 | CMC@5 | CMC@1 | CMC@5 | CMC@1 | CMC@5 | CMC@1 | CMC@5
SCPL [20] | 58.27 | 83.49 | 90.04 | - | - | - | - | - | -
OIFE [23] | 48.00 | 65.9 | 87.7 | - | - | - | - | - | -
VAMI [31] | 50.13 | 77.03 | 90.82 | 63.12 | 83.25 | 52.87 | 75.12 | 47.34 | 70.29
RAM [14] | 61.5 | 88.6 | 94.0 | 75.2 | 91.5 | 72.3 | 87.0 | 67.7 | 84.5
AAVER | 61.18 | 88.97 | 94.70 | 74.69 | 93.82 | 68.62 | 89.95 | 63.54 | 85.64
AAVER + Re-ranking | 66.35 | 90.17 | 94.34 | - | - | - | - | - | -

From Table 5, it can be observed that our proposed method is among the top
performers of vehicle re-identification task and is the state of the art for
most of the evaluation metrics on both VeRi-776 and VehicleID datasets. Note
that in the absence of a deterministic test set for VehicleID dataset, one
cannot provide the basis for a fair comparison. The reason lies in the fact
that random gallery construction yields different evaluation results with
relatively high variance even when averaged over ten repetitions. Finally, we
have to emphasize on the necessity of using re-ranking as a post processing
step whenever there are multiple instances of the probe image in the gallery.
Here for the VeRi-776 dataset, re-ranking shows significant improvement and
results in state of the art mAP and CMC@1 scores. Note that for VehicleID
dataset re-ranking is not applicable as there is only one true match in the
gallery for each probe image.

###  5.5 Ablation Studies

We designed a set of experiments to study the impact of complementary
information that the orientation conditioned branch provides. Note that in
these experiments we only use the test split $800$ for the VehicleID dataset
To this end, the following experiments have been conducted:

  1. 1.

In the first experiment we examined the depth of the layer in the global
branch from which the global features are pooled and then fed to the
orientation conditioned branch. To investigate this we tried pooling features
after Res2, Res3 and Res4 blocks of spatial size of $56\times 56$, $28\times
28$ and $14\times 14$. Table 6 demonstrates the results of this experiment. It
can be observed that as we go from shallow to deeper layers, the features
become more abstract and focusing on parts of deep feature maps do not help in
providing a robust representation of vehicles with minute differences.

Table 6: Experiment 1: Depth of pooled global features

Dataset | features size | mAP | CMC@1 | CMC@5
---|---|---|---|---
VeRi-776 | $56\times 56$ | 0.612 | 88.97 | 94.70
$28\times 28$ | 0.608 | 88.50 | 94.58
$14\times 14$ | 0.597 | 85.88 | 93.03
VehicleID | $56\times 56$ | - | 74.69 | 93.82
$28\times 28$ | - | 72.60 | 93.24
$14\times 14$ | - | 71.09 | 92.13

  2. 2.

In our method, we use two streams for extracting global and local features
from vehicle images, so we were keen to see whether a single branch can
extract discriminative features that encompass global as well as local
differences. To test this hypothesis, instead of pooling features from the
global branch we fused the selected heatmaps into the global branch by
concatenation and used the output as the representation for a vehicle image.
Table 7 depicts the result of this experiment, for both VeRi and VehicleID
datasets. we can infer that the re-identification performance drops
significantly by relying on a single branch.

Table 7: Experiment 2: Single versus Dual-branch feature extraction Dataset | Type | mAP | CMC@1 | CMC@5
---|---|---|---|---
VeRi-776 | Single | 0.528 | 80.93 | 90.52
Dual | 0.612 | 88.97 | 94.70
VehicleID | Single | - | 69.61 | 91.45
Dual | - | 74.69 | 93.82
  3. 3.

In the final set of experiments we scrutinize the way in which the information
from vehicle key-point heatmaps are incorporated in the proposed model. Our
work is in some aspects similar to [23] which groups a fixed set of key-points
and combines all the corresponding heatmaps into one map by adding them
together. Therefore, we conduct this experiment under the same settings as of
[23]. Table 8 shows the results of these experiments. The type ”Combined” in
Table 8 refers to the method in [23]. We can conclude that using all heatmaps
combined into one group does not result in competitive results as the adaptive
selection of heatmaps. This validates the hypothesis that not all the key-
points contribute to a discriminative representation of the vehicle.

Table 8: Experiment 3: Key-points heatmaps utilization

Dataset | Type | mAP | CMC@1 | CMC@5
---|---|---|---|---
VeRi-776 | Combined[23] | 0.606 | 87.66 | 94.17
AAVER | 0.612 | 88.97 | 94.70
VehicleID | Combined[23] | - | 71.79 | 92.10
AAVER | - | 74.69 | 93.82

##  6 Conclusions and Future Work

In this paper, we present a robust end-to-end framework for state of the art
vehicle re-identification. We present a dual path model AAVER which combines
macroscopic global features with localized discriminative features to
efficiently identify a probe image in a gallery of varying sizes. In addition,
we establish benchmarks for key-point detection and orientation prediction on
VeRi-776 dataset. Lastly, we advocate for the adoption of re-ranking when
considering the performance of future vehicle re-identification methods.
Adaptive key-point selection conditioned on vehicle orientation is vital for
discriminating between vehicles of the same make, model and color. Evaluating
on both VeRi-776 and VehicleID shows the strength of our proposed method.
Lastly, we conduct an ablation study to understand the influence of the
adaptive key-point selection step.

In the future, we plan to extend our key-point module to align vehicle images
to a canonical coordinates system before comparing a given pair of images.
Similarly, we can learn a 3D representation of vehicles to be used in other
tasks such as vehicular speed estimation.

##  7 Acknowledgement

This research is supported in part by the Northrop Grumman Mission Systems
Research in Applications for Learning Machines (REALM) initiative, It is also
supported in part by the Office of the Director of National Intelligence
(ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA
R&D Contract No. D17PC00345. The views and conclusions contained herein are
those of the authors and should not be interpreted as necessarily representing
the official policies or endorsements, either expressed or implied, of ODNI,
IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce
and distribute reprints for Governmental purposes notwithstanding any
copyright annotation thereon.

## References

  * [1] Y. Bai, Y. Lou, F. Gao, S. Wang, Y. Wu, and L. Duan.  Group-sensitive triplet embedding for vehicle reidentification.  IEEE Transactions on Multimedia, 20(9):2385–2399, Sep. 2018.
  * [2] Adrian Bulat and Georgios Tzimiropoulos.  Human pose estimation via convolutional part heatmap regression.  In European Conference on Computer Vision, pages 717–732. Springer, 2016.
  * [3] Ondrej Chum, James Philbin, Josef Sivic, Michael Isard, and Andrew Zisserman.  Total recall: Automatic query expansion with a generative feature model for object retrieval.  In 2007 IEEE 11th International Conference on Computer Vision, pages 1–8. IEEE, 2007.
  * [4] C. Cui, N. Sang, C. Gao, and L. Zou.  Vehicle re-identification by fusing multiple deep neural networks.  In International Conference on Image Processing Theory, Tools and Applications (IPTA), pages 1–6, 2017.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  2009\.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [7] Herve Jegou, Hedi Harzallah, and Cordelia Schmid.  A contextual dissimilarity measure for accurate and efficient image search.  In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1–8. IEEE, 2007.
  * [8] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [9] Amit Kumar and Rama Chellappa.  Disentangling 3d pose in a dendritic cnn for unconstrained 2d face alignment.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [10] Hongye Liu, Yonghong Tian, Yaowei Wang, Lu Pang, and Tiejun Huang.  Deep relative distance learning: Tell the difference between similar vehicles.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2167–2175, 2016.
  * [11] X. Liu, W. Liu, H. Ma, and H. Fu.  Large-scale vehicle re-identification in urban surveillance videos.  In IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2016.
  * [12] Xinchen Liu, Wu Liu, Tao Mei, and Huadong Ma.  A deep learning-based approach to progressive vehicle re-identification for urban surveillance.  In European Conference on Computer Vision, pages 869–884. Springer, 2016.
  * [13] X. Liu, W. Liu, T. Mei, and H. Ma.  Provid: Progressive and multimodal vehicle reidentification for large-scale urban surveillance.  IEEE Transactions on Multimedia, 20(3):645–658, 2018.
  * [14] Xiaobin Liu, Shiliang Zhang, Qingming Huang, and Wen Gao.  Ram: a region-aware deep model for vehicle re-identification.  In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2018.
  * [15] Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, and Ling-Yu Duan.  A large-scale dataset for vehicle re-identification in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [16] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In European Conference on Computer Vision, pages 483–499. Springer, 2016.
  * [17] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [18] Danfeng Qin, Stephan Gammeter, Lukas Bossard, Till Quack, and Luc Van Gool.  Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors.  In CVPR 2011, pages 777–784. IEEE, 2011.
  * [19] Rajeev Ranjan, Carlos D Castillo, and Rama Chellappa.  L2-constrained softmax loss for discriminative face verification.  arXiv preprint arXiv:1703.09507, 2017.
  * [20] Y. Shen, T. Xiao, H. Li, S. Yi, and X. Wang.  Learning deep neural networks for vehicle re-id with visual-spatio-temporal path proposals.  In IEEE International Conference on Computer Vision (ICCV), pages 1918–1927, 2017.
  * [21] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [22] Y. Tang, D. Wu, Z. Jin, W. Zou, and X. Li.  Multi-modal metric learning for vehicle re-identification in traffic surveillance environment.  In IEEE International Conference on Image Processing (ICIP), pages 2254–2258, 2017.
  * [23] Z. Wang, L. Tang, X. Liu, Z. Yao, S. Yi, J. Shao, J. Yan, S. Wang, H. Li, and X. Wang.  Orientation invariant feature embedding and spatial temporal regularization for vehicle re-identification.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 379–387, 2017.
  * [24] F. Wu, S. Yan, J. S. Smith, and B. Zhang.  Joint semi-supervised learning and re-ranking for vehicle re-identification.  In IEEE Conference on Pattern Recognition (ICPR), 2018.
  * [25] Jing Xu, Rui Zhao, Feng Zhu, Huaming Wang, and Wanli Ouyang.  Attention-aware compositional network for person re-identification.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [26] K. Yan, Y. Tian, Y. Wang, W. Zeng, and T. Huang.  Exploiting multi-grain ranking constraints for precisely searching visually-similar vehicles.  In IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [27] Linjie Yang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  A large-scale car dataset for fine-grained categorization and verification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3973–3981, 2015.
  * [28] Matthew D. Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  CoRR, abs/1311.2901, 2013.
  * [29] Y. Zhang, D. Liu, and Z.-J. Zha.  Improving triplet-wise training of convolutional neural network for vehicle re-identification.  In IEEE International Conference on Multimedia and Expo (ICME), pages 1386–1391, 2017.
  * [30] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li.  Re-ranking person re-identification with k-reciprocal encoding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1318–1327, 2017.
  * [31] Y Zhou and L Shao.  Viewpoint-aware attentive multi-view inference for vehicle re-identification.  In Proc. IEEE Conf. Comp. Vis. Patt. Recogn, volume 2, 2018.

## Supplementary Material

In this section we present the performance of our method on the newly released
VeRi-Wild dataset[15]. This dataset is collected in a network of 174
surveillance cameras covering a large urban area and captures unconstrained
scenarios. It is composed of 416,314 images (277,797/138,517 for train/test
sets) of 40,671 different vehicle identities. The test set, similar to
VehicleID dataset is divided into three Small (41,861), Medium (69,389) and
Large (138,517) images splits. The evaluation for this dataset follows the
same protocol as VeRi-776. Table 9 summarizes the result of our baseline and
proposed AAVER model.

Table 9: Performance comparison between baseline and proposed method on VeRi-
Wild dataset

|  | Baseline Model | AAVER Model
---|---|---|---
| Split | ResNet-50 | ResNet-101 | ResNet-50 | ResNet-101
mAP | Small | 60.22 | 60.99 | 60.62 | 62.23
Medium | 51.21 | 52.49 | 51.77 | 53.66
Large | 38.89 | 38.99 | 40.42 | 41.68
CMC@1 | Small | 71.37 | 72.97 | 74.60 | 75.80
Medium | 62.84 | 65.02 | 65.76 | 68.24
Large | 52.00 | 54.65 | 56.03 | 58.69
CMC@5 | Small | 90.10 | 91.57 | 91.60 | 92.70
Medium | 86.58 | 87.30 | 87.16 | 88.88
Large | 77.48 | 79.52 | 79.67 | 81.59

From Table 9 it can be seen that for all splits of VeRi-Wild dataset like
VeRi-776 and VehicleID datasets, a significant boost is obtained by
conditioning the features on the vehicle’s orientation and corresponding key-
points.

Figure 7 shows an examples of a query from VeRi-Wild dataset and the top three
results returned by both global and adaptive attention models.

|  |  |  |
---|---|---
|  |

(a) Probe Image

(b) Rank $1$

(c) Rank $2$

(d) Rank $3$

(e) Rank$1$

(f) Rank $2$

(g) Rank $3$

Figure 7: Top three returned results of the baseline model (sub-figures b-d)
versus the AAVER model (sub-figures e-g) on VeRi-Wild dataset


11institutetext: Knowledge Technology, Department of Informatics, University
of Hamburg 11email<EMAIL_ADDRESS>
www.knowledge-technology.info

# DRILL: Dynamic Representations for Imbalanced Lifelong Learning

Kyra Ahrens (🖂)  Fares Abawi  Stefan Wermter

###### Abstract

Continual or lifelong learning has been a long-standing challenge in machine
learning to date, especially in natural language processing (NLP). Although
state-of-the-art language models such as BERT have ushered in a new era in
this field due to their outstanding performance in multitask learning
scenarios, they suffer from forgetting when being exposed to a continuous
stream of non-stationary data. In this paper, we introduce DRILL, a novel
lifelong learning architecture for open-domain sequence classification. DRILL
leverages a biologically inspired self-organizing neural architecture to
selectively gate latent language representations from BERT in a domain-
incremental fashion. We demonstrate in our experiments that DRILL outperforms
current methods in a realistic scenario of imbalanced classification from a
data stream without prior knowledge about task or dataset boundaries. To the
best of our knowledge, DRILL is the first of its kind to use a self-organizing
neural architecture for open-domain lifelong learning in NLP.

###### Keywords:

Continual Learning NLP Imbalanced Learning Self-organization BERT.

##  1 Introduction

Humans possess the ability to continuously acquire, reorganize, integrate, and
enrich linguistic concepts throughout their lives. As early as infancy and
based on an innate inference capability, the conventional symbols of language
are learned within a socio-communicative context. The underlying neuro-
cognitive mechanisms involved in human language acquisition are still far from
being fully understood. However, they offer great potential for computational
models that are inspired by the neuroanatomical mechanisms in the mammalian
brain, enabling the continual integration of consolidated linguistic knowledge
with current experience [27].

With the advent of deep learning and the surge of computational resources and
data collection, state-of-the-art transformer-based language models (LM) such
as BERT [5] and OpenAI GPT [20] have gradually moved away from the symbolic
level and given way to isolated learning solutions revealing an outstanding
performance on downstream NLP tasks in a multitask set-up [4]. Yet when being
exposed to a sequence of tasks, catastrophic forgetting or catastrophic
interference of previously learned concepts was observed [17]. As re-training
on all prior data would be inefficient both in terms of computational cost and
memory capacity, this observation motivated the introduction of continual or
lifelong language learning (LLL).

Despite recent advances in LLL, most current methods make overly simplistic
assumptions that are in stark contrast to realistic, biologically inspired
learning settings. This includes enabling multiple passes over the input data
stream instead of single-epoch training, resulting in a surge of computational
cost. Such methods further rely on perfectly balanced and annotated data,
arranged in a way that the assumption of independent and identically
distributed samples holds. As a consequence, they are poorly applicable to
few-shot, unsupervised, or self-supervised learning scenarios [2].

Thus, striving for a more biologically grounded model architecture and
training set-up, we introduce DRILL, a text classification model applicable to
CL settings that involve the presence of a continuous stream of imbalanced
data without prior knowledge about task boundaries or probability
distributions. DRILL is a hybrid architectural and rehearsal-based CL method
that uses meta-learning and a self-organizing neural architecture to enable
rapid adaptation to novel data while minimizing catastrophic forgetting.

Due to the lack of a continual text classification benchmark of imbalanced
data, we introduce two sampling strategies to induce class imbalance
artificially. These strategies are evaluated on five text classification
datasets presented by Zhang et al. [30], commonly used as CL benchmarks in
NLP. With this setting, we show in our experiments that our model outperforms
current baselines while better generalizing to unseen data.

##  2 Related Work

###  2.1 Continual Learning

Striving for a balance between memory consolidation and generalization to new
input data from non-stationary distributions, also referred to as the
stability-plasticity dilemma [7], paved the way for various LLL approaches in
recent years. These approaches can be fully or partially categorized into
regularization, rehearsal, and dynamic architectures:

Regularization-based approaches constrain the plasticity of a learning model
either by introducing additional loss terms for weight adaptation at a fixed
model capacity [13, 29], or by setting an additional constraint on prior
tasks’ predictions to be kept invariant using knowledge distillation [15].
This fixed-capacity paradigm contrasts with architecture-based CL models that
assign some model capacity to each task and therefore dynamically expand in
response to novel input [18, 22]. Inspired by the concept of memory
consolidation, rehearsal-based (or memory replay) approaches maintain
performance on prior tasks by storing and retraining the model on old training
samples from an episodic memory [3, 16, 21].

To limit the associated memory overhead with an increasing number of tasks,
pseudo-rehearsal approaches employing generative network architectures have
been proposed. Such models rely on experience replay of task-representative
samples or latent representations based on statistical properties learned from
old training data [11, 19]. Two such generative replay approaches based on
GPT-2 [20], i.e. Language Modelling for Lifelong Language Learning (LAMOL)
[24] and Distill and Replay (DnR) [25] view LLL through the lens of question
answering. DnR deviates from LAMOL in that it bounds model complexity through
knowledge distillation following a teacher-student strategy. Although both
approaches are the current performance leaders on datasets benchmarked in this
work, they require multiple epochs of training and explicit knowledge about
task boundaries. Such preconditions deviate from the realistic CL scenario we
advocate for in this work.

###  2.2 Meta-Learning

Meta-learning [26] has become increasingly popular in recent years as it paved
the way for sophisticated algorithms capable of quickly adapting to new data.
Online aware Meta-Learning (OML) [10] combines the common meta-learning
objective of maximizing fast adaptation to new tasks with the CL objective of
minimizing catastrophic interference during training. A Neuromodulated Meta-
Learning Algorithm (ANML) [1] extends OML by an independent representation
learning stream to selectively gate latent activations. Holla et al. [8]
introduce a sparse experience replay mechanism to OML and ANML, denoting their
two novel methods by OML-ER and ANML-ER respectively. Both extensions
outperform state-of-the-art methods for text classification and question
answering benchmarks under a training set-up in which data becomes only
available over time and a lack of information about when a dataset or task
boundary is crossed.

###  2.3 Growing Memory and Self-Organization

In an attempt to mimic the explicit memory formation in the mammalian brain,
early artificial neural networks based on competitive learning mechanisms and
self-organization have been developed and refined [6, 14]. One more recent
extension to such topology learning methods is the Self-organizing Incremental
Neural Network (SOINN) algorithm, which regulates plasticity in unsupervised
learning tasks by means of dynamically creating, adapting, and deleting
neurons [23]. SOINN+ [28] extends the original SOINN algorithm by introducing
a novel node deletion mechanism based on (i) idle time, (ii) trustworthiness,
and (iii) non-usage of a network unit. Given that SOINN+ successfully
demonstrates its resilience to noisy data and its ability to learn a high-
quality topology from the input domain while keeping the number of nodes
small, we utilize it as a semantic memory component in our DRILL architecture.

##  3 Methods

With the challenge of achieving LLL from unbalanced data in mind, we lay the
theoretical foundation for our proposed DRILL method.

###  3.1 Task Formulation

Consider an ordered sequence of tasks
$\mathcal{T}=\\{T_{1},T_{2},\dots,T_{N}\\}$, where we observe $n_{k}$
annotated input samples from the $k$-th task, i.e.
$T_{k}=\\{(\boldsymbol{x}^{i}_{k},y^{i}_{k})\\}^{n_{k}}_{i=1}$ drawn from the
distribution $P_{k}(\mathcal{X},\mathcal{Y})$. Assuming a realistic scenario
of missing task and dataset descriptors, we have no knowledge about which task
each input sample belongs to. Following prior work [8], we define task in
terms of text classification domain, i.e. sentiment, news topic, question-and-
answer, and ontology. Our objective is to learn a model
$f_{\theta}:\mathcal{X}\to\mathcal{Y}$ with parameters $\theta$ to minimize
the negative log-likelihood averaged across all $N$ tasks

$\mathcal{L}(\theta)=-\frac{1}{N}\sum_{k=1}^{N}\text{ln}\,P(\boldsymbol{x}_{k}\,|\,y_{k}\,;\,\theta)$
(1)

###  3.2 Progressive Imbalancing

Prior work on lifelong text classification [8, 16, 24, 25] has traditionally
deployed the perfectly balanced version of the five NLP datasets by Zhang et
al. [30]. Following the idea of d’Autume et al. [16], we introduce two
sampling techniques called progressive reduction ($R$) and progressive
expansion ($E$), which exponentially increase or decrease the number of
samples for each incoming task, such that

$n^{R}_{k+1}\leftarrow\left\lfloor\frac{n^{R}_{k}}{2}\right\rfloor$ (2)

with progressive reduction and

$n^{E}_{k+1}\leftarrow 2\cdot n^{E}_{k}$ (3)

with progressive expansion respectively, and $k\in\\{1,\dots,N\\}$. Both
sampling techniques allow us to simulate two opposite LLL settings in which
data at an early or late stage are significantly less present.

###  3.3 Episode Generation

For the construction of training episodes and experience rehearsal from
episodic memory, we follow a commonly adopted set-up [8, 16]:

Under the assumption that samples arrive in batches of size $s$ and are
written into episodic memory module $\mathcal{M}_{\mathcal{E}}$ with
probability $p_{\mathcal{E}}$, we construct the $i$-th episode from $b$
batches, where the first $b-1$ batches denote support set $\mathcal{S}_{i}$
and the $b$-th batch denotes query set $\mathcal{Q}_{i}$.

After having observed $R_{I}$ samples from the stream, $\left\lfloor r\cdot
R_{I}\right\rfloor$ samples from $\mathcal{M}_{\mathcal{E}}$ are randomly
being drawn for rehearsal, where $r\in[0,1]$ denotes the predefined replay
ratio.

Aligned with the episodic fashion of meta-learning, we calculate the replay
frequency

$R_{F}=\left\lceil\frac{R_{I}/s+1}{b}\right\rceil$ (4)

Thus, every $R_{F}$-th episode can be considered as replay episode in a way
that its query set does not consist of data from the stream, but from the
episodic memory module $\mathcal{M}_{\mathcal{E}}$.

###  3.4 DRILL

The DRILL architecture comprises four main elements, namely a dual-memory
system of (1) an episodic memory module $\mathcal{M}_{\mathcal{E}}$ and (2) a
semantic memory module $\mathcal{M}_{\mathcal{S}}$, and, following the
original OML algorithm [10], (3) a representation learning network (RLN)
$h_{\phi}$ as well as (4) a prediction learning network (PLN)
$g_{\boldsymbol{W}}$.

Figure 1: Overview of the two variants $\text{DRILL}_{\text{M}}$ (left) and
$\text{DRILL}_{\text{C}}$ (right). Latent representation signals retrieved
from RLN are integrated with neural weight signals from
$\mathcal{M}_{\mathcal{S}}$ either by multiplication
($\text{DRILL}_{\text{M}}$) or concatenation ($\text{DRILL}_{\text{C}}$).
Input to the model is either an new observation $\boldsymbol{x}$ from the
stream or an episodic replay sample from $\mathcal{M}_{\mathcal{E}}$.

We use the SOINN+ [28] algorithm as semantic memory module
$\mathcal{M}_{\mathcal{S}}$. Each neural unit is a $d$-dimensional real-valued
vector. The network is parameterized by a pull factor $\eta$ denoting the
influence of a new observation on neighboring nodes. For the sake of
simplicity and as proposed by Wiwatcharakoses and Berrar [28], we set the pull
factor to a constant value $\eta=50$. Thus, our model $f_{\theta}$ optimizes
for the set of parameters $\theta=\phi\,\cup\,\boldsymbol{W}$, consisting of
parametrization $\phi$ from $\mathcal{X}\to\mathbb{R}^{d}$ of the RLN
$h_{\phi}$ and $\boldsymbol{W}$ from $\mathbb{R}^{d}\to\mathcal{Y}$ of the PLN
$g_{\boldsymbol{W}}$ respectively.

Taking inspiration from the mammalian thalamus as a ‘gate to consciousness’,
we propose two DRILL variants that differ in how latent representations from
the RLN are integrated with signals from the semantic memory
$\mathcal{M}_{\mathcal{S}}$, translating its internal selective plasticity to
the entire learning process.

The first variant, called Integration by Multiplication
($\text{DRILL}_{\text{M}}$), can be described as follows: On receiving input
$\boldsymbol{x}$, the model gates the activations $h_{\phi}(\boldsymbol{x})$
arriving from the RLN by multiplying them element-wise with a set of neural
weights $\boldsymbol{w}_{\mathcal{S}}$ drawn from $\mathcal{M}_{\mathcal{S}}$
in a procedure described in subsection 3.5. We express the model variant
$\text{DRILL}_{\text{M}}$ as

$f^{M}_{\theta}(\boldsymbol{x})=g_{\boldsymbol{W}}(\boldsymbol{w}_{\mathcal{S}}\cdot
h_{\phi}(\boldsymbol{x}))$ (5)

For the second variant called Integration by Concatenation
($\text{DRILL}_{\text{C}}$), each of the $d$-dimensional signals
$\boldsymbol{w}_{\mathcal{S}}$ and $h_{\phi}(\boldsymbol{x})$ retrieved from
$\mathcal{M}_{\mathcal{S}}$ and the RLN respectively are reduced to half of
their dimension $\frac{d}{2}$ and subsequently concatenated in a
$d$-dimensional linear layer that is allocated to the PLN, as shown in Figure
1. Thus, we derive the following model

$f^{C}_{\theta}(\boldsymbol{x})=g_{\boldsymbol{W}}(\,\left[\boldsymbol{w}_{\mathcal{S}},h_{\phi}(\boldsymbol{x})\right]\,)$
(6)

where $\left[\cdot,\cdot\right]$ denotes the concatenation operator. The meta-
learning procedure for both DRILL variants works as follows: During inner-loop
optimization of the $i$-th episode, the RLN is kept frozen while the PLN is
fine-tuned using SGD with an inner-loop learning rate $\alpha$, such that

$\boldsymbol{W}^{\prime}\leftarrow\text{SGD}(\mathcal{L}_{i}(\phi,\boldsymbol{W}),\mathcal{S}_{i},\alpha)$
(7)

Subsequently, both RLN and PLN are fine-tuned on the query set
$\mathcal{Q}_{i}$ during outer-loop optimization, such that all model
parameters are updated using the Adam optimizer [12] with an outer-loop
learning rate $\beta$ to give

$\theta^{\prime}\leftarrow\text{Adam}(\mathcal{L}_{i}(\phi,\boldsymbol{W}^{\prime}),\mathcal{Q}_{i},\beta)$
(8)

For the RLN, we use the state-of-the-art transformer-based language model
$\text{BERT}_{\text{BASE}}$ [5] with 12 transformer layers and $d=768$ hidden
dimensions. With $\text{DRILL}_{\text{M}}$, the PLN is a single linear layer
with softmax activation that outputs the class probabilities, while a linear
concatenation layer additionally precedes this layer with
$\text{DRILL}_{\text{C}}$.

###  3.5 Self-Supervised Sampling

In contrast to the episodic memory $\mathcal{M}_{\mathcal{E}}$ that we solely
use for experience replay, we use the semantic memory
$\mathcal{M}_{\mathcal{S}}$ for generating high-quality representations, which
influence the fine-tuning of the PLN. For every input sample
$(\boldsymbol{x}_{i},y_{i})$, we initiate a competitive voting mechanism among
all nodes in $\mathcal{M}_{\mathcal{S}}$ to determine the two neurons with
neural weights $\boldsymbol{w}_{\mathcal{S}}^{1}$ and
$\boldsymbol{w}_{\mathcal{S}}^{2}$ that have most frequently been best-
matching units (BMUs) for class $y_{i}$. According to the original SOINN+
algorithm [28], the network node that lies closest to the input in Euclidean
space is denoted as BMU.

The two winners are then either multiplied element-wise
($\text{DRILL}_{\text{M}}$) or concatenated ($\text{DRILL}_{\text{C}}$) with
the activations of the latent representation $h_{\phi}(\boldsymbol{x}_{i})$
coming from the RLN, thus generating two inputs to the PLN from one output of
the RLN. During the evaluation phase, only one signal from the winning node
$\boldsymbol{w}_{\mathcal{S}}$ is retrieved from $\mathcal{M}_{\mathcal{S}}$
for the purpose of unambiguous label prediction by the PLN.

##  4 Experiments

###  4.1 Benchmark Datasets

We train our model sequentially on five text classification datasets by Zhang
et al. [30] covering four different tasks: Sentiment analysis, news topic
detection, question-and-answer classification, and ontology categorization. We
summarize them in Table 1. Following d’Autume et al. [16], the datasets are
arranged in four randomized permutations reflecting the significant impact of
task ordering on evaluation results.

Table 1: The five balanced text classification datasets as in Zhang et al. [30], each containing 7,600 test samples randomly drawn from the original datasets. The number of training samples differs depending on order position and imbalanced sampling strategy. |  |  | Order Position
---|---|---|---
Classification Domain |  Dataset |  Classes |  I |  II |  III |  IV
|  Amazon |  |  4 |  4 |  3 |  3
Sentiment |  Yelp |  5 (merged) |  1 |  5 |  1 |  2
News Topic |  AGNews |  4 |  2 |  3 |  5 |  1
Question Topic |  Yahoo |  10 |  5 |  2 |  2 |  4
Ontology |  DBPedia |  14 |  3 |  1 |  4 |  5
Total: |  33 |

For evaluation, we follow prior work [8, 16, 24, 25] and randomly draw 7,600
samples from each of the five datasets, yielding a total test size of 38,000.
However, we depart from the perfectly balanced and thus poorly realistic
scenario of 115,000 training samples per dataset and instead apply progressive
imbalancing as described in subsection 3.2 with $n_{0}^{R}=115,000$ and
$n_{0}^{E}=7,187$, thus providing a total training size of $222,812$ for
either sampling strategy.

###  4.2 Baselines

For performance evaluation, we compare our two proposed model variations
$\textbf{DRILL}_{\textbf{M}}$ and $\textbf{DRILL}_{\textbf{C}}$ with the two
performance leaders given a realistic single-epoch set-up without prior task-
specific knowledge, i.e. ANML-ER and OML-ER [8]. Just like our method, they
use a pretrained $\text{BERT}_{\text{BASE}}$ language encoder. We further
implement the lower bound for CL model performance, SEQ, in which we fine-tune
both RLN and PLN on all tasks sequentially without any rehearsal. We also
compare our methods with REPLAY, an extension of SEQ towards experience
rehearsal with samples stored in an episodic memory. Finally, we train RLN and
PLN jointly in a multitask set-up MTL, which we consider as an upper bound for
CL model performance. For a fair comparison, we choose the same memory-write
and rehearsal policies for REPLAY, ANML-ER, and OML-ER, as well as our two
proposed DRILL variants.

###  4.3 Implementation Details

Our experimental set-up consists of three independent runs on seeds 42-44,
each run performed on the four order permutations and two sampling strategies
respectively. Accordingly, the comparison results are averaged over all three
runs.

Due to computational limitations, we train all baseline models on normalized
batches of size $s=8$ following the procedure of Ioffe and Szegedy [9] and
optimize based on the cross-entropy loss on all 33 classes. We truncate the
$\text{BERT}_{\text{BASE}}$ input sequences to length $448$ and set the buffer
size $b=6$. The inner-loop and outer-loop learning rates of the four meta-
learning-based models $\text{DRILL}_{\text{M}}$, $\text{DRILL}_{\text{C}}$,
OML-ER, and ANML-ER are set to $\alpha=8\mathrm{e}{-3}$ and
$\beta=1.5\mathrm{e}{-5}$ respectively. The learning rate of all remaining
baselines SEQ, REPLAY, and MTL is set to $1\mathrm{e}{-5}$.

All models are trained for a single epoch, whereas MTL is trained for two
epochs. The probability of storing an observation in the episodic memory
module $\mathcal{M}_{\mathcal{E}}$ is governed by the maximum write
probability $p_{\mathcal{E}}=0.8$. The $p_{\mathcal{E}}$ is inversely
proportional to the expansion or reduction for all rehearsal-based models,
restoring class balance within $\mathcal{M}_{\mathcal{E}}$. The learning rates
and $p_{\mathcal{E}}$ are derived using a Parzen–Rosenblatt estimator
111CometML Hyperparameter Optimizer: https://www.comet.ml/. The hyperparameter
optimization is applied to OML-ER as the representative model for all meta-
learning-based approaches and SEQ for inferring the learning rate of the
remaining models. Both OML and SEQ are trained on the full dataset (without
expansion or reduction) with order I and random seed 42. With both DRILL
architectures, the unsupervised SOINN+ algorithm is performed as described in
the original paper [28], including setting the pull factor $\eta=50$.

We follow the rehearsal and evaluation strategies adopted by Holla et al. [8],
setting $R_{I}=9,600$ and $r=1\%$, such that we draw 96 samples from
$\mathcal{M}_{\mathcal{E}}$ after observing 9,600 samples from the data
stream. The evaluation of the four meta-learning models is performed by
generating five episodes, each containing the test datasets as query sets. All
baseline models were trained on an NVIDIA TITAN RTX with 24GB VRAM and 64GB
RAM. The training time ranges between 1 and 7 hours, depending on the model
and number of observations.

##  5 Results

###  5.1 Imbalanced Lifelong Text Classification

Table 2: Text classification $F_{1}$ scores on four permutations of task orders and progressive expansion ($E$) and progressive reduction ($R$) sampling respectively. The two rightmost columns denote the macro-average and standard deviation across all orderings and sampling strategies. | Order ($E$) | Order ($R$) |  |
---|---|---|---|---
Method |  I |  II |  III |  IV |  I |  II |  III |  IV |  $\mathbf{\mu}$ |  $\mathbf{\sigma}$
SEQ |  $17.4$ |  $27.6$ |  $26.6$ |  $21.0$ |  $23.7$ |  $32.7$ |  $28.8$ |  $25.0$ |  $25.4$ |  $4.9$
REPLAY |  $55.3$ |  $67.9$ |  $58.6$ |  $\mathbf{65.7}$ |  $44.2$ |  $57.7$ |  $53.5$ |  $37.0$ |  $55.0$ |  $10.5$
ANML-ER |  $66.7$ |  $\mathbf{70.5}$ |  $55.0$ |  $62.9$ |  $57.0$ |  $58.6$ |  $62.7$ |  $45.2$ |  $59.8$ |  $8.8$
OML-ER |  $\mathbf{70.2}$ |  $64.9$ |  $52.2$ |  $64.4$ |  $56.0$ |  $\mathbf{62.0}$ |  $\mathbf{66.5}$ |  $48.7$ |  $60.6$ |  $8.1$
$\text{DRILL}_{\text{M}}$ |  $23.2$ |  $36.5$ |  $37.1$ |  $37.6$ |  $58.0$ |  $51.7$ |  $41.0$ |  $\mathbf{50.4}$ |  $41.9$ |  $13.7$
$\text{DRILL}_{\text{C}}$ |  $68.4$ |  $68.1$ |  $\mathbf{59.1}$ |  $65.5$ |  $\mathbf{61.8}$ |  $61.6$ |  $62.9$ |  $49.5$ |  $\mathbf{62.1}$ |  $6.2$
MTL |  $77.9$ |  $78.7$ |  $76.2$ |  $76.7$ |  $77.7$ |  $76.4$ |  $78.3$ |  $78.2$ |  $77.5$ |  $1.0$

Unlike prior work, we report $F_{1}$ scores rather than macro-averaged
classification accuracy due to the unbalanced nature of the training data. Our
main results are summarized in Table 2.

Our $\text{DRILL}_{\text{C}}$ variant outperforms existing methods in terms of
higher overall average performance and higher median under equal conditions
(the latter is depicted in Figure 2). In addition, it has a significantly
smaller variance than all other replay-based comparison methods, thus
demonstrating its robustness to the order of training data and the imbalancing
strategy. Consequently, it narrows the gap to the upper bound of multitask
learning.

Interestingly, the $\text{DRILL}_{\text{M}}$ method is trailing the current
models with respect to absolute performance. Yet, it provides a smaller
variance for progressively expanded data than all other baselines except SEQ,
exhibiting robustness against undersampled classes at the beginning of
training. The enormous performance difference of our two DRILL variants
motivates a more detailed analysis of the impact of knowledge integration
mechanisms from RLN and $\mathcal{M}_{\mathcal{S}}$.

Figure 2: $F_{1}$ scores of all comparison models aggregated across three
seeds and four orderings. Sequential (SEQ) and multitask (MTL) learning can be
viewed as lower and upper bound for model performance respectively.

###  5.2 Knowledge Integration Mechanisms

Although the introduction of class-representative signals drawn from semantic
memory yields greater robustness under a realistic training scenario, the
overall model performance varies greatly depending on how the latent signals
retrieved are integrated during training. The relatively poor performance of
$\text{DRILL}_{\text{M}}$ could be attributed to the multiplicative gating
mechanism that we adopted from the original ANML algorithm [1]. The ANML is
designed so that ‘gating parameters’ of preceding layers are learned in a
supervised fashion, which is in contrast to the unsupervised nature of SOINN+.

Conversely, with $\text{DRILL}_{\text{C}}$, signals from RLN are enriched with
those from the SOINN rather than fused, allowing for better linear separation,
thus resulting in an increase of model performance. From this, we conclude
that the concatenation of modalities in our training scenario provides a
better knowledge retention strategy.

###  5.3 Self-Organized Networks in NLP

A generally known problem of self-organizing networks is that they capture the
entire evolution of hidden representations in feature space along with
obsolete knowledge and are therefore unsuitable for training on shifting
latent distributions. With the DRILL architecture, we overcome this problem by
freezing the RLN parameters during inner-loop optimization and by the choice
of our retrieval strategy for neural weight signals coming from the SOINN. The
former leads to a more stable latent data distribution over a longer period.
The latter ensures that neural units residing in the current input
distribution are more likely to be considered as high-quality class
representatives.

As this is the first work to combine a self-organizing neural architecture
with a transformer-based language model in a CL setting, we advocate further
exploring such set-ups in future work. This is due to the intrinsic ability of
the SOINN and its various extensions to be applicable in an infinite learning
setting with an unlimited number of tasks. The model can additionally handle
partially annotated data, setting the basis for semi-supervised LLL scenarios.

##  6 Conclusion and Future Work

In this work, we introduce a novel, more challenging continual learning set-up
with imbalanced data. We further propose Dynamic Representations for
Imbalanced Lifelong Learning (DRILL), a neuroanatomically inspired CL method
which combines a state-of-the-art language model with a self-organizing neural
architecture. It outperforms current baselines, yet is more stable against
data ordering and imbalancing. Thus, the fusion of supervised language models
with unsupervised clustering algorithms has proven effective for lifelong
learning methods, further narrowing the gap to multitask learning approaches.
DRILL achieves the best results on imbalanced data, with the least overall
variance in comparison to other meta-learning-based lifelong learning
approaches. For future work, we plan to extend our model towards infinite
learning of an unknown number of tasks as well as sequence-to-sequence
learning.

## Acknowledgements

We would like to thank Dr. Cornelius Weber (University of Hamburg) and Katja
Kösters (University of Hamburg) for their feedback and suggestions. The
authors gratefully acknowledge partial support from the German Research
Foundation (DFG) under Project CML (TRR-169).

## References

  * [1] Beaulieu, S., Frati, L., Miconi, T., Lehman, J., Stanley, K.O., Clune, J., Cheney, N.: Learning to Continually Learn. In: 24th European Conference on Artificial Intelligence. vol. 325, pp. 992–1001. IOS Press (2020)
  * [2] Biesialska, M., Biesialska, K., Costa-jussà, M.R.: Continual Lifelong Learning in Natural Language Processing: A Survey. In: 28th International Conference on Computational Linguistics. pp. 6523–6541. International Committee on Computational Linguistics (2020)
  * [3] Chaudhry, A., Marc’Aurelio, R., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with A-GEM. In: 7th International Conference on Learning Representations (2019)
  * [4] Chen, Z., Liu, B.: Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 12(3), 1–207 (2018)
  * [5] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics. pp. 4171–4186. Association for Computational Linguistics (2019)
  * [6] Fritzke, B., et al.: A growing neural gas network learns topologies. Advances in Neural Information Processing Systems 7, 625–632 (1995)
  * [7] Grossberg, S.: How does a brain build a cognitive code? Studies of mind and brain pp. 1–52 (1982)
  * [8] Holla, N., Mishra, P., Yannakoudakis, H., Shutova, E.: Meta-Learning with Sparse Experience Replay for Lifelong Language Learning. arXiv preprint arXiv:2009.04891 (2020)
  * [9] Ioffe, S., Szegedy, C.: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In: 32nd International Conference on Machine Learning. vol. 37, pp. 448–456. PMLR (2015)
  * [10] Javed, K., White, M.: Meta-Learning Representations for Continual Learning. Advances in Neural Information Processing Systems 32, 1820–1830 (2019)
  * [11] Kemker, R., Kanan, C.: FearNet: Brain-inspired model for incremental learning. In: 6th International Conference on Learning Representations (2018)
  * [12] Kingma, D.P., Ba, J.L.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations (2015)
  * [13] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America 114(13), 3521–3526 (2017)
  * [14] Kohonen, T.: The self-organizing map. Proceedings of the IEEE 78(9), 1464–1480 (1990)
  * [15] Li, Z., Hoiem, D.: Learning without Forgetting. IEEE transactions on pattern analysis and machine intelligence 40(12), 2935–2947 (2017)
  * [16] de Masson d’Autume, C., Ruder, S., Kong, L., Yogatama, D.: Episodic Memory in Lifelong Language Learning. Advances in Neural Information Processing Systems 32 (2019)
  * [17] McCloskey, M., Cohen, N.J.: Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. In: Bower, G.H. (ed.) Psychology of Learning and Motivation, Psychology of Learning and Motivation, vol. 24, pp. 109–165. Academic Press (1989)
  * [18] Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S.: Continual lifelong learning with neural networks: A review. Neural Networks 113, 54–71 (2019)
  * [19] Parisi, G.I., Tani, J., Weber, C., Wermter, S.: Lifelong Learning of Spatiotemporal Representations With Dual-Memory Recurrent Self-Organization. Frontiers in neurorobotics 12, 78 (2018)
  * [20] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving Language Understanding by Generative Pre-Training. OpenAI (2018)
  * [21] Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: Incremental Classifier and Representation Learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [22] Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016)
  * [23] Shen, F., Hasegawa, O.: Self-Organizing Incremental Neural Network and Its Application. In: Diamantaras, K., Duch, W., Iliadis, L.S. (eds.) International Conference of Artificial Neural Networks. vol. 6354, pp. 535–540. Springer, Berlin, Heidelberg (2010)
  * [24] Sun, F.K., Ho, C.H., Lee, H.Y.: LAMOL: LAnguage MOdeling for Lifelong Language Learning. In: 8th International Conference on Learning Representations (2020)
  * [25] Sun, J., Wang, S., Zhang, J., Zong, C.: Distill and Replay for Continual Language Learning. In: 28th International Conference on Computational Linguistics. pp. 3569–3579. International Committee on Computational Linguistics, Barcelona, Spain (2020)
  * [26] Thrun, S., Pratt, L.: Learning to learn: Introduction and overview. In: Learning to learn, pp. 3–17. Springer (1998)
  * [27] Tomasello, M.: The social bases of language acquisition. Social development 1(1), 67–87 (1992)
  * [28] Wiwatcharakoses, C., Berrar, D.: SOINN+, a Self-Organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams. Expert Systems with Applications 143, 113069 (2020)
  * [29] Zenke, F., Poole, B., Ganguli, S.: Continual Learning Through Synaptic Intelligence. In: 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70, pp. 3987–3995. PMLR (2017)
  * [30] Zhang, X., Zhao, J., LeCun, Y.: Character-Level Convolutional Networks for Text Classification. In: 28th International Conference on Neural Information Processing Systems. pp. 649–657. MIT Press (2015)