# Multi-Task Self-Training for Learning General Representations

Golnaz Ghiasi, Barret Zoph11footnotemark: 1, Ekin D. Cubuk11footnotemark: 1,
Quoc V. Le, Tsung-Yi Lin
Google Research, Brain Team
<EMAIL_ADDRESS>Authors contributed equally.

###### Abstract

Despite the fast progress in training specialized models for various tasks,
learning a single general model that works well for many tasks is still
challenging for computer vision. Here we introduce multi-task self-training
(MuST), which harnesses the knowledge in independent specialized teacher
models (_e.g_., ImageNet model on classification) to train a single general
student model. Our approach has three steps. First, we train specialized
teachers independently on labeled datasets. We then use the specialized
teachers to label an unlabeled dataset to create a multi-task pseudo labeled
dataset. Finally, the dataset, which now contains pseudo labels from teacher
models trained on different datasets/tasks, is then used to train a student
model with multi-task learning. We evaluate the feature representations of the
student model on 6 vision tasks including image recognition (classification,
detection, segmentation) and 3D geometry estimation (depth and surface normal
estimation). MuST is scalable with unlabeled or partially labeled datasets and
outperforms both specialized supervised models and self-supervised models when
training on large scale datasets. Lastly, we show MuST can improve upon
already strong checkpoints [24] trained with billions of examples. The results
suggest self-training is a promising direction to aggregate labeled and
unlabeled training data for learning general feature representations.

##  1 Introduction

Visual representation learning is a core problem in computer vision.
Supervised and self-supervised pre-training have shown promising results in
transferring the learned feature representations to downstream tasks.
Typically, a model is pre-trained with a supervised [30, 11] or a self-
supervised objective [5, 17, 18]. Despite the wide adoption of transfer
learning from supervised training, the features may not necessarily be useful
for downstream tasks. For example, He _et al_. found that ImageNet pre-
training fails to improve COCO instance segmentation [19]. In contrast, Shao
_et al_. showed features learned from Objects365 detection dataset improve
COCO instance segmentation by a large margin [49]. Pre-training with a
specialized task that aligns with the downstream target task still yields the
best performance in object detection [34, 49] and semantic segmentation [4].

Figure 1: An overview of Multi-Task Self-Training (MuST). Specialized Teacher
represents a supervised model trained on a single task and dataset (_e.g_.,
classification model trained on ImageNet). Specialized Teacher models are
trained independently on their own tasks and datasets. They then generate
pseudo labels on a shared dataset. Finally, a single General Student model is
trained jointly using the pseudo (and supervised) labels on the shared
dataset.

Intuitively, it is possible to learn general features by training a model to
simultaneously do well on multiple tasks. Recent work in NLP started to show
promising results on learning a generalist model with multi-task learning [60,
9]. In computer vision, the biggest challenge of training a multi-task model
is in the data collection and annotation. Despite datasets like COCO [37],
collecting a wide variety of annotations (_e.g_., instance segmentation,
person keypoints, image caption) for the same image dataset is quite
challenging. Due to the time consuming nature of annotating images with
labels, it is hard to scale such efforts with the number of images and the
number of tasks. The lack of large scale multi-task datasets impedes the
progress in multi-task learning for computer vision.

In this work, we study using self-training to remedy the issue. We propose to
use pseudo labeling to enable large scale multi-task feature learning for
computer vision. Zoph _et al_. [67] observed that self-training further
improves pre-training for transfer learning, and that self-training works even
when pre-training fails to outperform a randomly initialized model. The gap
between pre-training and self-training suggests that self-training can learn
better features from pseudo labels. Inspired by this observation, we first
investigate whether good features can be learned by only using pseudo labels.
We train teacher models using datasets such as COCO or Objects365 to generate
pseudo labels on unlabeled images. Figure 2 shows example pseudo labels on
ImageNet. Surprisingly, we find a student model trained with only these pseudo
labels preserves most of the transfer learning performance of its specialized
teacher model. This finding suggests pseudo labels are effective at distilling
the knowledge in a supervised dataset. Therefore, we can use pseudo labels to
transfer knowledge from multiple teacher models to a single student model for
representation learning.

We propose Multi-Task Self-Training (MuST) to train a generalist student model
on the information distilled from teacher models trained on different tasks
and datasets. Figure 1 shows the overview of the algorithm. MuST has three
steps. First, it trains specialized teachers independently on labeled
datasets. For example, one teacher can be trained with depth prediction and
another teacher can be trained with object detection. The specialized teachers
are then used to label a larger unlabeled dataset to create a multi-task
pseudo labeled dataset. For example, these teachers can generate depth
estimations and object detections on the ImageNet dataset. Finally, the
dataset, which now contains pseudo labels from teacher models trained on
different datasets/tasks, is used to train a student model with multi-task
learning. Hence the student, for example, can do depth prediction and object
detection at the same time.

In our experiments, we have four teacher models: classification, semantic
segmentation, object box detection, and depth estimation. We design a simple
model architecture (Figure 3) based on ResNet [21] and feature pyramid
networks (FPN) [36]. The parameters in the ResNet-FPN backbone are shared
across different tasks. For each individual task, it has a small task-specific
head consisting of a few convolution layers followed by a linear prediction
layer. Our experiments show that this simple model architecture is able to
absorb the knowledge of different tasks in the shared backbone. The generalist
student model is on par with/outperforms its specialist teacher models for all
transfer learning tasks.

The recent self-supervised algorithms like SimCLR [5], MoCo [18] are shown to
create representations that are on par or better than its supervised
counterpart. In our experiments, MuST also outperforms SimCLR [5] by a large
margin on segmentation and depth estimation tasks. We also observe that the
representations learned by SimCLR is on par with those of supervised learning
on ImageNet (1.3M images) but does not scale as well on JFT (300M images). On
the contrary, MuST outperforms SimCLR [5] on both ImageNet and JFT. Moreover,
MuST also outperforms supervised JFT pre-training for 5 out of 6 tasks except
the image classification task. The results indicate the potential of MuST in
learning general feature representations that improve with more unlabeled
data.

Lastly, we show MuST can improve upon already strong checkpoints such as ALIGN
[24]. We fine-tune ALIGN checkpoints, previously trained with billions of
supervised examples, with MuST pseudo labels and find improvements on a suite
of downstream tasks: detection, segmentation, and depth estimation tasks.

We summarize our contributions below:

  * •

We propose Multi-Task Self-Training (MuST), a simple algorithm for creating
general visual representations by multi-task learning with pseudo labels.

  * •

We conduct experiments by jointly training across several datasets (_e.g_.,
ImageNet, Objects365, COCO, JFT) to learn general feature representations that
outperforms representations learned by supervised and self-supervised methods.

  * •

We perform experiments to compare supervised, self-supervised, and MuST on 6
computer vision tasks including tasks in image recognition (classification,
detection, segmentation) and 3D geometry estimation (depth and surface normal
estimation).

  * •

MuST can be used to improve upon already strong checkpoints and achieve
competitive results on a variety of tasks compared to task-specific state-of-
the-art models.

Figure 2: Examples of pseudo labels on ImageNet. Left: bounding boxes labeled
with an Objects365 teacher model. Middle: semantic segmentation labeled with a
COCO teacher model. Right: depth labeled with a MiDaS teacher model.

##  2 Related Work

#### Multi-Task learning:

Multi-task learning has a rich history in deep learning [46]. A common
strategy for multi-task learning is to share the hidden layers of a “backbone”
model for different tasks [2]. More recently, multi-task learning has led to
improved accuracy in NLP [9, 38]. Although, Raffel  _et al_. found that multi-
task learning generally underperformed compared to pre-training followed by
fine-tuning [42].

In the vision domain, Zamir  _et al_. studied the transfer learning
dependencies across 26 tasks with an indoor dataset [64]. Instead of exploring
the task dependencies, we are interested in pushing a single model that can
absorb knowledge of all tasks for learning general representations. Kokkinos
_et al_. [29] and Xiao _et al_. [57] trained models across multiple datasets
by simply zeroing losses for examples that don’t have labels for a particular
task. We propose to apply pseudo labels so every image is annotated with all
tasks. Girshick  _et al_. used a multi-task loss for classification and
bounding-box regression to improve the training of object detectors [15]. We
follow the similar approach of using one large backbone model and smaller
heads for multiple tasks.

#### Self-training:

Self-training is a popular technique to incorporate unlabeled data into
supervised learning [62, 48, 45, 33]. The method works by using a supervised
model to generate pseudo labels on unlabeled data. Then a student model is
trained on the pseudo labeled data. Yalniz _et al_. [61] showed a model “pre-
trained” with pseudo labels on a large unlabeled dataset (at hundreds millions
scale) can improve classification accuracy. Noisy Student [58] used self-
training to push state-of-the-art performance on ImageNet by training jointly
with 130M pseudo labeled images. Chen _et al_. [3] obtained state-of-the-art
panoptic segmentation results on Cityscapes with self-training. Zoph _et al_.
[67] improved the state-of-the-art on object detection and semantic
segmentation with self-training. All the above works focused on a single task.
On the contrary, our work focuses on using self-training for multi-task
learning to learn general representations.

#### Representation learning:

Transfer learning from ImageNet pre-training has been the most widely used
method in computer vision. BiT [30] and ViT [11] pre-trained the model on
JFT-300M dataset [51] and obtained strong performance when fine-tuned on
downstream vision tasks. In particular, Mahajan _et al_. showed model pre-
trained with Instagram benefits other classification tasks but possibly harms
localization performance [39]. Li  _et al_. found that OpenImagesV4 pre-
training [32] outperforms ImageNet pre-training when transferring to object
detection and semantic segmentation [34]. Shao  _et al_. showed similar
findings using the Objects365 dataset [49]. This finding indicates supervised
pre-training on a single classification task may not create representations
general enough for many kinds of downstream applications.

Self-supervised training is a popular method for representation learning
without supervised data [25, 5, 17, 18, 22, 53]. By forcing the
representations of an image to agree with each other under data augmentation
[1], SimCLR and MoCo trained representations useful for downstream
classification tasks [5, 18]. Grill  _et al_. proposed the use of online and
target neural networks for learning representations, which they evaluated on
classification tasks as well as semantic segmentation, object detection, and
depth estimation [17]. On the other hand, recent work has demonstrated the
limitations of current self-supervised learning methods [41]. They found that
aggressive cropping, commonly used in self-supervised learning (such as those
used in MoCo [18], PIRL [40], SimCLR [5] etc.), leads to representations that
are occlusion invariant, which can be effective for downstream classification
tasks. However, these representations are not necessarily invariant to other
symmetries of natural images (such as viewpoint invariance), which might be
necessary for other downstream tasks such as semantic segmentation [41].

Figure 3: The ResNet-FPN backbone architecture for multi-task learning.
Orange: the top-level features for classification. Cyan: multi-scale features
for box detection and instance segmentation. Green: the high resolution
features for pixel-wise tasks (_e.g_., segmentation, depth, and surface normal
estimation.)

##  3 Method

###  3.1 Specialized Teacher Models

We want to learn from a set of teachers that provide rich training signals
with their pseudo labels. We adopt four teacher models including four
important tasks in computer vision: classification, detection, segmentation,
and depth estimation. These tasks require visual understanding of objects and
3D geometry. Examples of the pseudo labels can be found in Figure 2. We train
the classification, detection, and segmentation teacher models from scratch on
medium/large scale datasets (_e.g_., ImageNet[47], Objects365 [49], COCO
[28]). For depth teacher model, we download the pre-trained checkpoint from
the open-source repository [44] 111https://github.com/intel-isl/MiDaS.

#### Pseudo labeling:

We transfer the knowledge in specialized teacher models to unlabeled or
partially labeled datasets by pseudo labeling. We follow the practice in [67]
to generate pseudo labels for detection and segmentation. For detection, we
use a hard score threshold of 0.5 to generate pseudo box labels. For
segmentation, we use a hard score threshold of 0.5 to generate semantic
segmentation masks whereas pixels with a smaller prediction score are set to
the _ignore_ label. For classification, we use soft labels, which contain the
probability distribution of all classes, because we find the performance is
better than hard labels. For depth, we simply use the predicted depth as
pseudo labels without further processing.

###  3.2 Multi-Task Student Model

#### Model architecture:

Our goal is to train the student with multiple tasks to learn general visual
representations. The first thing to design is a model architecture that can
share most of the parameters across tasks. We define three task categories:
(1) classification, (2) object detection, (3) pixel-wise prediction. The
pixel-wise prediction task includes semantic segmentation, depth estimation,
and surface normal prediction. Each category of task shares the same feature
representations in the backbone model.

We design the backbone model based on ResNet [21] and feature pyramid networks
(FPN) [36]. Figure 3 shows the overview of our architecture. We follow the
common practice to design the feature representations for classification and
detection tasks. We use C5 feature map (orange) for classification and {P3,
P4, P5, P6, P7} feature pyramid (cyan) for detection. We follow the practice
in [67] to fuse {P3, P4, P5, P6, P7} into P2 feature map (green) for pixel-
wise prediction. The fuse operation simply rescales all feature maps into
level 2 and sums them (which does not introduce any new parameters).

Each task category shares the same head architecture. The classification head
follows the ResNet design. It is a linear prediction layer followed by average
pooled C5 features. The object detection task follows the head architecture in
the Mask R-CNN [20]. We use 2 hidden convolution layers for RPN and 4 hidden
convolution layers and 1 fully connected layers for Fast R-CNN. The pixel-wise
prediction head has 3 convolution layers followed by the $C2$ features before
the final linear prediction layer. If the student model learns from multiple
tasks in the same task category (_e.g_., semantic segmentation and depth
prediction), each task owns its task specific head without sharing their
parameters.

#### Teacher-student training:

We want to study the effectiveness of learning from pseudo labels. Therefore,
we design the training of teacher and student models such that the main
differences between them are in the dataset and the labels. Unlike model
distillation [23] and noisy student [58], we use the same model capacity and
data augmentation in both the teacher and the student training. Despite that a
teacher can be trained with a more specialized architecture for its own task,
we train the teacher and student models using the same architecture shown in
Figure 3.

Training Datasets | Evaluation Datasets
---|---
Name | Task | Num Images | Name | Task | Num Images
ImageNet [47] | Classification | 1.2M |  CIFAR-100 [31] | Classification | 50k
Objects365 [49] | Detection | 600k |  Pascal [13] | Detection | 16.5k
COCO [37] | Segmentation | 118k |  Pascal [13] | Segmentation | 1.5k
MiDaS [44] | Depth | 1.9M |  NYU V2 [50] | Depth | 47k
JFT [51] | Classification | 300M |  ADE [66] | Segmentation | 20k
|  |  |  DIODE [54] | Surface Normal | 17k
Table 1: Datasets using for MuST and for downstream fine-tuning evaluation.

#### Learning From Multiple Teachers:

We propose Multi-Task Self-Training (MuST) to train a student model with
multiple teachers. Prior multi-task learning works, which harnessed the
information in multiple datasets, mainly focused on the scenario where each
example is only labeled with one task or a few tasks [9, 29]. In MuST, every
image has supervision for all tasks. The labels may come from supervised or
pseudo labels. For example, when training on ImageNet, we can use supervised
labels for classification and pseudo labels for detection, segmentation, and
depth.

Balancing the loss contribution for each task is an open research area in
multi-task learning [26, 6, 7, 63]. The loss of multi-task learning is the
weighted sum of the losses from all tasks $L=\sum_{i}w_{i}L_{i}$. The weight
$w_{i}$ decides the loss contribution for the task $i$. In ImageNet
experiments, we adopt $w_{i}=\frac{b_{s}lr_{it}}{b_{it}lr_{s}}$, where $b$
denotes the batch size, $lr$ denotes the learning rate, and the subscript
denotes student or teacher model. The equation is derived from the scaling
rule in [16], which scales the learning rate propotionally with batch size.
The only exception is the depth loss, of which we choose its weight by a
parameter sweep. In our experiments on JFT-300M, we use the algorithm in [26]
to learn $w_{i}$ for each task over the course of training.

#### Cross Dataset Training:

MuST has the flexibility to leverage both labeled and unlabeled data. It can
scale up the number of images by generating pseudo labels on the unlabeled
data. Or it can use images which are partially labeled with one or more tasks.
In our experiments, we show an example training across ImageNet, objects365,
and COCO datasets. We use supervised labels whenever they are available and
generate labels for all absent tasks using pseudo labels.

One challenge in cross dataset training is to balance the data coming from
datasets of different sizes. Instead of designing sampling heuristics [9], we
uniformly sample from the union of datasets. This works because every task is
labeled on every image in MuST, thus we do not need to worry about under/over-
sampling a task due to the imbalanced dataset size.

The second main difference compared to other self-training algorithm is that
the supervised and pseudo labels are treated equally. We do not batch the
examples of supervised and pseudo labels independently and assign them
different weights like in [67, 58]. The images are uniformly sampled from the
union of datasets and put into one mini-batch. Each example shares the same
weight on its loss regardless if the loss is computed against a supervised or
a pseudo label. This makes MuST significantly simpler to use and to scale with
multiple tasks.

###  3.3 Transfer Learning

To evaluate the representational quality of MuST and other baseline
representations, we fine-tune them on a suite of downstream computer visions
tasks. We adopt end-to-end fine-tuning instead of linear probe to the
performance of each fine-tuning task. We fine-tune on CIFAR-100
classification, Pascal detection, Pascal semantic segmentation, NYU depth, ADE
semantic segmentation and DIODE surface normal. Also note that all downstream
datasets are different than the ones the specialized teacher models are
trained on. Furthermore, surface normal prediction is a task that no
specialized teacher model was trained for, testing the robustness of
representations to held out tasks.

When fine-tuning a representation on a downstream task we sweep over the
learning rate and number of training steps (See Appendix for full details).
This allows for fair comparison between different representations.

##  4 Experiments

###  4.1 Experimental Settings

#### Training Datasets:

Table 1 provides an overview of the datasets we use in the experiments. We
experiment with four different datasets and tasks for training our supervised
teacher models. These supervised models will then be the ones to generate
pseudo labels on unlabeled/partially labeled images.

#### Evaluation Datasets:

Next we describe the datasets that all of our representations will be fine-
tuned on. Table 1 provides the list. We have different datasets with a total
of five different tasks. Note the Surface Normal task is never used as a
training task to test the task generality of the representations.

###  4.2 Learning with Multi-Task Self-Training

We run experiments to compare our MuST representation learning algorithm to
state-of-the-art self-supervised and supervised learning methods.

Settings | Transfer Learning Performance
---|---
Method | Epochs | CIFAR-100 | Pascal | Pascal | NYU | ADE | DIODE
|  | Classification | Detection | Segm. | Depth | Segm. | Normal
Self-supervised (SimCLR [5]) | 800 | 87.1 | 83.3 | 72.2 | 83.7 | 41.0 | 52.8
ImageNet Supervised | 90 | 85.4 | 79.3 | 70.6 | 81.0 | 39.8 | 48.9
\+ Multi-task Pseudo Labels | 90 |  86.3 (+0.9) |  85.1 (+5.8) |  80.6 (+10.0) |  87.8 (+6.8) |  43.5 (+3.7) |  52.7 (+3.8)
Table 2: Multi-Task Self-Training (MuST) outperforms supervised and self-
supervised representations on ImageNet. We compare MuST to state-of-the-art
self-supervised and supervised learning using the same pre-training dataset
(ImageNet). MuST learns more general features and achieves the best
performance on $4/6$ downstream fine-tuning tasks. The performance differences
show the impact of different training objectives.

#### MuST Improves Pre-training on ImageNet:

Table 2 compares the MuST algorithm to self-supervised and supervised learning
on ImageNet. On a suite of 6 downstream tasks MuST representations improves
over state-of-the-art self-supervised learning and supervised learning on $4$
and $5$ tasks, respectively. MuST makes use of not only the ImageNet
classification labels, but also pseudo labels generated from supervised models
trained on Objects365 detection, COCO semantic segmentation, and MiDas depth.
This additional information being trained on for ImageNet images leads to much
more generalizable feature representations. We observe that self-supervised
and supervised pre-training on ImageNet does not learn features that
generalize nearly as well to tasks other than image classification.

Settings | Transfer Learning Performance
---|---
Method | CIFAR-100 | Pascal | Pascal | NYU | ADE | DIODE
| Classification | Detection | Segm. | Depth | Segm. | Normal
ImageNet Supervised | 85.4 | 79.3 | 70.6 | 81.0 | 39.8 | 48.9
\+ Depth Pseudo Labels |  84.4(-1.0) |  79.3(+0.0) |  71.0(+0.4) |  86.0(+5.0) |  39.5(-0.3) |  51.3(+2.4)
\+ Depth / Segm. Pseudo Labels |  85.3(-0.1) |  81.6(+2.3) |  78.6(+8.0) |  87.2(+6.2) |  41.5(+1.7) |  52.4(+3.5)
\+ Depth / Segm. / Detection Pseudo Labels | 86.3(+0.9) | 85.1(+5.8) | 80.6(+10.0) | 87.8(+6.8) | 43.5(+3.7) | 52.7(+3.8)
Table 3: Multi-Task Self-Training (MuST) benefits from increasing the number
of different pseudo label tasks. We add depth, segmentation, and detection
pseudo labels in addition to supervised ImageNet classification labels and
test the representational quality. The results reveal that adding pseudo
labels from more tasks leads to more general pre-trained models. All models
are trained for 90 epochs on ImageNet.

#### MuST Improves With More Tasks/Datasets For Learning General Features:

The MuST algorithm makes use of pseudo labels generated from independent
supervised models trained on different datasets. We next study the importance
of having pseudo labels being generated from multiple different teacher models
trained on different tasks/datasets. Table 3 shows the representational
quality improvement starting from using only supervised ImageNet labels and
then adding three different types of pseudo labels obtained from three
different datasets. As we continue to add pseudo labels from different
tasks/datasets our representations improve in quality. For each new task added
we obtain strong improvement across all 6 of our downstream tasks.

#### Pseudo Labels Preserve Transfer Learning Performance of Teacher Model:

We next study how effectively pseudo labels preserve the transfer learning
performance of teacher models trained on supervised datasets. To test this we
train two supervised teacher models: object detection model on Objects365 and
semantic segmentation model on COCO. The first two rows in Table 4 shows their
supervised learning performance and their transfer learning performance on 6
downstream tasks. Next we generate pseudo labels on two datasets without
labels: ImageNet (1.2M images) and JFT (300M images) . Now we train models
from scratch on the pseudo labels on ImageNet and JFT. The next 4 rows in
Table 4 reveal these results. We observe for both object detection and
segmentation pseudo labels we obtain a degradation in the supervised learning
quality (e.g. 26.1 vs 20.6/20.7), but that when the representations are
_transferred_ they obtain similar or better transfer learning performance than
the teacher model. Furthermore, the representations obtained by training on
JFT over ImageNet typically lead to better transfer learning performance,
which reveals the scalability of the MuST method. As we get more and more
unlabeled data, our method can easily take advantage of it and the
representational quality improves.

Settings | Performance | Transfer Learning Performance
---|---|---
Task | Train Dataset | Obj365 | COCO | CIFAR-100 | Pascal | Pascal | NYU | ADE | DIODE
|  | Detection | Segm. | Classification | Detection | Segm. | Depth | Segm. | Normal
|  | Teacher Model |  |  |  |  |  |
Detection | Objects365 | 26.1 | — | 84.0 | 87.6 | 78.8 | 90.1 | 46.0 | 55.6
gray!15 Segmentation | COCO | — | 53.8 | 80.8 | 82.2 | 80.2 | 86.6 | 42.8 | 51.0
|  | Student Model |  |  |  |  |  |
Detection | ImageNet | 20.6 | — | 83.2 | 86.0 | 78.5 | 88.5 | 44.7 | 55.2
Detection | JFT | 20.7 | — | 85.2 | 87.7 | 79.5 | 89.6 | 45.4 | 55.0
gray!15 Segmentation | ImageNet | — | 55.5 | 82.3 | 80.5 | 79.2 | 86.3 | 41.8 | 51.2
gray!15 Segmentation | JFT | — | 49.0 | 83.1 | 82.8 | 78.2 | 86.6 | 41.9 | 51.6
Table 4: Models trained on supervised data or pseudo labeled data have similar
transfer learning performance. Results comparing how representations transfer
if they are trained on supervised data or on pseudo labels that are generated
by the supervised model. Pseudo labels effectively compress the knowledge in a
supervised dataset. The performance of student models increases with the size
of the unlabeled dataset. As the unlabeled dataset size increased, the
performance of student model increases. This reveals the scalability of MuST.
All student models are trained for the same training iterations (90 ImageNet
epochs and 0.36 JFT epochs).

#### Multi-Task Self-Training Across Datasets:

MuST utilized pseudo labels generated from teacher models trained on different
supervised learning datasets. A natural comparison is then to see how MuST
compared against supervised multi-task supervised training where a model is
trained on the union of the datasets and labels [29]. Table 5 compares the
representational quality of MuST versus supervised multi-task training on
three datasets: ImageNet, COCO and Objects365. For multi-task training we
sample examples from the datasets with equal probability. Sampling examples
with probabilities proportional to the size of the datasets does not work
well. Because ImageNet and Objects365 datasets are much larger than COCO
dataset and as a result for a batch size of 256 only 15 examples have non zero
loss values for segmentation. On the other hand, for MuST _every image has any
type of label_ and we can sample examples with probabilities proportional to
the size of datasets. When comparing the representation qualities MuST obtains
the best performance on $6/6$ downstream tasks.

Settings | Transfer Learning Performance
---|---
Method | CIFAR-100 | Pascal | Pascal | NYU | ADE | DIODE
| Classification | Detection | Segm. | Depth | Segm. | Normal
Supervised Multi-Task | 85.3 | 85.1 | 82.1 | 87.6 | 43.9 | 53.4
Supervised Multi-Task + Pseudo Labels |  86.3 (+1.1) |  86.2 (+1.1) |  82.3 (+0.2) |  88.2 (+0.6) |  45.4 (+1.5) |  54.7 (+1.3)
Table 5: Comparing Multi-Task Training versus Multi-Task Self-Training. We compare MuST against a baseline of doing supervised multi-task training on the union of all teacher datasets. We use three datasets: ImageNet, COCO and Objects365. Supervised model is jointly trained on the supervised labels of these three datasets. MuST trains jointly on all three supervised and pseudo labels generated by the teacher models. The transfer learning performance gets strong improvements by incorporating pseudo labels into every image. Settings | Transfer Learning Performance
---|---
Method | Epochs | CIFAR-100 | Pascal | Pascal | NYU | ADE | DIODE
|  | Classification | Detection | Segm. | Depth | Segm. | Normal
Self-Supervised with JFT images (SimCLR [5]) | 1 | 85.6 | 82.4 | 71.0 | 83.7 | 41.4 | 54.4
Self-Supervised with JFT images (SimCLR [5]) | 2 | 85.8 | 83.7 | 73.3 | 84.3 | 42.2 | 55.3
Self-Supervised with JFT images (SimCLR [5]) | 5 | 86.1 | 84.1 | 74.9 | 84.8 | 43.0 | 56.0
JFT supervised | 3 | 87.7 | 84.6 | 78.2 | 86.0 | 43.4 | 50.7
JFT supervised | 5 | 88.6 | 84.9 | 79.7 | 86.1 | 44.3 | 51.1
JFT supervised | 10 | 89.6 | 85.2 | 80.4 | 86.5 | 45.7 | 53.1
Multi-Task Pseudo Labels | 2.5 | 87.6 | 87.8 | 82.2 | 89.8 | 47.0 | 56.2
JFT supervised + Multi-Task Pseudo Labels | 2 |  88.3(+0.5) | 87.9(+0.1) | 82.9(+0.7) |  89.5(-0.3) | 47.2(+0.2) | 56.4(+0.2)
Table 6: Scaling Multi-Task Self-Training to 300M images. We repeat the
experiments in Table 2 on the JFT dataset (300M images with classification
labels). The supervised learning benefits more from the additional images and
annotations compared to the self-supervised SimCLR algorithm.

###  4.3 Scaling Multi-Task Self-Training

One benefit of MuST is that it can scale to unbounded amounts of unlabeled
images. To test this hypothesis we move from the ImageNet setup with 1.2M
images to JFT with 300M images.

#### Scaling Dataset Size and Training Iterations:

Now instead of generating pseudo labels on 1.2M images, we scale the MuST
training to have all three supervised teacher models to generate pseudo labels
on 300M images. This process is trivially parallelizable, which makes the
overall runtime low compared to training of the models. Table 6 shows the
comparison of MuST vs self-supervised learning and supervised learning on the
JFT dataset. On $5/6$ downstream tasks MuST outperforms the self-supervised
SimCLR algorithm when using the same unlabeled data. We also train a
supervised baseline on the multi-class labels available on JFT and find that
MuST, using only the unlabeled images, outperforms the representation on $5/6$
downstream tasks. This is quite impressive considering the total sum of
supervised images that MuST indirectly makes use of from the pseudo labels is
only about 3.7M images compared to the 300M labeled JFT images. Adding JFT
supervised labels can further improve the performance on image classification
and segmentation, showing the flexibility of MuST in using labeled and
unlabeled data. Lastly, the student model not only learns general features for
transferring, it is also capable of generating high quality predictions for
multiple tasks. Figure 4 shows the predictions made by our strongest model.

|  |
---|---|---
Figure 4: Visualization of the predictions generated by a multi-task student
model. The MuST student model not only learns general feature representations,
but also makes high quality visual predictions with a single model.

#### Bootstrapping from Pre-trained Models.

Next we study if MuST can improve upon checkpoints trained with billions of
training examples. We use ALIGN checkpoints [24], which are trained with 1.8B
image-text pairs, to initialize parameters for training both teacher models
and the student model. We use the same teacher model tasks as our previous
experiments. The pseudo labels are generated on JFT-300M dataset and the MuST
student model is trained on JFT for 1 epoch. Figure 5 shows relative transfer
learning performance gains of Noisy Student [58], ALIGN [24], and MuST w/
ALIGN compared to the ImageNet checkpoint [52]. The figure shows MuST w/ ALIGN
improves the ALIGN checkpoint by respectable margins for 4 out of the 6
downstream tasks. The performances are slightly worse for CIFAR-100 and DIODE
surface normal prediction. We repeat the experiments with EfficientNet-L2
architecture and train the student model for 0.36 epoch on JFT. We report 4
downstream tasks showing improvements over the ALIGN checkpoint in Table 7. We
find the student model trained with MuST improves the large ALIGN
EfficientNet-L2 checkpoint and is competitive to the state-of-the-art models
specialized for each dataset and task. Notably, MuST provides checkpoints
ready to be fine-tuned for short iterations to achieve state-of-the-art
performance while typical self-training methods [67] require pseudo labeling
and long training iterations for each downstream task.

Figure 5: Relative transfer learning performance gains over the ImageNet pre-trained model [52]. Checkpoints trained with more data or labels typically provide gains on transfer learning to downstream tasks. Fine-tuning the EfficientNet-B7 ALIGN checkpoint with MuST can further improve transfer learning performance for 4/6 downstream tasks. Settings | Transfer Learning Performance
---|---
Method | Pascal | Pascal | NYU | ADE
| Detection | Segm. | Depth | Segm.
Previous SoTA |  89.3 [14] |  90.0 [67] |  90.4 [43] |  54.1 [8]
ALIGN [24] | 86.2 | 86.6 | 91.1 | 54.0
MuST w/ [24] | 88.2 | 89.8 | 91.9 | 54.3
Table 7: MuST checkpoints are versatile and achieve competitive performance
compared to state-of-the-art models. MuST improves the transfer learning
performance of the ALIGN EfficientNet-L2 checkpoint on these four downstream
tasks.

##  5 Discussion

#### Which pre-training method performs the best with large scaling training?

Although self-supervised learning can outperform supervised learning on the
ImageNet size dataset (1.3 million images/1k classes), supervised learning is
still a better pre-training method on JFT size dataset (300 million images/18k
classes). The gap may be compensated by training with more unlabeled data for
self-supervised learning. However, self-training can also expand one or
multiple supervised models by generating pseudo labels on unlabeled data.
Overall, both self-supervised and self-training are able to scale, but at the
moment self-training presents better performance in learning general features.
A promising direction is to combine self-supervised and self-training for
representation learning [65, 12, 59].

#### Why use MuST over self-supervised learning?

Both of the methods are scalable with the unlabeled training data, however,
MuST can easily combine together all labeled and unlabeled data. However,
self-supervised learning relies on the generalization from the pre-text task
to downstream tasks, which does not always give good performance. It is easier
to design pseudo labels if the downstream tasks of interest are known in
advance. MuST also generalizes to unseen tasks (_e.g_., surface normal
prediction) given the tasks of the teacher model in this paper.

##  6 Conclusion

In this paper, we present MuST, a scalable multi-task self-training method for
learning general representations. We compare with supervised and self-
supervised learning approaches on ImageNet and JFT and evaluate on 6 datasets
including visual recognition, localization, and 3D geometry prediction. We
show that MuST outperforms or is on par with supervised and self-supervised
learning on 5 out of 6 transfer learning tasks, except the classification
task. Moreover, MuST can improve upon already strong checkpoints trained with
billions of examples. The results show multi-task self-training is a scalable
pre-training method and is able to learn general feature representations. We
hope this work will encourage further research towards creating universal
visual representations.

## Acknowledgements

We would like to thank Yin Cui, Aravind Srinivas, Simon Kornblith, and Ting
Chen for valuable feedback.

##  Appendix A Details of Training and Evaluation Datasets

###  A.1 Training Datasets

In this section, we describe 5 datasets we use to train teacher models.

#### ImageNet:

ImageNet [47] is a classification dataset with 1.2M training images and 1000
unique classes. All of its images are center cropped and have one primary
object per image.

#### Objects365:

Objects365 [49] is an object detection dataset that has 365 different classes
and 600k training images.

#### COCO:

The COCO dataset [37] contains 118k images that has a variety of different
labels (e.g. object detection, instance segmentation, panoptic segmentation).
For all experiments we use its pantopic segmentation labels.

#### MiDaS:

The MiDaS depth model [44] that is used for generating our depth pseudo labels
is trained on a diverse set of 5 depth datasets. The 5 depth datasets are DIML
Indoor [27] (220k images), MegaDepth [35] (130k images), ReDWeb [56] (3600),
WSVD [55] (1.5M), and 3D movies (75k). The model is trained to be invariant to
the depth range and scale across all datasets, leading to a model that
generates robust pseudo labels.

#### JFT:

JFT [51] is a large-scale image multi-label classification dataset with 300M
labeled images. This dataset is used to test the scale of MuST and various
self-supervised learning algorithms.

###  A.2 Evaluation Datasets

Next we describe the datasets that all of our representations will be fine-
tuned on. We have different datasets with a total of five different tasks.
Note the Surface Normal task is never used as a training task to test the task
generality of the representations.

#### CIFAR-100:

CIFAR-100 is a classification dataset with 50k images and 100 unique classes.

#### PASCAL Detection:

The Pascal Detection dataset [13] is an object detection dataset with 20
unique classes. We train the models on the trainval sets of PASCAL VOC 2007
and PASCAL VOC 2012 which include 16.5k images.

#### PASCAL Segmentation:

The Pascal Segmentation dataset [13] is a semantic segmentation dataset with
20 unique classes. We train the models on the train set of the PASCAL VOC 2012
segmentation dataset which has 1.5k images.

#### NYU Depth V2:

The NYU Depth v2 dataset [50] is a depth estimation dataset that contains
47584 train images and 654 validation images.

#### ADE Segmentation:

ADE20k [66] is a segmentation dataset that contains 20k images with 150 object
and stuff classes. The dataset contains a wide variety of different indoor and
outdoor scenes along with object classes.

#### DIODE Surface Normal:

The DIODE dataset [54] is a depth and surface normal dataset that contains
16884 images. The dataset contains a diverse set of both indoor and outdoor
scenes for training and testing. We only make use of the surface normal
labels.

##  Appendix B Implementation Details

###  B.1 Training Teacher Models

In this section, we introduce the details of training teacher models, which
are used to generate pseudo labels in MuST. All the models are trained with a
ResNet-152 backbone model.

#### Objects365 Detection:

We use batch size 256 and train for 140 epochs. The image size is $640$. We
apply scale jittering [0.5, 2.0] (i.e. randomly resample image between
$320\times 320$ to $1280\times 1280$ and crop it to $512\times 512$). The
learning rate is 0.32 and the weight decay is set as 4e-5. The model is
trained from random initialization. The final performance is 26.1 AP.

#### COCO Segmentation:

We use the annotations in COCO panoptic segmentation dataset [28]. We train a
semantic segmentation model that only predicts the semantic class for each
pixel, instead of also predicting the object instance. We use batch size 256
and train for 384 epochs. The image size is $896$. We apply scale jittering
[0.5, 2.0]. The learning rate is 0.32 and the weight decay is set as 4e-5. The
model is trained from random initialization. The final performance is 53.8
mIoU.

#### MiDaS Depth:

We directly download the pre-trained MiDaS from the github repository and use
it as a teacher model to generate pseudo labels.

#### ImageNet Classification:

We use batch size 2048 and train for 400 epochs. The image size is $224$. The
learning rate is 0.8 and weight decay is 4e-5. We apply random augmentation
[10] (2L-15M, 2 layers with magnitude 15) and label smoothing (0.1) to
regularize the model training. The final performance is 81.6 top-1 accuracy.

###  B.2 Training Multi-Task Student Models

We use a batch size 256 for training student models in our experiments. The
image size is $640$. We apply scale jittering [0.5, 2.0] during training. The
weight decay is 4e-5 in ImageNet experiments and 3e-6 in JFT experiments. No
random augmentation [10] or label smoothing is applied.

###  B.3 Fine-tuning on Evaluation Datasets

For fine-tuning we initialize the parameters in the ResNet and FPN backbone
with a pre-trained model and randomly initialize the rest of the layers. We
perform end-to-end fine-tuning with an extensive grid search of the
combinations of learning rate and training steps to ensure each pre-trained
model achieves its best fine-tuning performance. We experiment with different
weight decays but do not find it making a big difference and set it to 1e-4.
All models are trained with cosine learning rate for simplicity. Below we
describe the dataset, evaluation metric, model architecture, and training
parameters for each task.

#### CIFAR-100:

We use standard CIFAR-100 train and test sets and report the top-1 accuracy.
We resize the image resolution to $256\times 256$. We replace the
classification head in the pre-trained model with a randomly initialized
linear layer that predicts 101 classes, including background. We use a batch
size of 512 and search the combination of training steps from 5000 to 20000
and learning rates from 0.005 to 0.32. We find the best learning rate for
SimCLR (0.16) is much higher than the supervised model (0.005). This trend
holds for the following tasks.

#### PASCAL Segmentation:

We use PASCAL VOC 2012 train and validation sets and report the mIoU metric.
The training images are re-sampled into $512\times 512$ with scale jittering
[0.5, 2.0]. We initialize the model from the pre-trained backbone and FPN [36]
layers. We remove the pre-trained segmentation head and train from a randomly
initialized head. We use a batch size of 64 and search the combination of
training steps from 5000 to 20000 and learning rates from 0.005 to 0.32.

#### PASCAL Detection:

We use PASCAL VOC 2007+2012 trainval set and VOC 2007 test set and report the
$AP_{50}$ with 11 recall points to compute average precision. The training
images are resampled into $896$ with scale jittering [0.5, 2.0]. we initialize
the model from the pre-trained backbone and FPN [36] layers and randomly
initialize the heads. We use a batch size of 32 and search the combination of
training steps from 5000 to 20000 and learning rates from 0.005 to 0.32.

#### NYU Depth:

We use NYU depth v2 dataset with 47584 train and 654 validation images. We
report the percentage of predicted depth values within 1.25 relative ratio
compared to the ground truth. The training images are resampled into $640$
with scale jittering [0.5, 2.0]. we initialize the model from the pre-trained
backbone and FPN [36] layers and randomly initialize the heads. We use a batch
size of 64 and search the combination of training steps from 10000 to 40000
and learning rates from 0.005 to 0.32.

#### DIODE:

We use DIODE outdoor dataset with 16884 train and 446 validation images. We
report the percentage of the angle error less than 11.25°. We use the original
image resolution $768$ for training and evaluation. The training image is
applied with scale jittering [0.5, 2.0]. we initialize the model from the pre-
trained backbone and FPN [36] layers and randomly initialize the heads. We use
a batch size of 32 and search the combination of training steps from 20000 to
80000 and learning rates from 0.01 to 0.16.

##  Appendix C Visualization of Student Model Predictions

Figure 6 shows more visual examples of the predictions made by a single multi-
task student model. The images are sampled from the validation set in ImageNet
dataset.

|  |
---|---|---
|  |
|  |
|  |
|  |
Figure 6: The visualization of inference on ImageNet dataset made by single
MuST student model.

## References

  * [1] Suzanna Becker and Geoffrey E Hinton.  Self-organizing neural network that discovers surfaces in random-dot stereograms.  Nature, 355(6356):161–163, 1992.
  * [2] Rich Caruana.  Multitask learning.  Machine learning, 28(1):41–75, 1997.
  * [3] Liang-Chieh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D Collins, Ekin D Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shlens.  Naive-student: Leveraging semi-supervised learning in video sequences for urban scene segmentation.  In ECCV, 2020.
  * [4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [6] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich.  GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks.  JMLR, 2018.
  * [7] Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov.  Just pick a sign: Optimizing deep multitask models with gradient sign dropout.  In NeurIPS, 2020.
  * [8] Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov.  Per-pixel classification is not all you need for semantic segmentation.  CoRR, abs/2107.06278, 2021.
  * [9] Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoc V. Le.  Bam! born-again multi-task networks for natural language understanding.  In ACL, 2019.
  * [10] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le.  Randaugment: Practical automated data augmentation with a reduced search space, 2019.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
  * [12] Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau.  Self-training improves pre-training for natural language understanding, 2020.
  * [13] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  IJCV, 2010.
  * [14] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D. Cubuk, Quoc V. Le, and Barret Zoph.  Simple copy-paste is a strong data augmentation method for instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2918–2928, June 2021.
  * [15] Ross Girshick.  Fast r-cnn.  In ICCV, 2015.
  * [16] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch SGD: training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [17] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent: A new approach to self-supervised learning.  In NeurIPS, 2020.
  * [18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [19] Kaiming He, Ross Girshick, and Piotr Dollár.  Rethinking imagenet pre-training.  In ICCV, 2019.
  * [20] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [22] Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord.  Data-efficient image recognition with contrastive predictive coding.  arXiv preprint arXiv:1905.09272, 2019.
  * [23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [24] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  CoRR, abs/2102.05918, 2021.
  * [25] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [26] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [27] Youngjung Kim, Hyungjoo Jung, Dongbo Min, and Kwanghoon Sohn.  Deep monocular depth estimation via integration of global and local predictions.  IEEE transactions on Image Processing, 27(8):4131–4144, 2018.
  * [28] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar.  Panoptic segmentation.  In CVPR, June 2019.
  * [29] Iasonas Kokkinos.  Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory.  In CVPR, 2017.
  * [30] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.  Big transfer (bit): General visual representation learning.  In ECCV, 2020.
  * [31] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * [32] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  IJCV, 2020.
  * [33] Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In Workshop on challenges in representation learning, ICML, 2013\.
  * [34] Hengduo Li, Bharat Singh, Mahyar Najibi, Zuxuan Wu, and Larry S. Davis.  An analysis of pre-training on object detection.  CoRR, abs/1904.05871, 2019.
  * [35] Zhengqi Li and Noah Snavely.  Megadepth: Learning single-view depth prediction from internet photos.  In CVPR, 2018.
  * [36] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [38] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.  Improving multi-task deep neural networks via knowledge distillation for natural language understanding.  In ACL, 2019.
  * [39] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten.  Exploring the limits of weakly supervised pretraining.  In ECCV, September 2018.
  * [40] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In CVPR, 2020.
  * [41] Senthil Purushwalkam and Abhinav Gupta.  Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases.  arXiv preprint arXiv:2007.13916, 2020.
  * [42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  JMLR, 2020.
  * [43] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.  Vision transformers for dense prediction.  ArXiv preprint, 2021.
  * [44] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020.
  * [45] Chuck Rosenberg, Martial Hebert, and Henry Schneiderman.  Semi-supervised self-training of object detection models.  In IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05), 2005.
  * [46] Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  arXiv preprint arXiv:1706.05098, 2017.
  * [47] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  IJCV, 2015.
  * [48] H Scudder.  Probability of error of some adaptive pattern-recognition machines.  IEEE Transactions on Information Theory, 11(3):363–371, 1965.
  * [49] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In ICCV, 2019.
  * [50] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In ECCV, 2012.
  * [51] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta.  Revisiting unreasonable effectiveness of data in deep learning era.  In ICCV, 2017.
  * [52] Mingxing Tan and Quoc Le.  EfficientNet: Rethinking model scaling for convolutional neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 09–15 Jun 2019.
  * [53] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In ECCV, 2019.
  * [54] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al.  Diode: A dense indoor and outdoor depth dataset.  arXiv preprint arXiv:1908.00463, 2019.
  * [55] Chaoyang Wang, Simon Lucey, Federico Perazzi, and Oliver Wang.  Web stereo video supervision for depth prediction from dynamic scenes.  In 2019 International Conference on 3D Vision (3DV), pages 348–357. IEEE, 2019.
  * [56] Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo.  Monocular relative depth perception with web stereo data supervision.  In CVPR, 2018.
  * [57] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In European Conference on Computer Vision. Springer, 2018.
  * [58] Qizhe Xie, Eduard Hovy, Minh-Thang Luong, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In CVPR, 2020.
  * [59] Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli.  Self-training and pre-training are complementary for speech recognition, 2020.
  * [60] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  In NAACL, 2020.
  * [61] I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan.  Billion-scale semi-supervised learning for image classification.  CoRR, abs/1905.00546, 2019.
  * [62] David Yarowsky.  Unsupervised word sense disambiguation rivaling supervised methods.  In ACL, 1995.
  * [63] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning.  arXiv preprint arXiv:2001.06782, 2020.
  * [64] Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In CVPR, 2018.
  * [65] Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, and Yonghui Wu.  Pushing the limits of semi-supervised learning for automatic speech recognition, 2020.
  * [66] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Semantic understanding of scenes through the ade20k dataset.  IJCV, 127(3):302–321, 2019.
  * [67] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, and Quoc V. Le.  Rethinking pre-training and self-training.  In NeurIPS, 2020.


# Automatic Construction of Discourse Corpora for Dialogue Translation

###### Abstract

In this paper, a novel approach is proposed to automatically construct
parallel discourse corpus for dialogue machine translation. Firstly, the
parallel subtitle data and its corresponding monolingual movie script data are
crawled and collected from Internet. Then tags such as speaker and discourse
boundary from the script data are projected to its subtitle data via an
information retrieval approach in order to map monolingual discourse to
bilingual texts. We not only evaluate the mapping results, but also integrate
speaker information into the translation. Experiments show our proposed method
can achieve 81.79% and 98.64% accuracy on speaker and dialogue boundary
annotation, and speaker-based language model adaptation can obtain around 0.5
BLEU points improvement in translation qualities. Finally, we publicly release
around 100K parallel discourse data with manual speaker and dialogue boundary
annotation.

Keywords: Discourse Corpus, Dialogue, Machine Translation, Information
Retrieval, Movie Script, Movie Subtitle

languageresourceLanguage Resources

Automatic Construction of Discourse Corpora for Dialogue Translation

Longyue Wang∗, Xiaojun Zhang∗, Zhaopeng Tu†, Andy Way∗, Qun Liu∗
---
∗ ADAPT Centre, School of Computing, Dublin City University, Ireland
† Noah Ark Lab, Huawei Technologies, China
{lwang, xzhang, away<EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

##  1\. Introduction

Dialogue is an essential component of social behaviour to express human
emotions, moods, attitudes and personality. To date, few researchers have
investigated how to improve the machine translation (MT) of conversational
material by exploiting their internal structure. This lack of research on the
dialogue MT is a surprising fact, since dialogue exhibits more cohesiveness
than single sentence and at least as much than textual discourse.

Although there are a number of papers on corpus construction for various
natural language processing (NLP) tasks, dialogue corpora are still scarce for
MT. Some work regarding bilingual subtitles as parallel corpora exists, but it
lacks rich information between utterances (sentence-level corpus) [Lavecchia
et al., 2007, Tiedemann, 2007a, Tiedemann, 2007b, Itamar and Itai, 2008,
Tiedemann, 2008, Xiao and Wang, 2009, Tiedemann, 2012, Zhang et al., 2014].
Other work focuses on mining the internal structure in dialogue data from
movie scripts. However, these are monolingual data which cannot used for MT
[Danescu-Niculescu-Mizil and Lee, 2011, Banchs, 2012, Walker et al., 2012,
Schmitt et al., 2012]. In general, the fact is that bilingual subtitles are
ideal resources to extract parallel sentence-level utterances, and movie
scripts contain rich information such as dialogue boundaries and speaker tags.

Inspired by these facts, our initial idea was to build dialogue discourse
corpus by bridging the information in these two kinds of resources (i.e.,
scripts and subtitles). The corpus should be parallel, align at the segment-
level as well as contain rich dialogue information. We propose a simple but
effective approach to build our dialogue corpus. Firstly, we extract parallel
sentences from bilingual subtitles, and mine dialogue information from
monolingual movie scripts. Secondly, we project dialogue information from
script utterances to its corresponding parallel subtitle sentences using an
information retrieval (IR) approach. Finally, we apply this approach to build
a Chinese–English dialogue corpus, and also manually annotate dialogue
boundaries and speaker tags based on automatic results.

To validate the effect of the proposed approach, we carried out experiments on
the generated corpus. Experimental results show that the automatic annotation
approach can achieve around 82% and 98% on speaker and dialogue boundaries
annotation, respectively. Furthermore, we explore the integration of speaker
information into MT via domain-adaptation techniques. Results show that we can
improve translation performance by around 0.5 BLEU points compared to baseline
system.

Generally, the contributions of this paper include the following:

  * •

We propose an automatic method to build a segment-level dialogue parallel
corpus with useful information, for building large-scale dialogue MT systems;

  * •

Through exploring dialogue information with MT, we show that speaker
information is really helpful to dialogue MT systems;

  * •

We also manually annotate about 100K sentences from our dialogue corpus. The
gold standard dataset111We release our DCU English-Chinese Dialogue Corpus in
http://computing.dcu.ie/~lwang/resource.html. can be further used to search
for the coherence and consistency clues in discourse structure to implement a
dialogue MT system.

The rest of the paper is organized as follows. In Section 2, we describe
related work. Section 3 describes in detail our approaches to build a dialogue
corpus as well as the structure of the generated database. The experimental
results for both corpus annotation and translation are reported in Section 4.
Finally, Section 5 presents our conclusions and future work plans.

##  2\. Related Work

In the specific case of dialogue MT system, data acquisition can impose
challenges including data scarcity, translation quality and scalability. The
release of the Penn Discourse Treebank (PDTB)222Available at
https://www.seas.upenn.edu/p̃dtb. [Prasad et al., 2008] helped bring about a
new sense of maturity in discourse analysis, finally providing a high-quality
large-scale resource for training discourse parsers for English. Based on
PDTB, some have applied the insights to MT [Meyer and Popescu-Belis, 2012]. A
resource like the PDTB is extremely valuable, and it would be desirable to
have a similar resource in dialogue or conversation as well.

There are two directions of work related to dialogue corpus construction. One
is parallel corpora construction for dialogue or conversation MT [Lavecchia et
al., 2007, Tiedemann, 2007a, Tiedemann, 2007b, Tiedemann, 2008, Itamar and
Itai, 2008, Xiao and Wang, 2009, Tiedemann, 2012]. Thanks to the effects of
crowdsourcing and fan translation in audiovisual translation [O’Hagan, 2012],
we can regard subtitles as parallel corpora. ?) leveraged the existence of
bilingual subtitles as a source of parallel data for the Chinese-English
language pair to improve the MT systems in the movie domain. However, their
work only considers sentence-level data instead of extracting more useful
information for dialogues. Besides, Japanese researchers constructed a speech
dialogue corpus for a machine interpretation system [Aizawa et al., 2000,
Matsubara et al., 2002, Ryu et al., 2003, Takezawa, 2003]. They collected
speech dialogue corpora for machine interpretation research via recording and
transcribing Japanese/English interpreters’ consecutive/simultaneous
interpreting in the booth. The German VERBMOBIL speech-to-speech translation
programme [Wahlster, 2013] also collected and transcribed task-oriented
dialogue data. This related work focused on speech-to-speech translation
including three modules of automatic speech recognition (ASR), MT and text-to-
speech(TTS).

The other one is mining rich information from other resources such as movie
scripts. ?) created a conversation corpus containing large metadata-rich
collections of fictional conversations extracted from raw movie scripts. Both
?) and CMU released dialogue corpora extracted from the Internet Movie Script
Database (IMSDb).333Available at http://www.imsdb.com. Based on IMSDb, ?)
annotated 862 film scripts to learn and characterize the character style for
an interactive story system, and ?) annotated 347 dialogues to explore a
spoken dialogue system. The resource of movie scripts, such as IMSDb, is good
enough to generate conversational discourse for dialogue processing. However,
monolingual movie scripts are not enough for MT which needs a large-scale
bilingual dialogue corpus to train and tune translation models.

##  3\. Building A Parallel Dialogue Corpus

As already stated, our presented parallel dialogue corpus is extracted from
bilingual movie/episode subtitles and monolingual scripts. We extend previous
work on movie scripts to scripts of TV series such as Friends. From IMSDb and
SimplyScripts444Available at http://www.simplyscripts.com. and the like, we
crawled movie/episode scripts data. In addition, we collected the English-
Chinese bilingual subtitles from multiple audiovisual translation web
resources such as Shooter555Available at http://sub.makedie.me. and
Opensubtitles. 666Available at http://www.opensubtitles.org. Based on the
hypothesis that both a script and a subtitle exist for the same movie or
episode, the method can be described in a pipeline as follows:

  * (1)

given a monolingual movie/episode script, we identify dialogue boundaries and
speaker tags using clues such as format and story structure tags in the
script;

  * (2)

for a bilingual subtitle, we align each sentence with its translation using
clues such as format and time information;

  * (3)

for each utterance in a processed script, we apply IR techniques to match it
with the line(s) in its corresponding processed subtitle according to the
shared language;

  * (4)

for each matched term, we map the useful annotations such as speaker and
dialogue boundaries from the script side to the matched line(s) in its
subtitle side.

###  3.1. Script and Subtitle

Figure 1 depicts a browser snapshot illustrating an episode script layout of
Friends. There are three kinds of information: speaker, shot/scene and action
information in the script. The speaker element (red ellipses) contains the
corresponding character who says the utterance(s). The shot/scene tags (e.g.,
“SCENE”, “SHOT”, “CUT INTO:” and “CUT TO:” etc.) can be regarded as the
boundaries of dialogues. For instance, the tags “SCENE J” and “CUT TO:” refer
to the beginning and end of a dialogue, respectively. The action (green
frames) contains all additional information of a narrative nature and explains
what is happening in the scene. In this work, we focus on the first two
information type while ignoring final one.

Figure 1: Examples of scripts of Friends in English

Figure 2 is the corresponding bilingual subtitle of the script in Figure 1.
Subtitles are often organized in two formats: Advanced SubStation Alpha (ASS)
and SubRip Text (SRT). As most lines are one-to-one aligned on two language
sides, it easy to process them into a parallel corpus. We also use line id and
time line information to deal with one-to-many or mismatching cases.

Figure 2: Examples of bilingual subtitles (SRT) of Friends in English and
Chinese

Based on the above rules, we extract useful information from both scripts and
subtitles. In order to obtain high-quality data, we also apply a series of
techniques including language detection, simplified-traditional Chinese
conversation, coding conversation and punctuation normalization. After
processing scripts and subtitles, the next step is to match and project terms
from script side to subtitle side.

###  3.2. Matching and Projection

Comparing examples in Figures 1 and 2, we found that the script and the
subtitle share the same language (i.e., English). However, subtitle lines are
not always the same as the utterances in a script for the actors may change
their lines on site, either slightly or to a greater extent. For example, the
first utterance in the script is Later, when Monica’s around, I want you to
ask me about fire trunks while the corresponding line in the subtitle is When
Monica’s around, ask me about fire trunks.. Another phenomenon is that one
utterance on script side may be split into several lines on subtitle side.
This change is made to accommodate the size of the TV screen. It is a big
challenge to deal with these changed, missing or duplicated terms during
matching. All the above problems make the task a complex $N$-to-$N$ matching
where $N\geq 0$.

Therefore, we regard the matching and projection as an IR task [Wang et al.,
2012a]. The Vector Space Model (VSM) [Salton et al., 1975] is a state-of-the-
art IR model in which each document is represented as a vector of identifiers
(here we describe each identifier as a term). The $i$th utterance $D_{i}$ in
the script is represented as a vector $D_{i}=[w_{1,i},w_{2,i},...w_{k,i}]$, in
which $k$ is the size of the term vocabulary. Many similarity functions can be
employed to calculate the similarity between two utterance vectors [Cha,
2007]. Here we apply the cosine distance:

$\displaystyle\textit{sim}(d_{i},d_{j})={\sum_{k=1}^{N}w_{i,k}\cdot
w_{j,k}}{\sqrt{\sum_{k=1}^{N}w_{i,k}}\cdot{\sqrt{\sum_{k=1}^{N}w_{j,k}}}}$ (1)

where $N$ is the number of terms in an utterance vector, and $w_{i,k}$ and
$w_{j,k}$ represent the weight of the $i\textit{th}$/$j\textit{th}$ term in
the utterance $D_{i}$/$D_{j}$ respectively. Technically, the distance between
documents in VSM is calculated by comparing the deviation of angles between
vectors. A Boolean Retrieval Model sets a term weight to be either 0 or 1,
while an alternative solution is calculating the term weights according to the
appearance of a term within the document collection.

To calculate the term weights according to the appearance of a term within the
document collection, we apply term frequency-inverse document frequency (TF-
IDF) [Ramos, 2003] as one term-weighting model. The weight $w$ of each term
$t$ is determined by its own term frequency $tf(t,d)$ in a document $d$ and
its inverse document frequency $idf(t,d,D)$ within the search collection. The
definition of term weight $w_{t,d}$ is shown as in Eq. (2) and (3):

$\displaystyle w_{t,d}$ $\displaystyle=$ $\displaystyle tf(t,d)\cdot
idf(t,d,D)$ (2) $\displaystyle\textit{idf}(t,d,D)$ $\displaystyle=$
$\displaystyle\log\big{(}\frac{|D|}{|\\{d\in D|t\in d\\}|}\big{)}$ (3)

where $D$ is the total number of documents in the document collection.

In practice, we regard each utterance as a document and build the index for
each movie script. Then we use each subtitle sentence as a query to search for
target related utterances. In order to deal with inconsistency problems, we
employ several strategies:

  * •

For better indexing and searching, we split the sentences/utterances into the
smallest units using a sentence splitter;

  * •

Except for punctuation mark, we do not remove any stop words. Furthermore we
low-case each word;

  * •

For each original query, it can be split into $n$ sub-queries. For each sub-
query, we apply $1$-best search. Then search results of the sub-queries are
combined to vote for the best candidate for the original query.

  * •

One query may be similar to several utterances in different lines of a script.
The candidate closest to the last matched term is more likely to be the right
answer. Thus we impose a dynamic window for subspace searching.

After the script and subtitle are bridged, we project speaker tags and
dialogue boundaries in scripts to their corresponding lines in subtitles.
Finally, we preserve the results in XML format, which is illustrated in Figure
3.

Figure 3: A sample of generated XML of dialogue in episode script

##  4\. Experiments and Results

For dialogue corpus construction, we apply our methods to a ten-season sitcom
Friends. We extract and process both scripts and subtitles of Friends
(described in Section 3.1) and then bridge them (described in Section 3.2) to
build a dialogue corpus in the format of Figure 3. For data processing, we
employ the sentence splitter and English tokenizer in the Moses toolkit and
our in-house Chinese segmentor [Wang et al., 2012b]. Furthermore, we employ
Apache Lucene777Available at https://lucene.apache.org. for indexing and
search tasks. Table 1 presents the main statistics of the resulting bilingual
dialogue corpus. We obtained 5,428 bilingual dialogues with annotated speaker
and dialogue boundary information.

Item | Size
---|---
Total number of scripts processed | 236
Total number of dialogues | 5,428
Total number of speakers | 42
Total number of utterances | 109,268
Average amount of dialogues per script | 23
Average amount of speakers per dialogue | 3.5
Average amount of utterances per dialogue | 20
Table 1: Statistics of generated parallel dialogue corpus

To verify the validity of our methods (described in Section 3), we conduct an
evaluation on the matching accuracy of speaker tags and dialogue boundaries in
the generated corpus. To generate gold standard reference, we also manually
annotate the dialogue information based on the generated parallel dialogue
corpus. The agreements between automatic labels and manual labels is 81.79% on
speaker and 98.64% on dialogue boundary, respectively. This indicates that the
proposed automatic annotation strategy through mapping is reasonably
trustworthy.

Furthermore, we conduct a simple experiment to explore the effects of speaker
tags on dialogue MT. We first build a baseline MT engine using Moses [Koehn et
al., 2007] on our generated parallel corpus (described in Table 1). We train a
$5$-gram language model (LM) using the SRI Language Toolkit [Stolcke, 2002] on
the target side of parallel corpus. Besides, we use GIZA++ [Och and Ney, 2003]
for word alignment and minimum error rate training [Och, 2003] to optimize
feature weights. Based on the hypothesis that different types of speakers may
have specific speaking styles, we employ a language model adaptation method to
boost the MT system [Wang et al., 2014]. Instead of building a LM on the whole
data, we split the data into two separate parts according the speakers’ sex
and then build two separate LMs. As Moses supports multiple LM integration, we
directly feed Moses two LMs. The translation results are listed in Table 2.

Systems | Lang. | Dev Set | Test Set
---|---|---|---
ZH-EN | Baseline | 20.32 | 16.33
SpeakerLM | 21.05 | 16.83 (+0.50)
EN-ZH | Baseline | 16.78 | 14.11
SpeakerLM | 17.23 | 14.54 (+0.43)
Table 2: Translation results on speaker based language model adaption.

For Chinese-to-English (i.e. “ZH-EN”), the baseline system achieves 20.32 and
16.33 in BLEU score on development and test data, respectively, while for
English-to-Chinese (i.e. “EN-ZH”), the scores are 16.78 and 14.11 in BLEU
score. The BLEU scores are relatively low because 1) we have only one
reference, 2) the training corpus is small, and 3) dialogue MT is a
challenging task. By using LM adaptation, we improve the performance on test
data by +0.50 and +0.43 BLEU points on Chinese-to-English and English-to-
Chinese tasks respectively.

##  5\. Conclusions and Future Work

We propose a novel approach to build a parallel dialogue discourse corpus from
monolingual scripts and their corresponding bilingual subtitles. We identify
the dialogue boundaries according to the scene or shot tags in the script to
segment the monolingual dialogue, and then map the matched monolingual
dialogues to the source part of the bilingual subtitles with the speaker and
utterance elements in order to obtain the bilingual discourse dialogues.
Finally we align the bilingual dialogue subtitle lines to produce suitable MT
training material.

We expand the current dialogue generation resources from movie scripts to
movie/episode scripts, and specify the current parallel corpus construction to
bilingual dialogue corpus building based on bilingual subtitles. We pilot this
approach on a 10-season sitcom Friends and automatically generated 5,428
bilingual parallel dialogue discourses. This is a quick way to generate a
bilingual dialogue corpus.

To validate the effect of the proposed approach, we annotated the speaker and
dialogue boundary elements manually in 4-season Friends data and compared the
manual results with our automatic findings. Experimental results show that the
automatic annotation approach can achieve around 81.79% and 98.64% on dialogue
boundaries and speaker tags, respectively. Furthermore, we explore the
integration of speaker tags into MT using domain-adaptation techniques. The
experiments show that we can improve translation performance compared to a
baseline system.

As far as future work is considered, we intend to explore automatic dialogue
detection from bilingual subtitles. A reachable goal is to utilize the
resulting bilingual dialogue corpus based on our approach to also summarize
the discourse elements such as coherence and co-reference, speaker
relationship and time information of the subtitle lines. Some supervised and
semi-supervised methods and machine learning approaches can be used on these
tasks.

##  6\. Acknowledgements

This work is supported by the Science Foundation of Ireland (SFI) ADAPT
project (Grant No.:13/RC/2106), and partly supported by the DCU-Huawei Joint
Project (Grant No.:201504032-A (DCU), YB2015090061 (Huawei)). It is partly
supported by the Open Projects Program of National Laboratory of Pattern
Recognition (Grant 201407353) and the Open Projects Program of Centre of
Translation of GDUFS (Grant CTS201501).

##  7\. Bibliographical References

## References

  * Aizawa et al., 2000 Aizawa, Y., Matsubara, S., Kawaguchi, N., Toyama, K., and Inagaki, Y.  (2000).  Spoken language corpus for machine interpretation research.  In Proceedings of the 6th International Conference on Spoken Language Processing, volume 3, pages 398–401, Beijing, China.
  * Banchs, 2012 Banchs, R. E.  (2012).  Movie-dic: A movie dialogue corpus for research and development.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers, volume 2, pages 203–207, Jeju Island, Korea.
  * Cha, 2007 Cha, S.-H.  (2007).  Comprehensive survey on distance/similarity measures between probability density functions.  International Journal of Mathematical Models and Methods in Applied Sciences, 1:300–307.
  * Danescu-Niculescu-Mizil and Lee, 2011 Danescu-Niculescu-Mizil, C. and Lee, L.  (2011).  Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs.  In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, pages 76–87, Portland, Oregon.
  * Itamar and Itai, 2008 Itamar, E. and Itai, A.  (2008).  Using movie subtitles for creating a large-scale bilingual corpora.  In Proceedings of the 6th International Conference on Language Resources and Evaluation, pages 269–272, Marrakech, Morocco.
  * Koehn et al., 2007 Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E.  (2007).  Moses: Open source toolkit for statistical machine translation.  In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic.
  * Lavecchia et al., 2007 Lavecchia, C., Smaili, K., and Langlois, D.  (2007).  Building parallel corpora from movies.  In Proceedings of the 4th International Workshop on Natural Language Processing and Cognitive Science, pages 201–210, Funchal, Madeira, Portugal.
  * Matsubara et al., 2002 Matsubara, S., Takagi, A., Kawaguchi, N., and Inagaki, Y.  (2002).  Bilingual spoken monologue corpus for simultaneous machine interpretation research.  In Proceedings of the 3rd International Conference on Language Resources and Evaluation, pages 153–159, Las Palmas, Canary Islands - Spain.
  * Meyer and Popescu-Belis, 2012 Meyer, T. and Popescu-Belis, A.  (2012).  Using sense-labeled discourse connectives for statistical machine translation.  In Proceedings of the Joint Workshop on Exploiting Synergies Between Information Retrieval and Machine Translation and Hybrid Approaches to Machine Translation, pages 129–138, Avignon, France.
  * Och and Ney, 2003 Och, F. J. and Ney, H.  (2003).  A systematic comparison of various statistical alignment models.  Computational Linguistics, 29(1):19–51.
  * Och, 2003 Och, F. J.  (2003).  Minimum error rate training in statistical machine translation.  In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, pages 160–167, Sapporo, Japan.
  * O’Hagan, 2012 O’Hagan, M.  (2012).  From fan translation to crowdsourcing: Consequences of web 2.0 user empowerment in audiovisual translation.  Approaches to Translation Studies, 36:25–41.
  * Prasad et al., 2008 Prasad, R., Dinesh, N., Lee, A., Miltsakaki, E., Robaldo, L., Joshi, A. K., and Webber, B. L.  (2008).  The penn discourse treebank 2.0.  In Proceedings of the 6th International Conference on Language Resources and Evaluation, Marrakech, Morocco.
  * Ramos, 2003 Ramos, J.  (2003).  Using tf-idf to determine word relevance in document queries.  In Proceedings of the 1st instructional conference on machine learning, Piscataway, NJ USA.
  * Ryu et al., 2003 Ryu, K., Matsubara, S., Kawaguchi, N., and Inagaki, Y.  (2003).  Bilingual speech dialogue corpus for simultaneous machine interpretation research.
  * Salton et al., 1975 Salton, G., Wong, A., and Yang, C.-S.  (1975).  A vector space model for automatic indexing.  Communications of the ACM, 18:613–620.
  * Schmitt et al., 2012 Schmitt, A., Ultes, S., and Minker, W.  (2012).  A parameterized and annotated spoken dialog corpus of the cmu let’s go bus information system.  In Proceedings of the 8th International Conference on Language Resources and Evaluation, pages 3369–3373, Istanbul, Turkey.
  * Stolcke, 2002 Stolcke, A.  (2002).  Srilm - an extensible language modeling toolkit.  In Proceedings of the 7th International Conference on Spoken Language Processing, pages 901–904, Colorado, USA.
  * Takezawa, 2003 Takezawa, T.  (2003).  Collecting machine-translation-aided bilingual dialogues for corpus-based speech translation.  In Proceedings of the 8th European Conference on Speech Communication and Technology, pages 2757–2760, Geneva, Switzerland.
  * Tiedemann, 2007a Tiedemann, J.  (2007a).  Building a multilingual parallel subtitle corpus.  In Proceedings of the 17th Conference on Computational Linguistics in the Netherlands, pages 1–14, Leuven, Belgium.
  * Tiedemann, 2007b Tiedemann, J.  (2007b).  Improved sentence alignment for movie subtitles.  In Proceedings of the 3rd Conference on Recent Advances in Natural Language Processing, volume 7, pages 582–588, Borovets, Bulgaria.
  * Tiedemann, 2008 Tiedemann, J.  (2008).  Synchronizing translated movie subtitles.  In Proceedings of the 6th International Conference on Language Resources and Evaluation, pages 1902–1906, Marrakech, Morocco.
  * Tiedemann, 2012 Tiedemann, J.  (2012).  Parallel data, tools and interfaces in opus.  In Proceedings of the 8th International Conference on Language Resources and Evaluation, pages 2214–2218, Istanbul, Turkey.
  * Wahlster, 2013 Wahlster, W.  (2013).  Verbmobil: foundations of speech-to-speech translation.  Springer Science & Business Media, Berlin.
  * Walker et al., 2012 Walker, M. A., Lin, G. I., and Sawyer, J.  (2012).  An annotated corpus of film dialogue for learning and characterizing character style.  In Proceedings of the 8th International Conference on Language Resources and Evaluation, pages 1373–1378, Istanbul, Turkey.
  * Wang et al., 2012a Wang, L., Wong, D. F., and Chao, L. S.  (2012a).  An improvement in cross-language document retrieval based on statistical models.  In Proceedings of the 24th Conference on Computational Linguistics and Speech Processing, pages 144–155, Chung-Li, Taiwan.
  * Wang et al., 2012b Wang, L., Wong, D. F., Chao, L. S., and Xing, J.  (2012b).  Crfs-based chinese word segmentation for micro-blog with small-scale data.  In Proceedings of the 2nd conference jointly organized by the Chinese Language Processing Society of China and the Association for Computational Linguistics Special Interest Group on Chinese Language Processing, pages 51–57, Tianjin, China.
  * Wang et al., 2014 Wang, L., Lu, Y., Wong, D. F., Chao, L. S., Wang, Y., and Oliveira, F.  (2014).  Combining domain adaptation approaches for medical text translation.  In Proceedings of the 9th Workshop on Statistical Machine Translation, pages 254–259, Baltimore, USA.
  * Xiao and Wang, 2009 Xiao, H. and Wang, X.  (2009).  Constructing parallel corpus from movie subtitles.  In Proceedings of the 22nd International Conference on Computer Processing of Oriental Languages. Language Technology for the Knowledge-based Economy, pages 329–336, Hong Kong, China.
  * Zhang et al., 2014 Zhang, S., Ling, W., and Dyer, C.  (2014).  Dual subtitles as parallel corpora.  In Proceedings of the 10th International Conference on Language Resources and Evaluation, pages 1869–1874, Reykjavik, Iceland.