# Auxiliary Class Based Multiple Choice Learning

Sihwan Kim Dae Yon Jung Taejang Park
Hana Institute of Technology
Hana TI, Seoul, Korea
{shnasc.kim, dyjung16<EMAIL_ADDRESS>

###### Abstract

The merit of ensemble learning lies in having different outputs from many
individual models on a single input, _i.e_., the diversity of the base models.
The high quality of diversity can be achieved when each model is specialized
to different subsets of the whole dataset. Moreover, when each model
explicitly knows to which subsets it is specialized, more opportunities arise
to improve diversity. In this paper, we propose an advanced ensemble method,
called Auxiliary class based Multiple Choice Learning (AMCL), to ultimately
specialize each model under the framework of multiple choice learning (MCL).
The advancement of AMCL is originated from three novel techniques which
control the framework from different directions: 1) the concept of auxiliary
class to provide more distinct information through the labels, 2) the
strategy, named memory-based assignment, to determine the association between
the inputs and the models, and 3) the feature fusion module to achieve
generalized features. To demonstrate the performance of our method compared to
all variants of MCL methods, we conduct extensive experiments on the image
classification and segmentation tasks. Overall, the performance of AMCL
exceeds all others in most of the public datasets trained with various
networks as members of the ensembles.

##  1 Introduction

Ensemble learning is the process to train multiple models and combine into a
single result. The spotlights to ensemble methods, even until recently from
numerous application fields [1, 2, 3, 4], are derived from their superior
predictive performances compared to that of a single model. This high
performance of the ensemble results primarily from the diversity of the
multiple models. If all individual models provide similar outputs, any
strategic combination of them cannot reduce the error. Therefore, achieving
high diversity while training an ensemble model, _i.e_. producing a set of
various possible outputs, is where the quality of ensemble originates.

Diversity can usually be achieved by alternating either datasets or model
configurations, such as hyperparameters and/or initializations, for training
each individual model. Relying on randomization, many of these ensembles
require a good number of individual models to fulfill the law of large
numbers. However, rather than differentiating models by randomness, multiple
choice learning (MCL) [5] obtains high diversity by making each model
specialize to a certain subset of the dataset. These specialized models from
MCL are directly trained by minimizing a new concept of loss function called
oracle loss, which assigns each example to one or more, depending on the
setting of the constraints, models with the highest accuracy.

Inheriting the concept of the oracle loss, [6] applied MCL to ensembles of
deep neural networks with a trainable algorithm and named stochastic multiple
choice learning (sMCL). Since each specialized model takes the loss only from
a certain set of examples, overconfident predictions to unassigned examples
occur by nature. To address this overconfidence issue, a series of progressive
methods are introduced. [7] added a regularization term to the oracle loss in
order to coerce the predictive probability of an unassigned example to be a
uniform distribution. [8] modified the regularization term to set a fixed
margin between the aggregated probability of the ground truth and the others.
In addition, a choice network is appended to the architecture to combine the
predictions from all specialized models with different weights. Both methods
substantially improved top-1 error rates while maintaining as good or better
oracle errors compared to those of the sMCL method.

Although the accuracies of the ensembles are boosted, all previous methods,
utilizing the concept of the oracle loss, do not provide explicit information
about the unassigned examples to each specialized model. Setting the
distribution [7] or the value range [8] of the predictive probabilities of
unassigned examples is not a direct manner to encourage model specialization.
The best way to lead the model specialization is to set labels of the
unassigned examples with low entropies, directly indicating that the examples
are not from any subsets of the model’s specialties. This clear indication
will lead every model in the ensemble to an extreme specialization, resulting
in highly diverse outputs with lower prediction errors even when simply
combined using an averaging scheme.

Inspired by this extreme model specialization, in this paper, we propose an
advanced method, called Auxiliary class based Multiple Choice Learning (AMCL),
that expressly aims to train each base model of an ensemble to become the best
specialist. As the main contributions of AMCL, we firstly develop the
auxiliary class, an extra element appended to the ground truth label. This
class indicates whether or not the input example belongs to a part of the
model’s specialties. From a perspective of training, the auxiliary class helps
to efficiently handle the examples from the non-specialized classes. Secondly,
unlike all previous MCL methods that designate each input to a particular
model based on the loss value, _i.e_. loss-based assignment, we suggest a
principled technique, _i.e_. memory-based assignment, based on the connections
between the input examples and the models in the early iterations. Once the
assignment of the inputs to the models are fixed after the first few
iterations, the input examples are directly forwarded in the later iterations
that helps each model to solely focus on its specialization. Finally, we
propose a module to fuse low-level features from each model and train the most
informative feature through the attention mechanism. This module, named
feature fusion module, supports to overcome weak generalization which
typically occurs under the framework of MCL.

We evaluate AMCL on two representative computer vision tasks: image
classification and segmentation. With image classification, extensive
experiments are conducted with three large-scale neural networks, including
VGGNet [9], ResNet [10], and GoogLeNet [11], on three public image datasets,
including CIFAR-10/100 [12] and TinyImageNet [13]. Compared to the existing
MCL methods, AMCL significantly outperforms in both the oracle and the top-1
errors. We also provide general comparisons with the recent ensemble models,
such as the method by Opitz _et al_. [14] (abbreviated as EMADNN), ONE [15],
and KDCL [16], in which AMCL performs better than any of those baselines.
Additionally, the performance on the foreground-background segmentation is
evaluated with the iCoseg [17] dataset. The overall results show that our
proposed method achieves impressive performance in most of the experiments,
surpassing the top-1 accuracy as well as the oracle accuracy compared to the
other MCL extended methods.

##  2 Preliminaries

Figure 1: The learning framework of AMCL method with 3 networks specialized to
2 networks per class, $i.e.,M=3$ and $K=2$. All three techniques (auxiliary
class, memory-based assignment and feature fusion module) are visualized in
this diagram. “Fixed specialization” is a matrix indicating the specialization
of each model to the classes, which is used in the process of memory-based
assignment for training after $T_{\tau}$ epochs.

In this section, we describe the formulation of the oracle loss from multiple
choice learning [5], on which all MCL related algorithms are based. Let
$D=\\{(\mathbf{x}_{j},y_{j})_{j=1}^{N}\\}$ be a dataset, where
$\mathbf{x}_{j}$ represents a training example, and $y_{j}$ is the ground
truth label, consisting of $N$ i.i.d. observations. Each individual model is
denoted by $f_{m}(\mathbf{x}),~{}m=1,\cdots,M$, where $M$ implies the ensemble
size. Formally, a traditional independent ensemble (IE) method trains each
model with the following objective:

$\mathcal{L}_{IE}(D)=\sum_{j=1}^{N}\sum_{m=1}^{M}\ell\Big{(}y_{j},f_{m}(\mathbf{x}_{j})\Big{)},$
(1)

where $\ell(\cdot,\cdot)$ implies the task-specific loss function. Differently
from the traditional IE method, the objective of MCL is to minimize the oracle
loss, which accounts only the lowest loss from the predictions of the $M$
models, defined by:

$\mathcal{L}_{oracle}(D)=\sum_{j=1}^{N}\min_{m\in\\{1,\cdots,M\\}}\ell\Big{(}y_{j},f_{m}(\mathbf{x}_{j})\Big{)}.$
(2)

Owing to this loss function, the most accurate model for an individual example
$\mathbf{x}_{j}$ is determined, and thus, each model in the ensemble is
trained to become a specialist to certain subsets or classes of the dataset.

Since the oracle loss (2) is a non-continuous function, an iterative block
coordinate decent algorithm is used to optimize the function. However,
directly applying MCL to an ensemble of deep neural networks is not efficient
due to its training costs. As a solution, Stochastic Multiple Choice Learning
(sMCL) [6] proposed a training scheme based on the stochastic gradient descent
algorithm to concurrently train ensembles of deep neural networks.
Specifically, sMCL relaxed the objective function to the following integer
programming problem:

$\displaystyle\mathcal{L}_{sMCL}(D)=\min_{v_{j}^{m}}\sum_{j=1}^{N}\sum_{m=1}^{M}v_{j}^{m}\ell\Big{(}y_{j},\mathbf{P}_{m}(\mathbf{x}_{j})\Big{)},\quad\text{subject
to}\quad\sum_{m=1}^{M}v_{j}^{m}=K,$ (3)

where $\mathbf{P}_{m}(\mathbf{x}_{j})$ is the prediction of $m$-th network,
and $v_{j}^{m}\in\\{0,1\\}$ is an indicator variable representing the
assignment of the input $\mathbf{x}_{j}$ to the $m$-th model.
$K\in\\{1,\cdots,M\\}$ is the number of specialized models per each input
example. We note that if $K=1$, the algorithm chooses the single most accurate
network for every input example and back-propagates only in the corresponding
network. Consequently, each network is trained and specialized to a certain
subset of the dataset. When $K=M$, sMCL is identical to the IE method.

##  3 Auxiliary class based Multiple Choice Learning

In this section, we describe the technical details and novelties of our
proposed training method, consisting of three major parts: (a) auxiliary
class, (b) memory-based assignment, and (c) feature fusion module. These
techniques combined enhance the specialization of each model in the ensembles
which, in turn, ultimately promotes the performance in both oracle and top-1
error rates. We remark that the three newly introduced parts have no
restriction on the architecture of each individual models, which lead our
method applicable to general fields. The overall learning framework of AMCL is
illustrated in Figure 1.

###  3.1 Auxiliary Class

The oracle loss (2) used for MCL is useful for producing diverse solutions in
high quality because only the most accurate network corresponding to each
input example is trained, and thus, each model eventually focuses on a certain
subset of the dataset. However, this property yields to the overconfidence
issue. To address this issue, confident multiple choice learning (CMCL) [7]
proposed a new loss function named $\mathit{confident~{}oracle~{}loss}$. The
confident oracle loss adds a regularization term to the original oracle loss
to minimize Kullback-Leibler divergence between the predictive distribution of
an unassigned example and a uniform distribution.

To understand better about the effect of this regularization term, assume that
we have an image dataset with two classes: dog and airplane. If model ‘A’ and
‘B’ are specialized in each class, dog and airplane, respectively, the
predictive probability of model ‘A’ for any input of airplane images will be
uniform, $[1/2,1/2]^{T}$. Thus, CMCL forces each model to be less confident
about non-specialized classes, and in terms of performance, it notably reduces
the top-1 error while keeping the oracle error similar to that of MCL.
Nevertheless, the interpretation of the probability is ambiguous since model
‘A’ returns a dog class with the confidence level of 50% to an airplane image.

From a human perspective, it makes more sense when model ‘A’ rejects an image
of airplane with the confidence level of 100% because model ‘A’ is specialized
in the dog class, not airplane. To this end, we add an auxiliary class to
efficiently handle the non-specialized classes of each model. When model ‘A’
is fully trained with this auxiliary class, the predictive probability of
model ‘A’ for an input of airplane will be $[0,0,1]^{T}$, where the last
component is the auxiliary class. This probability explicitly indicates that
the input is definitely not a dog image. At the inference stage, model ‘A’
regards the probabilities of airplane images as $[0,0]^{T}$ through a simple
post-processing, which directly removes the auxiliary class. Since model ‘B’
is specialized in the airplane class, the predictive probability to an example
of airplane will be $[0,1]^{T}$ after the post-processing, and thus, the
ensemble prediction via averaging scheme becomes $[0,1/2]^{T}$. When the
prediction is normalized to make the sum equal to 1, AMCL results in
$[0,1]^{T}$ while CMCL’s prediction is $[1/4,3/4]^{T}$ by calculating the
average between $[1/2,1/2]^{T}$ from model ‘A’ and $[0,1]^{T}$ from model ‘B’.
Through this simple yet effective technique, AMCL produces a more intuitive
and plausible solution compared to CMCL.

(a) The histogram of dog (b) The histogram of airplane

Figure 2: The histogram of the confidence level in the class of dog and
airplane. Each histogram from AMCL and CMCL is in red and blue, respectively.
The test dataset consists of 1,000 samples from each class.

To demonstrate the example mentioned above, we trained an ensemble model with
a simple CNN using only two classes, dog and airplane, from CIFAR-10 and
compared the confidence level between CMCL and AMCL. The simple CNN consists
of three convolutional layers with 32, 64 and 128 filters followed by a fully-
connected layer. As shown in Figure 2, the results indeed show that the
confidence level of both classes in CMCL lies on near 0.75 while that of AMCL
lies on near 1. Additionally, AMCL shows the oracle / top-1 error rates of
0.03$\%$ / 3.12$\%$, which is superior to those of CMCL, 0.21$\%$ / 3.21$\%$.
AMCL achieves up to 85.71$\%$ relative reduction in the oracle error rate from
the corresponding model trained with the CMCL scheme.

###  3.2 Memory-based Assignment

Following the constraints of the oracle loss (2), only the networks with the
lowest loss for the corresponding input examples are trained with the ground
truth labels. Since all the variants of MCL adopt the same oracle loss, this
procedure occurs at each training step. With this strategy, loss-based
assignment, the examples which do not belong to the specialized class of a
model might yield to the lowest loss, and as a consequence, these wrongly
assigned examples diminish the purity of the model specialization.

To tackle this problem, we introduce a simple technique to raise the purity of
input to model assignment, named memory-based assignment. The memory-based
assignment counts, by online manner, the number of examples per class that
belongs to each model within the first few epochs of the training. The numbers
of examples in some classes becomes incrementally larger than those of the
other classes, so after couple epochs, the specialized classes for each model
can be naturally decided. After a set number of epochs, the specialized
classes are hard-coded according to the counts of examples that results in the
“Fixed specialization” matrix in Figure 1. This strategy helps the model to
focus more on a certain subset of the dataset for the rest of the training
without any inputs that interrupt the specialization.

To describe in more detail with Figure 1, whether to assign the label to the
auxiliary class or not is initially decided based on the loss values. After
the set number of epochs, the specialization of each model is fixed according
to the history of the past assignments, which is the “Fixed specialization”.
Therefore, as shown in the right part of the figure, even when the probability
of cat turns out to be the lowest given one of the cat images, the assignment
of the label to the second model is still cat, and this individual model will
be trained to specialize in the images of cats.

Originated from the objective function of sMCL (3), a modified loss for our
algorithm implements both of the auxiliary class and the memory-based
assignment. Let $P_{\theta_{m}}(\widetilde{y}~{}|~{}\mathbf{x})$ be the
predictive distribution where $\widetilde{y}$ is the output with the auxiliary
class appended, and $\theta_{m}$ denotes the $m$-th model parameters.
$\mathcal{A}(\widetilde{y})$ is the one-hot encoded vector with only the
auxiliary class being 1. Also, $t_{s}$ is the current epoch number, and
$T_{\tau}$ denotes the threshold of epochs from loss-based to memory-based.
Then, the objective of AMCL is defined as follows:

$\displaystyle\mathcal{L}_{AMCL}(D)=\begin{cases}\mathcal{L}_{LBA}(D)&\mbox{if}~{}t_{s}\leq
T_{\tau},\\\ \mathcal{L}_{MBA}(D)&\mbox{otherwise},\end{cases}$ (4)

where ${L}_{LBA}(D)$ and ${L}_{MBA}(D)$ are the loss functions from loss-based
assignment (LBA) and memory-based assignment (MBA), which are presented below.

$\displaystyle\begin{aligned}
&\mathcal{L}_{LBA}(D)=\min_{v_{j}^{m}}\displaystyle\sum_{j=1}^{N}\displaystyle\sum_{m=1}^{M}v_{j}^{m}\ell\Big{(}\widetilde{y}_{j},P_{\theta_{m}}(\widetilde{y}~{}|~{}\mathbf{x}_{j})\Big{)}+\beta(1-v_{j}^{m})D_{KL}\Big{(}\mathcal{A}(\widetilde{y})~{}||~{}P_{\theta_{m}}(\widetilde{y}~{}|~{}\mathbf{x}_{j})\Big{)}\\\
\vskip 3.0pt plus 1.0pt minus
1.0pt&\qquad\qquad\qquad\qquad\qquad\qquad\text{subject
to}\quad\enspace\displaystyle\sum_{m=1}^{M}v_{j}^{m}=K,\end{aligned}$ (5)

and

$\displaystyle\mathcal{L}_{MBA}(D)=\displaystyle\sum_{j=1}^{N}\displaystyle\sum_{m=1}^{M}w_{c(j)}^{m}\ell\Big{(}\widetilde{y}_{j},P_{\theta_{m}}(\widetilde{y}~{}|~{}\mathbf{x}_{j})\Big{)}+\gamma(1-w_{c(j)}^{m})D_{KL}\Big{(}\mathcal{A}(\widetilde{y})~{}||~{}P_{\theta_{m}}(\widetilde{y}~{}|~{}\mathbf{x}_{j})\Big{)},$
(6)

where $D_{KL}$ denotes the KL divergence, $\beta$ and $\gamma$ are the penalty
parameters, and $v_{j}^{m}$ is the indicator variable as defined in (3).
$w_{c(j)}^{m}$ is an element of binary flag matrix of the dimension
$(N_{c}\times M)$ where $c(j)$ denotes the class index for each input example
$\mathbf{x}_{j}$, and $N_{c}$ is the total number of classes. This matrix is
generated by the cumulative sum of $v_{j}^{m}$ until the epoch reaches the
threshold, $T_{\tau}$. Each element of the matrix is converted to a binary
flag by choosing the top-$K$ counts, row-wise, where $K$ means the set number
of models for each class. Therefore, $w_{c(j)}^{m}$ implies that the $m$-th
model is specialized in the class of the input example $\mathbf{x}_{j}$, and
this specific class will be assigned to the $m$-th model until the end of the
training. As shown in Figure 3, when the epoch is 10, about 2$\sim$3 $\%$ of
the inputs from each class are not assigned properly to the specialized models
using CMCL. From the perspective of specialization, these inputs become
potential noise, which indeed affect the performance. In the case of AMCL,
each model becomes a specialist for certain classes with the purity of high
intensity compared to the other MCL methods.

Figure 3: The change in within-class ratios for the cumulative sum of
$v_{j}^{m}$ with $K=1$ at certain epochs. As the training lasts, AMCL with the
memory-based assignment shows more intensive ratios towards the specialized
classes for each model even with respect to $v_{j}^{m}$, which is the
indicator variable from the loss-based assignment. This demonstration results
are from the ensemble with simple CNN models (See Section 3.1) with the epoch
threshold $T_{\tau}$ at 10.

###  3.3 Feature Fusion Module

Only with the auxiliary class and the memory-based assignment, it is
challenging to overcome the weak generalization problem from each model under
the MCL framework as mentioned in [18]. When the model specialization is
completed with only the auxiliary class and the memory-based assignment, each
model is trained without the general information about the whole dataset
because all the examples belonging to the non-specialized classes are labeled
to a single auxiliary class. Therefore, to further address this generalization
issue, we suggest an additional module, called feature fusion module, in the
lower level of the network with the architecture similar to the attention
networks [19, 20]. As each model in the ensemble is forced to learn low-level
features differently, these features are concatenated and fed into the feature
fusion module to generate a more comprehensive but concise feature, which is
uniformly provided back to each model for further training. The fusion module
is deployed just after few layers of each model since the first few layers
tend to learn more common features as described in [21]. Owing to our feature
fusion module, the different perspectives about a single input from each
ensemble members are combined in the early layer, and all model can learn
their own specialized expertise with more generalized features.

In the previous methods, similar techniques to enhance the low-level features
are applied as well. CMCL proposes the feature sharing scheme, which
stochastically shuffles the features among the ensemble models. Meanwhile,
versatile multiple choice learning (vMCL) [8] suggests to share the weights of
the couple foremost convolutional layers among the models. However, in
contrast to these previous techniques, feature fusion module approaches this
problem in terms of learning the optimal knowledge from all models. The effect
of this module compared to the other techniques is experimentally examined in
Section 4.1.

##  4 Experiments

We evaluate the experiments on classification and foreground-background
segmentation tasks to verify the effectiveness of our algorithm, AMCL. The
three public image datasets, including CIFAR-10/100 [12] and TinyImageNet
[13], are used for classification, and the iCoseg [17] dataset is used for
segmentation. In all experiments, we compare AMCL with the variants of MCL
under the same experimental setups for fair comparisons. In addition to this
section, the details about the experimental setups and the hyerparameter
studies are provided in the supplementary material.

###  4.1 Image Classification

We compare all methods using three conventional large CNN models. ResNet-18 is
used for the ablation and the effect studies while VGGNet-16, ResNet-34, and
GoogLeNet are applied for the comprehensive experiments evaluating
classification tasks. All experiments in this section uses ensembles of 5
networks, $M=5$. The experiment with a larger number of networks is presented
in the supplementary material.

Both oracle and top-1 error rates are measured for evaluating the
classification performances. The oracle error is the ratio when all models
failed to classify for a given test image, and the top-1 error is calculated
by averaging the output probabilities of all models. We note that a lower
oracle error rate implies high diversity but not always the highest top-1
accuracy.

Ablation study. We study the contributions of our three major techniques under
the comparison with IE and the variants of MCL. All methods in comparison are
trained on CIFAR-10 with an ensemble of 5 ResNet-18. As shown in Table 1, the
performance improves gradually as each technique is incrementally applied.
Through this study, we note that memory-based assignment contributes more to
lowering the oracle error while feature fusion module reduces the top-1 error.
When all techniques are applied, AMCL achieves the best performance compared
to all others. We find that AMCL provides 36.3$\%$ and 10.1$\%$ relative error
reductions in the oracle and top-1 error rates, respectively, from CMCL, the
second best method in terms of harmonic mean of the oracle and top-1 error.

Table 1: The classification error rates (%) on CIFAR-10 with the IE and the previous MCL methods. “MBA” and “FFM” are short for memory-based assignment and feature fusion module, respectively. The best score is highlighted in bold, and the values in parentheses represent the relative error reductions from CMCL, which is the second best result in terms of harmonic mean. Ensemble Method | MBA | FFM | Oracle | Top-1
---|---|---|---|---
IE | - | - | 5.10 | 10.18
sMCL | - | - | 3.27 | 56.54
CMCL | - | - | 3.36 | 10.83
vMCL | - | - | 3.54 | 11.49
AMCL | - | - | 3.42 | 12.42
$\surd$ | - | 2.54 | 11.63
- | $\surd$ | 3.13 | 10.86
$\surd$ | $\surd$ |  2.14 (-36.3%) |  9.74 (-10.1%)

(a) Probability of auxiliary class

(b) From specialized classes

(c) From non-specialized classes

Figure 4: The distribution of measurements for analyzing the effect of
auxiliary class. The classes of CIFAR-10 are separated into specialized and
non-specialized by the “Fixed specialization” matrix generated during the
training. (a) The probability density of auxiliary class on specialized, non-
specialized, and SVHN (unseen) test data. (b)-(c) The cross-entropy of the
test examples from (b) specialized and (c) non-specialized classes.

Analysis of auxiliary class. Due to the auxiliary class, AMCL effectively
prevents overconfident predictions to the non-specialized classes by setting
the ground truth label to $\mathcal{A}(\widetilde{y})$(See Equation (5) and
(6)). Since the probability of auxiliary class explicitly indicates whether or
not an input example is specialized to the corresponding model, it can be
exploited to measure the uncertainty on unseen examples. To verify the quality
of the uncertainty level through AMCL, we investigate the probability of the
auxiliary class for the images from SVHN [22] using an ensemble model consists
of 5 ResNet-18 trained by only the CIFAR-10 dataset with the overlap parameter
$K=1$. As presented in Figure 4(a), AMCL produces a high probability towards
the auxiliary class on the unseen data, which is almost as high as the
probability of the non-specialized classes used in training. Additionally, we
examine the cross-entropies between the ground truth labels and the
predictions of the images from specialized and non-specialized classes,
separately, as presented in Figures 4(b) and 4(c). The predictions are from
the models trained with three variants of MCL. In contrast to the other
methods, AMCL shows a large margin between the cross-entropies from the
specialized and the non-specialized classes because AMCL distinctively knows
the specialization of each model through the auxiliary class. This cross-
entropy results demonstrate that AMCL achieves the most extreme specialization
compared to the previous MCL approaches.

Table 2: The classification error rates (%) and the number of parameters with different generalization modules. Auxiliary class and memory-based assignment are applied in all comparisons. The best score is highlighted in bold, and the values in parentheses represent the relative error reductions from Feature sharing scheme from CMCL, the second best result in terms of harmonic mean. Method | Oracle | Top-1 | Total Parameters | Relative Ratio
---|---|---|---|---
Weight sharing [8] | 2.87 | 10.91 | 55,458K | 1.000
Feature sharing [7] | 2.57 | 10.34 | 56,058K | 1.011
Feature fusion module |  2.14 (-16.7%) |  9.74 (-5.8%) | 56,444K | 1.018

Effect of feature fusion module. We also examine the comparative performance
of feature fusion module by replacing it with the feature sharing techniques
from CMCL and vMCL. The dataset and the network used for this study is
identical to the previous ablation study. The feature fusion module is
inserted just before the first pooling layer, which as well is the same to the
application of feature sharing in CMCL. As shown in Table 2, the use of
feature fusion module results in more improvement over the other techniques.
Although the number of total parameters in the network increases by 1.8$\%$,
the improvement of classification performance, 16.7$\%$ and 5.8$\%$ relative
error reductions in the oracle and the top-1 errors, shows that feature fusion
module achieves better generalization.

Table 3: The comprehensive classification results with various CNNs. We report the mean and standard deviation (as subscript) of the oracle and the top-1 error rates(%) by repeating 3 times. Among the results with the overlap parameter $K=1,\cdots,4$, we pick the best result in terms of harmonic mean. The best score is highlighted in bold, and the second best result is in blue. Model | Ensemble | CIFAR-10 | CIFAR-100 | Tiny-ImageNet
---|---|---|---|---
Name | Method | Oracle | Top-1 | Oracle | Top-1 | Oracle | Top-1
VGGNet-16 | IE |  3.44${}_{\pm{\text{0.04}}}$ |  8.26${}_{\pm{\text{0.12}}}$ |  17.78${}_{\pm{\text{0.11}}}$ |  31.08${}_{\pm{\text{1.13}}}$ |  27.42${}_{\pm{\text{0.46}}}$ | 40.63${}_{\pm{\text{0.18}}}$
sMCL |  1.98${}_{\pm{\text{0.69}}}$ |  21.29${}_{\pm{\text{1.40}}}$ | 15.97${}_{\pm{\text{0.13}}}$ |  32.06${}_{\pm{\text{0.38}}}$ |  26.68${}_{\pm{\text{0.22}}}$ |  41.60${}_{\pm{\text{0.33}}}$
CMCL |  2.70${}_{\pm{\text{0.12}}}$ | 7.72${}_{\pm{\text{0.24}}}$ |  16.63${}_{\pm{\text{0.23}}}$ | 29.29${}_{\pm{\text{0.29}}}$ | 26.11${}_{\pm{\text{0.28}}}$ |  40.88${}_{\pm{\text{0.17}}}$
vMCL |  1.75${}_{\pm{\text{0.09}}}$ |  11.43${}_{\pm{\text{0.34}}}$ |  24.19${}_{\pm{\text{0.67}}}$ |  43.77${}_{\pm{\text{1.15}}}$ |  39.05${}_{\pm{\text{0.48}}}$ |  56.02${}_{\pm{\text{1.07}}}$
AMCL | 1.30${}_{\pm{\textbf{0.12}}}$ | 7.24${}_{\pm{\textbf{0.13}}}$ | 14.79${}_{\pm{\textbf{0.15}}}$ | 28.58${}_{\pm{\textbf{0.22}}}$ | 24.24${}_{\pm{\textbf{0.15}}}$ | 38.59${}_{\pm{\textbf{0.21}}}$
ResNet-34 | IE |  4.64${}_{\pm{\text{0.12}}}$ |  10.00${}_{\pm{\text{0.20}}}$ |  21.23${}_{\pm{\text{0.11}}}$ |  34.47${}_{\pm{\text{0.18}}}$ |  25.62${}_{\pm{\text{0.36}}}$ | 38.03${}_{\pm{\text{0.26}}}$
sMCL |  2.95${}_{\pm{\text{0.17}}}$ |  13.43${}_{\pm{\text{0.18}}}$ |  22.44${}_{\pm{\text{0.32}}}$ |  39.58${}_{\pm{\text{0.15}}}$ |  26.12${}_{\pm{\text{0.37}}}$ |  40.91${}_{\pm{\text{0.38}}}$
CMCL | 2.22${}_{\pm{\text{0.19}}}$ | 8.13${}_{\pm{\text{0.20}}}$ | 16.09${}_{\pm{\text{0.32}}}$ | 29.73${}_{\pm{\text{0.33}}}$ | 24.60${}_{\pm{\text{0.33}}}$ |  39.17${}_{\pm{\text{0.45}}}$
vMCL |  2.63${}_{\pm{\text{0.11}}}$ |  12.05${}_{\pm{\text{0.18}}}$ |  19.21${}_{\pm{\text{0.36}}}$ |  33.35${}_{\pm{\text{0.17}}}$ |  26.80${}_{\pm{\text{0.38}}}$ |  39.42${}_{\pm{\text{0.83}}}$
AMCL | 2.09${}_{\pm{\textbf{0.10}}}$ | 7.42${}_{\pm{\textbf{0.11}}}$ | 14.76${}_{\pm{\textbf{0.24}}}$ | 27.94${}_{\pm{\textbf{0.21}}}$ | 22.56${}_{\pm{\textbf{0.37}}}$ | 36.94${}_{\pm{\textbf{0.31}}}$
GoogLeNet | IE |  3.22${}_{\pm{\text{0.05}}}$ |  7.53${}_{\pm{\text{0.08}}}$ |  14.99${}_{\pm{\text{0.29}}}$ | 25.65${}_{\pm{\text{0.14}}}$ |  21.49${}_{\pm{\text{0.35}}}$ | 32.77${}_{\pm{\text{0.20}}}$
sMCL |  2.31${}_{\pm{\text{0.11}}}$ |  9.72${}_{\pm{\text{0.19}}}$ | 13.40${}_{\pm{\text{0.17}}}$ |  28.36${}_{\pm{\text{0.29}}}$ |  21.38${}_{\pm{\text{0.36}}}$ |  34.67${}_{\pm{\text{0.30}}}$
CMCL |  2.28${}_{\pm{\text{0.08}}}$ | 7.52${}_{\pm{\text{0.26}}}$ |  13.51${}_{\pm{\text{0.37}}}$ |  25.97${}_{\pm{\text{0.34}}}$ | 21.38${}_{\pm{\text{0.47}}}$ |  33.06${}_{\pm{\text{0.42}}}$
vMCL | 2.09${}_{\pm{\text{0.18}}}$ |  8.67${}_{\pm{\text{0.32}}}$ |  14.11${}_{\pm{\text{0.28}}}$ |  26.67${}_{\pm{\text{0.46}}}$ |  23.26${}_{\pm{\text{0.67}}}$ |  33.07${}_{\pm{\text{0.53}}}$
AMCL | 1.76${}_{\pm{\textbf{0.10}}}$ | 7.05${}_{\pm{\textbf{0.22}}}$ | 12.86${}_{\pm{\textbf{0.09}}}$ | 24.42${}_{\pm{\textbf{0.26}}}$ | 20.24${}_{\pm{\textbf{0.32}}}$ | 31.65${}_{\pm{\textbf{0.28}}}$
Table 4: The classification error rates (%) on CIFAR-10/100 with various ensemble methods. All methods are used to train an ensemble of 3 networks. EMADNN uses 16 multi-branch networks, and KDCL is trained with KDCL-Naive for the ensemble logits. AMCL is trained with the overlap parameter $K=2$. We report the mean and standard deviation (as subscript) by repeating each test 3 times. The best score is highlighted in bold, and the values in parentheses as subscript represent the relative reductions from the second best result in blue. Base Models | Ensemble | CIFAR-10 | CIFAR-100
---|---|---|---
Method | Oracle | Top-1 | Oracle | Top-1
| IE |  7.60${}_{\pm{\text{0.70}}}$ |  12.12${}_{\pm{\text{0.12}}}$ |  28.87${}_{\pm{\text{0.34}}}$ |  41.79${}_{\pm{\text{0.12}}}$
PreAct | EMADNN |  7.36${}_{\pm{\text{0.64}}}$ |  11.84${}_{\pm{\text{0.10}}}$ |  31.32${}_{\pm{\text{0.59}}}$ |  41.22${}_{\pm{\text{0.83}}}$
ResNet-10 | ONE |  8.76${}_{\pm{\text{0.21}}}$ | 11.30${}_{\pm{\text{0.11}}}$ |  33.88${}_{\pm{\text{0.27}}}$ |  40.76${}_{\pm{\text{0.15}}}$
| KDCL | 7.29${}_{\pm{\text{0.22}}}$ |  11.91${}_{\pm{\text{0.08}}}$ | 28.83${}_{\pm{\text{0.11}}}$ | 39.89${}_{\pm{\text{0.27}}}$
| AMCL |  4.88${}_{\pm{\textbf{0.04}}}$ (-33.1%) |  10.78${}_{\pm{\textbf{0.07}}}$ (-4.6%) |  27.45${}_{\pm{\textbf{0.48}}}$ (-4.8%) |  37.91${}_{\pm{\textbf{0.52}}}$ (-5.0%)
MobileNetV2 | IE | 5.60${}_{\pm{\text{0.51}}}$ |  10.25${}_{\pm{\text{0.20}}}$ |  22.48${}_{\pm{\text{0.45}}}$ |  31.87${}_{\pm{\text{0.28}}}$
EMADNN |  6.84${}_{\pm{\text{0.33}}}$ |  10.53${}_{\pm{\text{0.72}}}$ |  29.68${}_{\pm{\text{0.30}}}$ |  35.91${}_{\pm{\text{0.30}}}$
ONE |  7.31${}_{\pm{\text{0.06}}}$ | 10.19${}_{\pm{\text{0.09}}}$ |  23.48${}_{\pm{\text{0.08}}}$ |  31.89${}_{\pm{\text{0.11}}}$
KDCL |  6.22${}_{\pm{\text{0.16}}}$ |  10.90${}_{\pm{\text{0.08}}}$ | 22.41${}_{\pm{\text{0.12}}}$ | 30.08${}_{\pm{\textbf{0.40}}}$
AMCL |  3.99${}_{\pm{\textbf{0.18}}}$ (-28.8%) |  9.29${}_{\pm{\textbf{0.13}}}$ (-8.8%) |  21.79${}_{\pm{\textbf{0.17}}}$ (-2.8%) |  30.56${}_{\pm{\text{0.17}}}$
Table 5: The foreground-background segmentation error rates (%) on the iCoseg dataset with varying values of the ensemble size $M$. We report the mean by repeating 3 times without the standard deviation due to the space limitation. The best score is highlighted in bold, and the values in parentheses as subscript represent the relative reductions from the second best result in blue. Method | sMCL | CMCL | vMCL | AMCL
---|---|---|---|---
M | Oracle | Top-1 | Oracle | Top-1 | Oracle | Top-1 | Oracle | Top-1
1 | 15.05 | 15.05 | 15.05 | 15.05 | 15.05 | 15.05 | 15.05 | 15.05
2 | 11.71 | 15.12 | 13.33 | 13.31 | 10.73 | 15.49 | 10.48(-2.3%) | 11.53(-13.4%)
3 | 10.65 | 12.67 | 10.86 | 11.20 | 10.27 | 12.26 | 8.68(-15.5%) | 9.60(-14.3%)
4 | 9.52 | 13.25 | 10.42 | 11.19 | 9.30 | 12.88 | 7.97(-14.3%) | 8.83(-21.1%)
5 | 9.37 | 11.82 | 9.75 | 10.58 | 9.44 | 12.06 | 7.66(-18.2%) | 8.61(-18.6%)

Overlapping. Similar to other MCL methods, AMCL can pick the top-$K$
specialized models instead of only one at the training stage. Overlapping
implies that there exist $K$ specialized models for an individual example,
_i.e_., $\sum_{m=1}^{M}v_{j}^{m}=K$. As $K$ increases, the weak generalization
issue can be naturally resolved since data capacity of each model increases.
Hence, all ensemble methods including AMCL tend to show better performance in
the top-1 error as $K$ increases. The detailed results, with varying values of
the overlap parameter $K$, are provided in the supplementary material.

Large-scale CNNs. We now compare the performance on standard, large-scale
convolutional networks on the three public datasets. As presented in Table 3,
AMCL consistently shows consistent improvements compared to the other MCL
methods. Specifically, in the case with the CIFAR-10 dataset, our method with
GoogLeNet shows the most impressive result producing $15.79\%$ and $6.25\%$
relative error reductions from the second best result in terms of the oracle
and the top-1 errors, respectively. Even though both CIFAR-100 and Tiny-
ImageNet are more challenging datasets with significantly larger numbers of
classes compared to CIFAR-10, AMCL shows the superiority with all types of
CNNs. Interestingly, we observe that only our method improves the top-1 error
rate compared to the results of IE with Tiny-ImageNet, which none of the other
methods have achieved.

Comparison with non-MCL ensemble methods. To show the reliability of AMCL, we
compare the performance of the recent ensemble methods, including EMADNN [14],
ONE [15] and KDCL [16], on CIFAR-10/100. EMADNN enforces the diversity by
minimizing the similarity between all pairs of the base models, while ONE and
KDCL apply the concept of knowledge distillation [23] among the base models
during training. The main difference of our method compared to these previous
works is to lead each base model to become a specialist for a certain subset
of the dataset, not on the entire set. In this comparision, we test PreAct
ResNet-10 [24] and MobileNetV2 [25] as the base models. As presented in Table
4, AMCL outperforms by large margins in terms of oracle error rates and shows
impressive top-1 error rates superior to those of all baselines in most cases.

###  4.2 Image segmentation

In this section, we evaluate to verify that AMCL produces higher performance
in the image segmentation task using the iCoseg dataset. To evaluate the image
segmentation task, we define the prediction error rate as the percentage of
incorrectly labeled pixels. The oracle error rate is defined by the lowest
prediction error among the ensemble models. Meanwhile, the top-1 error is
measured by the error rate of the final prediction. For a fair comparison, we
report both the oracle and the top-1 errors with varying ensemble sizes. Table
5 shows that AMCL significantly outperforms all ensemble methods. With 5 model
ensemble, AMCL provides $18.2\%$ and $18.6\%$ relative reductions from the
second best results in the oracle and the top-1 errors, respectively. These
experiments demonstrate the reliability of AMCL in image segmentation beyond
classification. The network architecture and the experiment results with more
elaborated visualization is illustrated in the supplementary material.

##  5 Conclusion

This paper presents an advanced ensemble method to train for an extreme model
specialization which results in diverse solutions of high quality. For this
purpose, we introduce simple but effective techniques: 1) auxiliary class, 2)
memory-based assignment, and 3) feature fusion module. The extensive
experiments with various networks and large datasets show that AMCL results in
better performances in most cases. We believe that our newly proposed approach
can be applied in many related application fields with slight modifications
and provide better results.

## References

  * [1] H. Siqueira, S. Magg, and S. Wermter.  Efficient facial feature learning with wide ensemble-based convolutional neural networks.  AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * [2] W. Zhao, B. Zheng, Q. Lin, and H. Lu.  Enhancing diversity of defocus blur detectors via cross-ensemble network.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 8897–8905, 2019.
  * [3] H. Zheng, Y. Zhang, L. Yang, P. Liang, Z. Zhao, C. Wang, and D. Chen.  A new ensemble learning framework for 3d biomedical image segmentation.  AAAI Conference on Artificial Intelligence (AAAI), pages 5909–5916, 2019.
  * [4] T. Zhu, Y. Sun, X. Ma, and X. Lin.  Hand pose ensemble learning based on grouping features of hand point sets.  International Conference on Computer Vision Workshop (ICCVW), pages 2856–2865, 2019.
  * [5] A. Guzman-Rivera, D. Batra, and P. Kohli.  Multiple choice learning: Learning to produce multiple structured outputs.  Advances in Neural Information Processing Systems (NIPS), pages 1799–1807, 2012.
  * [6] S. Lee, S. P. S Prakash, M. Cogswell, V. Ranjan, D. Crandall, and D. Batra.  Stochastic multiple choice learning for training diverse deep ensembles.  Advances in Neural Information Processing Systems (NIPS), pages 2119–2127, 2016.
  * [7] K. Lee, C. Hwang, K. Park, and J. Shin.  Confident multiple choice learning.  International Conference on Machine Learning (ICML), pages 2014–2023, 2017.
  * [8] K. Tian, Y. Xu, S. Zhou, and J. Guan.  Versatile multiple choice learning and its application to vision computing.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 6349–6357, 2019.
  * [9] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  International Conference on Learning Representation (ICLR), 2015\.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 770–778, 2016.
  * [11] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.  Going deeper with convolutions.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 1–9, 2015.
  * [12] A. Krizhevsky and G. Hinton.  Learning multiple layers of features from tiny images.  2009\.
  * [13] Y. Le and X. Yang.  Tiny imagenet visual recognition challenge.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), 2015\.
  * [14] M.Opitz, H. Possegger, and H. Bischof.  Efficient model averaging for deep neural networks.  Asian Conference on Computer Vision (ACCV), pages 205–220, 2016\.
  * [15] X. Lan, X. Zhu, and S. Gong.  Knowledge distillation by on-the-fly native ensemble.  Advances in Neural Information Processing Systems (NIPS), 2018.
  * [16] Q. Guo, X. Wang, Y. Wu, Z. Yu, D. Liang, X. Hu, and P. Luo.  Online knowledge distillation via collaborative learning.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 11020–11029, 2020.
  * [17] D. Batra, A. Kowdle, D. Parikh, J. Luo, and T. Chen.  icoseg: Interactive co-segmentation with intelligent scribble guidance.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 3169–3176, 2010.
  * [18] J. Mun, K. Lee, J. Shin, and B. Han.  Learning to specialize with knowledge distillation for visual question answering.  Advances in Neural Information Processing Systems (NIPS), page 8092–8102, 2018.
  * [19] J. Park, S. Woo, J. Lee, and I. Kweon.  Bam: Bottleneck attention module.  arXiv preprint arXiv:1807.06514, 2018.
  * [20] S. Woo, J. Park, J. Lee, and I. Kweon.  Cbam: Convolutional block attention module.  European Conference on Computer Vision (ECCV), pages 3–19, 2018\.
  * [21] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson.  How transferable are features in deep neural networks?  Advances in Neural Information Processing Systems (NIPS), pages 3320–3328, 2014.
  * [22] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [24] K. He, X. Zhang, S. Ren, and J. Sun.  Identity mappings in deep residual networks.  European Conference on Computer Vision (ECCV), pages 630–645, 2016\.
  * [25] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  IEEE Conference on Computer Vision Pattern Recognition (CVPR), pages 4510–4520, 2018.


# Specializing Multi-domain NMT via Penalizing Low Mutual Information

Jiyoung Lee† , Hantae Kim‡, Hyunchang Cho‡
Edward Choi†, and Cheonbok Park‡
†KAIST , ‡Papago, NAVER Corp.
{jiyounglee0523<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work done during an
internship at NAVER Corp.

###### Abstract

Multi-domain Neural Machine Translation (NMT) trains a single model with
multiple domains. It is appealing because of its efficacy in handling multiple
domains within one model. An ideal multi-domain NMT should learn distinctive
domain characteristics simultaneously, however, grasping the domain
peculiarity is a non-trivial task. In this paper, we investigate domain-
specific information through the lens of mutual information (MI) and propose a
new objective that penalizes low MI to become higher. Our method achieved the
state-of-the-art performance among the current competitive multi-domain NMT
models. Also, we empirically show our objective promotes low MI to be higher
resulting in domain-specialized multi-domain NMT.

##  1 Introduction

Multi-domain Neural Machine Translation (NMT) (Sajjad et al., 2017; Farajian
et al., 2017) has been an attractive topic due to its efficacy in handling
multiple domains with a single model. Ideally, a multi-domain NMT should
capture both general knowledge (e.g., sentence structure, common words) and
domain-specific knowledge (e.g., domain terminology) unique in each domain.
While the shared knowledge can be easily acquired via sharing parameters
across domains (Kobus et al., 2017), obtaining domain specialized knowledge is
a challenging task. Haddow and Koehn (2012) demonstrate that a model trained
on multiple domains sometimes underperforms the one trained on a single
domain. Pham et al. (2021) shows that separate domain-specific adaptation
modules are not sufficient to fully-gain specialized knowledge.

In this paper, we reinterpret domain specialized knowledge from mutual
information (MI) perspective and propose a method to strengthen it. Given a
source sentence $X$, target sentence $Y$, and corresponding domain $D$, the MI
between $D$ and the translation $Y|X$ (i.e., $\text{MI}(D;Y|X)$) measures the
dependency between the domain and the translated sentence. Here, we assume
that the larger $\text{MI}(D;Y|X)$, the more the translation incorporates
domain knowledge. Low MI is undesirable because it indicates the model is not
sufficiently utilizing domain characteristics in translation. In other words,
low MI can be interpreted as a domain-specific information the model has yet
to learn. For example, as shown in Fig. 1, we found that a model with low MI
translates an IT term ‘computing totals’ to the vague and plain term
‘calculation’. However, once we force the model to have high MI, ‘computing
totals’ is correctly retained in its translation. Thus, maximizing MI promotes
multi-domain NMT to be domain-specialized.

Figure 1: Overview of two models with different MI distributions. The example
sentence is from IT domain. Model A mostly has low MI and Model B has large
MI. For an identical sample, model A outputs a generic term ‘calculation’
while model B properly maintains ‘computing totals’.

Motivated by this idea, we introduce a new method that specializes multi-
domain NMT by penalizing low MI. We first theoretically derive
$\text{MI}(D;Y|X)$, and formulate a new objective that weights more penalty on
subword-tokens with low MI. Our results show that the proposed method improves
the translation quality in all domains. Also, the MI visualization ensures
that our method is effective in maximizing MI. We also observed that our model
performs particularly better on samples with strong domain characteristics.

The main contributions of our paper are as follows:

  * •

We investigate MI in multi-domain NMT and present a new objective that
penalizes low MI to have higher value.

  * •

Extensive experiment results prove that our method truly yields high MI,
resulting in domain-specialized model.

##  2 Related Works

#### Multi-Domain Neural Machine Translation

Multi-Domain NMT focuses on developing a proper usage of domain information to
improve translation. Early studies had two main approaches: injecting source
domain information and adding a domain classifier. For adding source domain
information, Kobus et al. (2017) inserts a source domain label as an
additional tag with input or as a complementary feature. For the second
approach, Britz et al. (2017) trains the sentence embedding to be domain-
specific by updating using the gradient from the domain-classifier.

While previous work leverages domain information by injection or implementing
an auxiliary classifier, we view domain information from MI perspective and
propose a loss that promotes model to explore domain specific knowledge.

#### Information-Theoretic Approaches in NMT

Mutual information in NMT is primarily used either as metrics or a loss
function. For metrics, Bugliarello et al. (2020) proposes cross-mutual
information (XMI) to quantify the difficulty of translating between languages.
Fernandes et al. (2021) modifies XMI to measure the usage of the given context
during translation. For the loss function, Xu et al. (2021) proposes bilingual
mutual information (BMI) which calculates the word mapping diversity, further
applied in NMT training. Zhang et al. (2022) improves the model translation by
maximizing the MI between a target token and its source sentence based on its
context.

Above work only considers general machine translation scenarios. Our work
differs in that we integrate mutual information in multi-domain NMT to learn
domain-specific information. Unlike other methods that require training of an
additional model, our method can calculate MI within a single model which is
more computation-efficient.

##  3 Proposed Method

In this section, we first derive MI in multi-domain NMT. Then, we introduce a
new method that penalizes low MI to have high value resulting in a domain-
specialized model.

###  3.1 Mutual Information in Multi-Domain NMT

Mutual Information (MI) measures a mutual dependency between two random
variables. In multi-domain NMT, the MI between the domain ($D$) and
translation ($Y|X$), expressed as $\text{MI}(D;Y|X)$, represents how much
domain-specific information is contained in the translation.
$\text{MI}(D;Y|X)$ can be written as follows:

$\displaystyle\text{MI}(D;Y|X)$
$\displaystyle=\mathbb{E}_{D,X,Y}\bigg{[}\log\frac{p(Y|X,D)}{p(Y|X)}\bigg{]}.$
(1)

The full derivation can be found in Appendix B. Note that the final form of
$\text{MI}(D;Y|X)$ is a log quotient of the translation considering domain and
translation without domain.

Since the true distributions are unknown, we approximate them with a
parameterized model (Bugliarello et al., 2020; Fernandes et al., 2021), namely
the cross-MI (XMI). Naturally, a generic domain-agnostic model (further
referred to as general and abbreviated as G) output would be the appropriate
approximation of $p(Y|X)$. A domain-adapted (further shortened as DA) model
output would be suitable for $p(Y|X,D)$. Hence, $\text{XMI}(D;Y|X)$ can be
expressed as Eq. (2) with each model output.

$\begin{multlined}\text{XMI}(D;Y|X)=\mathbb{E}_{D,X,Y}\bigg{[}\log\frac{p_{\textit{DA}}(Y|X,D)}{p_{{G}}(Y|X)}\bigg{]}\end{multlined}\text{XMI}(D;Y|X)=\mathbb{E}_{D,X,Y}\bigg{[}\log\frac{p_{\textit{DA}}(Y|X,D)}{p_{{G}}(Y|X)}\bigg{]}$
(2)

###  3.2 MI-based Token Weighted Loss

To calculate XMI, we need outputs from both general and domain-adapted models.
Motivated by the success of adapters (Houlsby et al., 2019) in multi-domain
NMT (Pham et al., 2021), we assign adapters $\phi_{1},\cdots\phi_{N}$ for each
domain ($N$ is the total number of domains) and have an extra adapter
$\phi_{G}$ for general. We will denote the shared parameter (e.g., self-
attention and feed-forward layer) as $\theta$. For a source sentence $x$ from
domain $d$, $x$ passes the model twice, once through the corresponding domain
adapter, $\phi_{d}$, and the other through the general adapter, $\phi_{G}$.
Then, we treat the output probability from domain adapter as $p_{{DA}}$ and
from general adapter as $p_{{G}}$. For the $i^{th}$ target token, $y_{i}$ ,we
calculate XMI as in Eq. (3),

${p(y_{i}|y_{<i},x,\theta,\phi_{d})}-{p(y_{i}|y_{<i},x,\theta,\phi_{G})}$ (3)

, where $y_{<i}$ is the target subword-tokens up to, but excluding $y_{i}$.
For simplicity, we will denote Eq. (3) as $\text{XMI}(i)$. Low $\text{XMI}(i)$
means that our domain adapted model is not thoroughly utilizing domain
information during translation. Therefore, we weight more on the tokens with
low $\text{XMI}(i)$, resulting in minimizing Eq. (4),

$\mathcal{L_{\text{MI}}}=\sum_{i=0}^{n_{T}}(1-\text{XMI}(i))\cdot(1-p(y_{i}|y_{<i},x,\theta,\phi_{d}))$
(4)

, where $n_{T}$ is the number of subword-tokens in the target sentence.

The final loss of our method is in Eq. (7), where $\lambda_{1}$ and
$\lambda_{2}$ are hyperparameters.

$\displaystyle\mathcal{L_{\text{DA}}}$
$\displaystyle=-\sum_{i=0}^{n_{T}}\log(p(y_{i}|y_{<i},x,\theta,\phi_{d}))$ (5)
$\displaystyle\mathcal{L_{\text{G}}}$
$\displaystyle=-\sum_{i=0}^{n_{T}}\log(p(y_{i}|y_{<i},x,\theta,\phi_{G}))$ (6)
$\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L_{\text{DA}}}+\lambda_{1}\mathcal{L_{\text{G}}}+\lambda_{2}\mathcal{L_{\text{MI}}}$
(7)

##  4 Experiments

###  4.1 Experiment Setting

#### Dataset.

We leverage the preprocessed dataset released by Aharoni and Goldberg (2020)
consisting of five domains (IT, Koran, Law, Medical, Subtitles) available in
OPUS Tiedemann (2012); Aulamo and Tiedemann (2019). More details on the
dataset and preprocessing are described in Appendix A.

#### Baseline.

We compare our method with the following baseline models: (1) Mixed trains a
model on all domains with uniform distribution, (2) Domain-Tag (Kobus et al.,
2017) inserts domain information as an additional token in the input, (3)
Multitask Learning (MTL) (Britz et al., 2017) trains a domain classifier
simultaneously and encourage the sentence embedding to encompass its domain
characteristics, (4) Adversarial Learning (AdvL) (Britz et al., 2017) makes
the the sentence embedding to be domain-agnostic by flipping the gradient from
the domain classifier before the back-propagation, (5) Word-Level Domain
Context Discrimination (WDC) (Zeng et al., 2018) integrates two sentence
embedding which are trained by MTL and AdvL respectively, (6) Word-Adaptive
Domain Mixing111We conducted experiments using publicly available code. (Jiang
et al., 2020), has domain-specific attention heads and the final
representation is the combination of each head output based on the predicted
domain proportion, and (7) Domain-Adapter (Pham et al., 2021) has separate
domain adapters (Houlsby et al., 2019) and a source sentence passes through
its domain adapters. This can be regarded as our model without general adapter
and trained with $\mathcal{L}_{\text{DA}}$.

###  4.2 Main Results

| IT | Koran | Law | Medical | Subtitles | Average
---|---|---|---|---|---|---
Mixed | 43.87$\pm$0.505 | 20.31$\pm$0.371 | 58.33$\pm$0.474 | 55.19$\pm$0.737 | 30.36$\pm$0.424 | 41.61
62.00$\pm$0.403 | 41.75$\pm$0.343 | 73.41$\pm$0.303 | 69.14$\pm$0.346 | 45.73$\pm$0.424 | 58.40
Domain-Tag | 44.29$\pm$0.142 | 20.44$\pm$0.236 | 58.47$\pm$0.275 | 55.39$\pm$0.288 | 30.61$\pm$0.220 | 41.84
62.30$\pm$0.111 | 41.75$\pm$0.203 | 73.56$\pm$0.190 | 69.28$\pm$0.160 | 45.99$\pm$0.268 | 58.58
MTL | 44.00$\pm$0.298 | 20.40$\pm$0.198 | 58.27$\pm$0.327 | 55.24$\pm$0.564 | 30.52$\pm$0.478 | 41.69
62.11$\pm$0.169 | 41.78$\pm$0.174 | 73.42$\pm$0.197 | 69.16$\pm$0.235 | 45.87$\pm$0.316 | 58.47
AdvL | 43.86$\pm$0.167 | 20.33$\pm$0.275 | 58.40$\pm$0.195 | 55.56$\pm$0.245 | 30.43$\pm$0.367 | 41.71
61.91$\pm$0.099 | 41.79$\pm$0.206 | 73.42$\pm$0.193 | 69.30$\pm$0.184 | 45.80$\pm$0.208 | 58.44
WDC | 44.44$\pm$0.193 | 20.75$\pm$0.212 | 58.49$\pm$0.193 | 55.43$\pm$0.308 | 30.52$\pm$0.242 | 41.93
62.27$\pm$0.175 | 42.05$\pm$0.198 | 73.58$\pm$0.182 | 69.20$\pm$0.203 | 45.87$\pm$0.125 | 58.59
Word-Adaptive Domain Mixing | 41.88$\pm$0.240 | 19.84$\pm$0.297 | 55.82$\pm$0.594 | 52.88$\pm$0.785 | 30.39$\pm$0.141 | 40.16
60.37$\pm$0.113 | 41.02$\pm$0.212 | 71.79$\pm$0.290 | 67.62$\pm$0.396 | 45.63$\pm$0.113 | 57.29
Domain-Adapter | 44.50$\pm$0.342 | 20.37$\pm$0.193 | 58.22$\pm$0.169 | 56.00$\pm$0.243 | 31.02$\pm$0.334 | 42.02
62.30$\pm$0.248 | 41.65$\pm$0.160 | 73.40$\pm$0.066 | 69.54$\pm$0.149 | 46.30$\pm$0.306 | 58.64
Ours (w/o $\mathcal{L_{\text{MI}}}$) | 44.65$\pm$0.318 | 20.43$\pm$0.286 | 58.21$\pm$0.692 | 55.38$\pm$0.684 | 30.82$\pm$0.498 | 41.90
62.49$\pm$0.221 | 41.77$\pm$0.262 | 73.40$\pm$0.416 | 69.28$\pm$0.377 | 46.16$\pm$0.414 | 58.62
Ours |  |  45.89$\pm$0.215
---
(+1.39)
|  20.80$\pm$0.298
---
(+0.43)
|  59.22$\pm$0.306
---
(+1.00)
|  56.34$\pm$0.238
---
(+0.34)
|  31.56$\pm$0.218
---
(+0.54)
| 42.76
---
(+0.74)
|  63.19$\pm$0.204
---
(+0.89)
|  42.05$\pm$0.274
---
(+0.39)
|  74.02$\pm$0.219
---
(+0.62)
|  69.94$\pm$0.238
---
(+0.40)
|  46.46$\pm$0.261
---
(+0.16)
| 59.13
---
(+0.49)

Table 1:  Average and standard deviation of BLEU (upper line) and chrF (bottom
line) from five random seed experiments. Bold indicates the best performance
within a domain. Our model outperforms all baselines with significant margins
($p<0.05$).

Table 1 presents sacreBLEU Post (2018) and chrF (Popović, 2015) score from
each model in all domains. For a fair comparison, we matched the number of
parameters for all models. Baseline results following its original
implementation with different parameter size are provided in Appendix C.
Interestingly, Mixed performs on par with Domain-Tag and outperforms Word-
Adaptive Domain Mixing, suggesting that not all multi-domain NMT methods are
effective. Although adapter-based models (i.e., Ours (w/o
$\mathcal{L_{\text{MI}}}$) and Domain-Adapter) outperform Mixed, the
performance increase is still marginal. Our model has gained 1.15 BLEU
improvement over Mixed. It also outperforms all baselines with statistically
significant difference.

As an ablation study of our MI objective, we conduct experiments without
$\mathcal{L_{\text{MI}}}$ to prove its effectiveness. The result confirms that
$\mathcal{L}_{\text{MI}}$ encouraged the model to learn domain specific
knowledge leading to refined translation.

###  4.3 Mutual Information Distribution

We visualize $\text{XMI}(i)$ in Eq. (3) to verify that our proposed loss
penalizes low XMI. Figure 2 is the histogram of $\text{XMI}(i)$ from the test
samples in Law. Other domain distributions are in Appendix D. We use Domain-
Adapter for comparison since it performs the best among the baselines. For
$p_{G}$, we use the output probability of Mixed for both cases. From the
distributions, our method indeed penalizes low XMI and encourages the model to
have high XMI in all domains.

Figure 2: XMI distribution in Law. X-axis is XMI and Y-axis is the density.
Green is Domain-Adapter and blue is our model. Our model has more high XMI
values.

###  4.4 Translation Performance for Domain Specialized Sentences

| -Q1 | Q1-Q2 | Q2-Q3 | Q3-Q4 | Average
---|---|---|---|---|---
IT | 2.26 | 1.14 | 1.12 | 1.27 | 1.39
Koran | 0.97 | 0.65 | 0.49 | 0.41 | 0.43
Law | 0.75 | 0.73 | 1.41 | 1.17 | 1.00
Medical | 0.25 | 0.08 | 0.60 | 0.40 | 0.34
Subtitles | 0.36 | 0.83 | 0.83 | 2.10 | 0.54

Table 2:  BLEU improvements in each quartiles. Q1, Q2, Q3 and Q4 represents
25%, 50%, 75% 100% respectively. The higher the quartile, the more domain
specific the samples.

Since the ultimate goal is to specialize multi-domain NMT, we calculate BLEU
score improvement according to the domain specificity. We extract top 1% TF-
IDF words in train source sentences in each domain (examples are in Appendix
E) and consider them as domain-specific keywords. We assume that the more
these keywords are included in the source sentence, the more domain
specialized the sample is. We divide the test set into quartiles based on the
number of the keywords the source sentence contains.

Table 2 reports BLEU score improvement compared to Domain-Adapter in each
quartile along with averaged performance increases. In Law, Medical and
Subtitles, BLEU score improvement increases as quartile gets higher.
Furthermore, the improvements in Q2-Q3 and Q3-Q4 are larger than the averaged
improvement score (i.e., Average column). However, in IT and Koran, -Q1 has
the largest performance increases. We conjecture the reason is that in both
domains, the number of top TF-IDF words include in higher quartiles is fewer
than the other domains. This weak distinction among quartiles in IT and Koran
can be the root cause of marginal performance improvement. Details on number
of captured keywords are in Appendix E.

###  4.5 Experiment Results on Korean-English

| Finance | Ordinance | Tech | Average
---|---|---|---|---
Mixed | 52.50$\pm$0.220 | 56.65$\pm$0.100 | 66.00$\pm$0.242 | 58.38
72.64$\pm$0.105 | 75.36$\pm$0.091 | 81.60$\pm$0.121 | 76.53
Domain-Tag | 52.71$\pm$0.231 | 56.60$\pm$0.115 | 66.03$\pm$0.360 | 58.45
72.77$\pm$0.175 | 75.38$\pm$0.058 | 81.64$\pm$0.185 | 76.60
WDC | 52.75$\pm$0.136 | 56.56$\pm$0.124 | 65.93$\pm$0.214 | 58.41
72.78$\pm$0.135 | 75.34$\pm$0.053 | 81.53$\pm$0.099 | 76.55
Domain-Adapter | 53.13$\pm$0.186 | 56.97$\pm$0.129 | 66.25$\pm$0.103 | 58.78
72.98$\pm$0.170 | 75.48$\pm$0.066 | 81.76$\pm$0.079 | 76.74
Ours |  |  53.87$\pm$0.188
---
(+0.74)
|  57.47$\pm$0.086
---
(+0.50)
|  66.66$\pm$0.191
---
(+0.41)
| 59.33
---
(+0.55)
|  73.41$\pm$0.162
---
(+0.43)
|  75.81$\pm$0.033
---
(+0.33)
|  81.99$\pm$0153
---
(+0.23)
| 77.07
---
(+0.33)

Table 3:  Average and standard deviation of BLEU (upper line) and chrF (bottom
line) from five random seeds on Ko-En dataset. Bold indicates the best
performance within a domain.

To verify the effectiveness of our proposed method in different language, we
additionally conducted experiment on Korean-English dataset which has
approximately 1M samples with three domains: Finance, Ordinance and Tech. The
dataset is obtained from AIhub222https://aihub.or.kr/ which is publicly
available. Model configuration is identical with the main experiment on OPUS.
More experimental details are in Appendix A.

Table 3 demonstrates the results from the major baselines and our model, where
we select top-4 baselines in the experiment on OPUS (Table 1). Our model
achieves the best performance in all three domains, outperforming Domain-
Adpater by 0.55 BLEU score on average. This result confirms that our proposed
method can be further extended to other languages.

###  4.6 Samples with MI Visualization

Figure 3: Example visualizations with MI values from IT and Medical. The more
intense the red, the more higher MI value.

Figure 3 demonstrates test set outputs with MI values generated by our model.
Color intensity is correlated with MI value; the more intense the red, the
more higher MI value. Note that the model has high MI especially when generate
domain-specific words (e.g., ‘password’ in IT and ‘omalizumab’ in Medical).
This result is analogous to our motivation in that high MI value encourages
model to translate domain peculiar terms. More samples are provided in
Appendix F.

###  4.7 Computation Cost for Training

| Domain-Adapter | Ours
---|---|---
Number of Iterations ($\downarrow$) | 51.7K | 48.5K
Peak Memory (GB) ($\downarrow$) | 26.79 | 27.09
Words per Second ($\uparrow$) | 22.9K | 22K
Updates per Second ($\uparrow$) | 0.4 | 0.38

Table 4:  Comparison of training computation cost between Domain-Adapter and
Ours. The values are averaged across five seed experiments.

We compare computation cost between Domain-Adapter and our proposed model.
Domain-Adapter was chosen because it shares the same model architecture with
ours. Table 4 provides four computation cost during training: Number of
Iterations until converge, Peak Memory, Words per Second, and Updates per
Second. Our model requires fewer number of iterations needed to be trained
(3.2K difference on average across five seeds). Our model has slightly higher
peak memory (0.3GB, 0.94%) than Domain-Adapter, however, we believe this is
acceptable when considering the performance improvement. Furthermore, there
was not much difference in words per second and updates per second during
training.

##  5 Conclusion

We build a specialized multi-domain NMT by adding MI-based loss. We
reinterpret domain-specific knowledge from MI perspective and promote a model
to explore domain knowledge by penalizing low MI. Our results prove that the
proposed method is effective in increasing overall MI.

## Limitations

Although many previous multi-domain NMT studies regard the source of the given
sentence as its domain, equating domain and corpora is a naive approach and
can partially represent the data. (Aharoni and Goldberg, 2020) For instance,
some sentences may incorporate multiple domain characteristics or can be
better translated under different domain other than its source domain. (Currey
et al., 2020; Pham et al., 2021) This problem is not limited to our work but
is applicable to other previous multi-domain NMT studies. Establishing a more
proper definition of domain is a future work and a critical challenge in
multi-domain NMT.

## Acknowledgement

The authors would like to thank all members in Papago Team, NAVER Corporation
for their valuable comments. Also, we sincerely thank Jin-Hwa Kim at NAVER AI
Lab for the insightful feedback. This work was partly supported by Institute
of Information & Communications Technology Planning & Evaluation (IITP) grant
(No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), and
National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945)
funded by the Korea government (MSIT).

## References

  * Agarap (2018) Abien Fred Agarap. 2018.  Deep learning using rectified linear units (relu).  _arXiv preprint arXiv:1803.08375_.
  * Aharoni and Goldberg (2020) Roee Aharoni and Yoav Goldberg. 2020.  Unsupervised domain clusters in pretrained language models.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 7747–7763.
  * Aulamo and Tiedemann (2019) Mikko Aulamo and Jörg Tiedemann. 2019.  The OPUS resource repository: An open package for creating parallel corpora and machine translation services.  In _Proceedings of the 22nd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 389–394.
  * Bapna et al. (2019) Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. 2019.  Simple, scalable adaptation for neural machine translation.  _arXiv preprint arXiv:1909.08478_.
  * Britz et al. (2017) Denny Britz, Quoc Le, and Reid Pryzant. 2017.  Effective domain mixing for neural machine translation.  In _Proceedings of the Second Conference on Machine Translation (WMT)_ , pages 118–126.
  * Bugliarello et al. (2020) Emanuele Bugliarello, Sabrina J. Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki Okazaki. 2020.  It’s easier to translate out of English than into it: Measuring neural translation difficulty by cross-mutual information.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 1640–1649.
  * Currey et al. (2020) Anna Currey, Prashant Mathur, and Georgiana Dinu. 2020.  Distilling multiple domains for neural machine translation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4500–4511.
  * Farajian et al. (2017) M Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. 2017.  Multi-domain neural machine translation through unsupervised adaptation.  In _Proceedings of the Second Conference on Machine Translation (WMT)_ , pages 127–137.
  * Fernandes et al. (2021) Patrick Fernandes, Kayo Yin, Graham Neubig, and André F. T. Martins. 2021.  Measuring and increasing context usage in context-aware machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL)_ , pages 6467–6478.
  * Haddow and Koehn (2012) Barry Haddow and Philipp Koehn. 2012.  Analysing the effect of out-of-domain data on smt systems.  In _Proceedings of the Seventh Workshop on Statistical Machine Translation (SMT)_ , pages 422–432.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for nlp.  In _International Conference on Machine Learning (ICML)_ , pages 2790–2799.
  * Jiang et al. (2020) Haoming Jiang, Chen Liang, Chong Wang, and Tuo Zhao. 2020.  Multi-domain neural machine translation with word-level adaptive layer-wise domain mixing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 1823–1834.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kobus et al. (2017) Catherine Kobus, Josep Crego, and Jean Senellart. 2017.  Domain control for neural machine translation.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, (RANLP)_ , pages 372–378.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007\.  Moses: Open source toolkit for statistical machine translation.  In _Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions (ACL)_ , pages 177–180.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) (NAACL)_ , pages 48–53.
  * Pham et al. (2021) MinhQuang Pham, Josep Maria Crego, and François Yvon. 2021.  Revisiting multi-domain machine translation.  _Transactions of the Association for Computational Linguistics (TACL)_ , pages 17–35.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation (SMT)_ , pages 392–395.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers (WMT)_ , pages 186–191.
  * Sajjad et al. (2017) Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Yonatan Belinkov, and Stephan Vogel. 2017\.  Neural machine translation training in a multi-domain scenario.  _arXiv preprint arXiv:1708.08712_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 1715–1725.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC)_ , pages 2214–2218.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NIPS)_.
  * Xu et al. (2021) Yangyifan Xu, Yijin Liu, Fandong Meng, Jiajun Zhang, Jinan Xu, and Jie Zhou. 2021\.  Bilingual mutual information based adaptive training for neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL)_ , pages 511–516.
  * Zeng et al. (2018) Jiali Zeng, Jinsong Su, Huating Wen, Yang Liu, Jun Xie, Yongjing Yin, and Jianqiang Zhao. 2018.  Multi-domain neural machine translation with word-level domain context discrimination.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 447–457.
  * Zhang et al. (2022) Songming Zhang, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jian Liu, and Jie Zhou. 2022.  Conditional bilingual mutual information based adaptive training for neural machine translation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 2377–2389.

##  Appendix A Experiment Settings

###  A.1 Dataset

For preprocessing, we conducted tokenization and normalize-punctuation by
using Moses (Koehn et al., 2007) pipeline. We additionally eliminated samples
with (i) sequences shorter than one subword-token, (ii) sequences longer than
250 subword-tokens, (iii) severe length imbalance between the language pair
(top, bottom 5% for each domain) for both De-En and Ko-En. Table 5 and 6 show
the final number of samples.

De-En | Train | Dev | Test
---|---|---|---
IT | 211,374 | 1,888 | 2,000
Koran | 16,952 | 1,872 | 2,000
Law | 434,555 | 1,861 | 2,000
Medical | 233,167 | 1,873 | 2,000
Subtitles | 470,611 | 1,899 | 2,000
Table 5:  Number of samples in De-En Ko-En | Train | Dev | Test
---|---|---|---
Finance | 156,569 | 9,510 | 5,000
Ordinance | 79,802 | 9,335 | 5,000
Tech | 711,885 | 9,251 | 5,000
Table 6:  Number of samples in Ko-En

###  A.2 Experiment Details

For De-En, we use a joint BPE vocabulary (Sennrich et al., 2016) learned with
32k merge operations. For Ko-En, we train BPE vocabulary with 32k size
separately for each language since Korean and English do not share characters.
Remainig experiment settings are identical for both language pairs. Our
experiments are conducted under open-source
fairseq333https://github.com/facebookresearch/fairseq (Ott et al., 2019)
framework. We built upon Transformer model (Vaswani et al., 2017) which has 6
encoder and decoder layers with embedding dimension of 512, feed-forward
dimension of 2048, and attention heads of 8. Parameters of encoder embedding,
decoder embedding and decoder last layer are shared. We also utilize the same
sinusoidal positional embedding following the original work. We fix dropout to
0.1 and used ReLU (Agarap, 2018) as an activation function. Following Bapna et
al. (2019), the domain and general adapters
($\phi_{1},\cdots\phi_{N},\phi_{G}$) are inserted after feed-forward layer
following the multi-head attention. Note that our training differs in that we
jointly train all parameters from scratch including adapters on all domains.

The bottleneck size of the adapters is 256. Adapters are initialized from
zero-mean Gaussian with standard deviation $10^{-2}$ which proven to be most
effective in the proposed work. We searched the best combination of
$\lambda_{1}$ and $\lambda_{2}$ by grid search ranging from 0.5 to 1.0. Then,
we set $\lambda_{1}$ and $\lambda_{2}$ both to 1. All experiments are trained
with label-smoothing cross-entropy loss with smoothing parameter of 0.1. All
experiments are conducted using 8 NVIDIA V100 GPU.

In all experiments, a model is trained until early stopping with patience of
10 based on BLEU. We use Adam (Kingma and Ba, 2014) optimizer with an initial
learning rate of 5$\cdot e^{-4}$, where the learning rate is searced within
the range of 0.001 to 0.0001. The tokens per batch is 8192 in all experiments.
We compute sacreBLEU score 444sacreBLEU signature: BLEU+c.mixed+l.de-en+#.1+s.
exp+tok.13a+v.2.0.0 from outputs using beam search with a beam size of 5.

###  A.3 Baselines

In this section, we provide a detailed explanation on each baseline. Mixed and
Domain-Tag employ identical model architecture with our model excluding
adapters. Mixed does not utilize domain information and treat all samples are
from identical distribution. On the other hand, Domain-Tag distinguishes
domain by adding domain tag in front of the input sentence. We enlarged Mixed
and Domain-Tag to Mixed-Big and Domain-Tag-big by increasing encoder and
decoder embedding dimension to 608.

For Word-Adaptive Domain Mixing, we borrowed publicly available code and
applied on our dataset. We applied word-adaptive training in both encoder and
decoder because it had the best performance in the original paper. Other
configurations are the same with ours.

Domain-Adapter has the domain-specific adapters $\phi_{1},\cdots,\phi_{N}$,
and the input sentence passes through only its domain adapter. Note that there
are two major differences between Domain-Adapter and ours. First, Domain-
Adapter does not need general adapter $\phi_{G}$ since it does not calculate
$p_{G}(Y|X)$. Second, a source sentence only passes through the model once
only through its domain adapter $\phi_{d}$. Similar to our method, all the
parameters including adapters are jointly trained from scratch.

##  Appendix B Full Derivation of Domain-Aware Mutual Information

Below is the full derivation of Domain-Aware Mutual Information.

$\displaystyle MI(D;Y|X)$
$\displaystyle=\mathbb{E}_{D,X,Y}\bigg{[}\log\frac{p(D,Y|X)}{p(D|X)\cdot
p(Y|X)}\bigg{]}$
$\displaystyle=\mathbb{E}_{D,X,Y}\bigg{[}\log\frac{p(D|Y,X)\cdot\cancel{p(Y|X)}}{p(D|X)\cdot\cancel{p(Y|X)}}\bigg{]}$
$\displaystyle=\mathbb{E}_{D,X,Y}\bigg{[}\log\frac{p(X,Y,D)\cdot
p(X)}{p(X,Y)\cdot p(X,D)}\bigg{]}$
$\displaystyle=\mathbb{E}_{D,X,Y}\bigg{[}\log\frac{p(Y|X,D)}{p(Y|X)}\bigg{]}$

The proof from the first to the second line is provided below.

$\displaystyle P(X,Y,D)$ $\displaystyle=P(D|Y,X)\cdot P(Y|X)\cdot P(X)$
$\displaystyle\Rightarrow P(D,Y|X)$ $\displaystyle=P(D|Y,X)\cdot P(Y|X)$

##  Appendix C Additional Baseline Results

|  | # of
---
Parameters
IT | Koran | Law | Medical | Subtitles | Average
Mixed-Small | 60M | 43.64$\pm$0.253 | 20.74$\pm$0.155 | 57.47$\pm$0.376 | 54.88$\pm$0.553 | 30.58$\pm$0.273 | 41.46
61.96$\pm$0.204 | 42.01$\pm$0.159 | 72.95$\pm$0.275 | 68.98$\pm$0.362 | 46.06$\pm$0.293 | 58.39
Mixed | 76M | 43.87$\pm$0.505 | 20.31$\pm$0.371 | 58.33$\pm$0.474 | 55.19$\pm$0.737 | 30.36$\pm$0.424 | 41.61
62.00$\pm$0.403 | 41.75$\pm$0.343 | 73.41$\pm$0.303 | 69.14$\pm$0.346 | 45.73$\pm$0.424 | 58.40
Domain-Tag-Small | 60M | 44.00$\pm$0.409 | 20.39$\pm$0.251 | 57.80$\pm$0.206 | 54.91$\pm$0.197 | 31.10$\pm$0.316 | 41.64
62.09$\pm$0.242 | 41.71$\pm$0.141 | 73.14$\pm$0.183 | 68.97$\pm$0.056 | 46.34$\pm$0.208 | 58.45
Domain-Tag | 76M | 44.29$\pm$0.142 | 20.44$\pm$0.236 | 58.47$\pm$0.275 | 55.39$\pm$0.288 | 30.61$\pm$0.220 | 41.84
62.30$\pm$0.111 | 41.75$\pm$0.203 | 73.56$\pm$0.190 | 69.28$\pm$0.160 | 45.99$\pm$0.268 | 58.58
MTL | 76M | 44.00$\pm$0.298 | 20.40$\pm$0.198 | 58.27$\pm$0.327 | 55.24$\pm$0.564 | 30.52$\pm$0.478 | 41.69
62.11$\pm$0.169 | 41.78$\pm$0.174 | 73.42$\pm$0.197 | 69.16$\pm$0.235 | 45.87$\pm$0.316 | 58.47
AdvL | 76M | 43.86$\pm$0.167 | 20.33$\pm$0.275 | 58.40$\pm$0.195 | 55.56$\pm$0.245 | 30.43$\pm$0.367 | 41.71
61.91$\pm$0.099 | 41.79$\pm$0.206 | 73.42$\pm$0.193 | 69.30$\pm$0.184 | 45.80$\pm$0.208 | 58.44
WDC | 76M | 44.44$\pm$0.193 | 20.75$\pm$0.212 | 58.49$\pm$0.193 | 55.43$\pm$0.308 | 30.52$\pm$0.242 | 41.93
62.27$\pm$0.175 | 42.05$\pm$0.198 | 73.58$\pm$0.182 | 69.20$\pm$0.203 | 45.87$\pm$0.125 | 58.59
Word-Adaptive Domain Mixing Big | 218M | 44.08$\pm$0.561 | 20.34$\pm$0.257 | 59.63$\pm$0.308 | 56.81$\pm$0.386 | 29.34$\pm$0.488 | 42.04
61.93$\pm$0.273 | 41.23$\pm$0.334 | 74.26$\pm$0.196 | 69.96$\pm$0.245 | 44.62$\pm$0.460 | 58.40
Word-Adaptive Domain Mixing | 76M | 41.88$\pm$0.240 | 19.84$\pm$0.297 | 55.82$\pm$0.594 | 52.88$\pm$0.785 | 30.39$\pm$0.141 | 40.16
60.37$\pm$0.113 | 41.02$\pm$0.212 | 71.79$\pm$0.290 | 67.62$\pm$0.396 | 45.63$\pm$0.113 | 57.29
Domain-Adapter | 76M | 44.50$\pm$0.342 | 20.37$\pm$0.193 | 58.22$\pm$0.169 | 56.00$\pm$0.243 | 31.02$\pm$0.334 | 42.02
62.30$\pm$0.248 | 41.65$\pm$0.160 | 73.40$\pm$0.066 | 69.54$\pm$0.149 | 46.30$\pm$0.306 | 58.64
Ours (w/o $\mathcal{L_{\text{MI}}}$) | 76M | 44.65$\pm$0.318 | 20.43$\pm$0.286 | 58.21$\pm$0.692 | 55.38$\pm$0.684 | 30.82$\pm$0.498 | 41.90
62.49$\pm$0.221 | 41.77$\pm$0.262 | 73.40$\pm$0.416 | 69.28$\pm$0.377 | 46.16$\pm$0.414 | 58.62
Ours | 76M |  |  45.89$\pm$0.215
---
(+1.39)
|  20.80$\pm$0.298
---
(+0.43)
|  59.22$\pm$0.306
---
(+1.00)
|  56.34$\pm$0.238
---
(+0.34)
|  31.56$\pm$0.218
---
(+0.54)
| 42.76
---
(+0.74)
|  63.19$\pm$0.204
---
(+0.89)
|  42.05$\pm$0.274
---
(+0.39)
|  74.02$\pm$0.219
---
(+0.62)
|  69.94$\pm$0.238
---
(+0.40)
|  46.46$\pm$0.261
---
(+0.16)
| 59.13
---
(+0.49)

Table 7:  Average and standard deviation of BLEU (upper line) and chrF (bottom
line) from baselines with different parameter size and our model.

Table 7 shows sacreBLEU Post (2018) and chrF (Popović, 2015) score from
baseline models (Mixed-Small, Domain-Tag-Small, Word-Adaptive Domain Mixing
Big) following its original implementation with different number of
parameters. Note that Mixed and Domain-Tag underperform the models in the main
experiment, and Word-Adaptive Domain Mixing becomes effective when enlarge
model size.

##  Appendix D MI Histogram

Figure 4: XMI distributions from all domains. X-axis is XMI and Y-axis is the
density. Adpater is colored in green and our model is in blue. Our XMI
distribution has more higher values than baseline.

XMI histograms from all doamins are in Fig. 4. Adpater (baseline) is colored
in green and our model is in blue. From the plot, we can verify that in all
domains, our model outputs higher XMI compared to the baseline.

##  Appendix E Details on Translation Performance for Domain Specialized
Experiment

|  Examples of TF-IDF Keywords
---|---
IT |  übertragungsrate (transfer rate), zwischensumme (temporally save), YouTube, übertragungsfortschritt (transfer progress), speichergröße (memory size), yahoomail, zusammensetzungswerkzeug (composition tool), übersichtsmodus (overview mode), webserver, zwischengespeichert (cached), übersetzungsprogramm (translation game), zwischenablagename (clipboard name)
Koran |  übertreter (transgressor), zwingherr (tyrant), widmest (dedicate), unterwürfig (submissive), unterworfen (subjected), städte (cities), sterben (die), schutzherr (patron), religion, muslime (muslims)
Law |  überwachungszollstelle (supervising customs office), änderungsverfahren (change procedure), zustellungsmängel (delivery defects), wirtschaftsjahre (fiscal years), überstunden (overtime), zuschusssatz (subsidy rate), widerklänge (echoes), übernahmeprotokoll (takeover protocol), zulassungsvoraussetzungen (admission requirements), verwaltungskommission (administrative commission), tarife (rates)
Medical |  überlebenswahrscheinlichkeit (probability of survival), verletzung (injury), tagesgesamtdosis (total daily dose), überempfindlichkeitsreaktionen (hypersensitivity reactions), zäpfchen (suppository), tremor, zytotoxisch (cytotoxic), urin (urine), Schatzungen (estimates), wirkstoffmatrix (active ingredient matrix), vorsichtsmaßnahmen (precautions)
Subtitles |  übungen (exercises) , wähle (choose), übersehen (overlook), öffentlichkeitsarbeit (public relation), äußern (to express), ärger (trouble), ältester (oldest), zähflüssig (viscous), zwischenmahlzeiten (snacks), wäsche (laundry), werbepause (commercial break)
Table 8:  Examples of TF-IDF extracted words of the source language (i.e.,
German). We randomly sampled 11 words from each domain among the top 1%
keywords. We also write its meaning in english words in the bracket.

Examples of extracted TF-IDF keywords are in Table 8. We removed stop words
and conducted lemmatization before extracting keywords. As expected, chosen
words are correlated to its domain are chosen.

| IT | Koran | Law | Medical | Subtitles
---|---|---|---|---|---
-Q1 | 0 | 0 | 1.91 | 0.96 | 1.70
Q1-Q2 | 1 | 1 | 5.07 | 4.05 | 3.57
Q2-Q3 | 2 | 2 | 8.36 | 6.83 | 5.42
Q3-Q4 | 4.10 | 3.42 | 13.21 | 11.50 | 8.11

Table 9:  Averaged number of captured top TF-IDF keywords in each quartile.

Averaged number of captured keywords in each quartile is presented in Table 9.
Compared to Law, Medical and Subtitles where a clear distinction among
quartiles by the averaged number of keywords exists, IT and Koran have a
minimal change indicating a weak distinction.

##  Appendix F Samples with MI Visualization

Figure 5 provides more visualizations of test examples with MI values from IT,
Law, and Medical. Color intensity is correlated with MI value; the more
intense the red, the more higher MI value. From the result, domain-specific
words (e.g., ‘account’ in IT, ‘Regulation’ in Law, ‘pharmacokinetic’ in
Medical) are translated with high MI values.

Figure 5: Visualization of examples with MI values from IT, Law, and Medical.
Color intensity is correlated with MI value; the more intense the red, the
more higher MI value.