# Disentangled Latent Spaces Facilitate Data-Driven Auxiliary Learning

Geri Skenderi 1, Luigi Capogrosso1, Andrea Toaiari1,
Matteo Denitto2, Franco Fummi 1, Simone Melzi 3, Marco Cristani1

1 University of Verona
2 HUMATICS - SYS-DAT Group
3 University of Milano-Bicocca Corresponding author<EMAIL_ADDRESS>
###### Abstract

In deep learning, auxiliary objectives are often used to facilitate learning
in situations where data is scarce, or the principal task is extremely
complex. This idea is primarily inspired by the improved generalization
capability induced by solving multiple tasks simultaneously, which leads to a
more robust shared representation. Nevertheless, finding optimal auxiliary
tasks that give rise to the desired improvement is a crucial problem that
often requires hand-crafted solutions or expensive meta-learning approaches.
In this paper, we propose a novel framework, dubbed _Detaux_ , whereby a
weakly supervised disentanglement procedure is used to discover new unrelated
classification tasks and the associated labels that can be exploited with the
principal task in any Multi-Task Learning (MTL) model. The disentanglement
procedure works at a representation level, isolating a subspace related to the
principal task, plus an arbitrary number of orthogonal subspaces. In the most
disentangled subspaces, through a clustering procedure, we generate the
additional classification tasks, and the associated labels become their
representatives. Subsequently, the original data, the labels associated with
the principal task, and the newly discovered ones can be fed into any MTL
framework. Extensive validation on both synthetic and real data, along with
various ablation studies, demonstrate promising results, revealing the
potential in what has been, so far, an unexplored connection between learning
disentangled representations and MTL. The code will be made publicly available
upon acceptance.

##  1 Introduction

Human learning is often considered to be a combination of processes (_e.g_.,
high-level acquired skills and evolutionary encoded physical perception) that
are used together and can be transferred from one problem to another. Inspired
by this, Multi-Task Learning (MTL) (Caruana, 1997) represents a machine
learning paradigm where multiple tasks are learned together to improve the
generalization ability of a model by using shared knowledge that derives from
considering different aspects of the input. Specifically, this is achieved by
jointly optimizing the model’s parameters across different tasks, allowing the
model to learn task-specific and task-shared representations simultaneously.
As a result, MTL can lead to better generalization, improved efficiency at
inference time, and enhanced performance on individual tasks by exploiting
their underlying relationships.

A specific form of this learning approach referred to as auxiliary learning,
has garnered considerable interest in recent years (Liebel and Körner, 2018).
In particular, auxiliary learning is a specific type of MTL, where auxiliary
tasks are intentionally crafted to ultimately boost the performance of the
principal task. As of now, auxiliary tasks are found by meta-learning (Liu et
al., 2019b; Navon et al., 2021), but this requires the a priori definition of
the hierarchy of the desired auxiliary tasks and is computationally
inefficient. Thus, the question is: can we discover with no prior knowledge
one or more additional auxiliary tasks in order to improve the performance of
the principal task?

In this paper, we explore this difficult problem by proposing Detaux, a weakly
supervised strategy that discovers auxiliary classification tasks that enable
solving a single-task classification problem in a multi-task fashion.
Specifically, Detaux is capable of individuating unrelated auxiliary tasks:
unrelatedness in MTL means to have two or more tasks that do not share any
features, as proven to be effective in the MTL literature (Wang et al., 2003;
Zhou et al., 2011; Paredes et al., 2012; Jayaraman et al., 2014; Zheng et al.,
2019; Liu et al., 2019a).

In particular, our method takes roots in the idea from Paredes et al. (2012),
which starts with two groups of tasks, the principal task and the auxiliary
tasks, which are given and known to be unrelated, and assumes the claim that
joint learning of unrelated tasks can improve the performance on the principal
task. They propose to generate a shared low-dimensional representation for
both the principal task and the unrelated auxiliary tasks forcing these two
representations to be orthogonal.

Figure 1:  Detaux involves two steps. First, we use weakly supervised
disentanglement to isolate the structural features specific to the principal
task in one subspace. Next, we identify which is the subspace with the most
disentangled factor of variation related to the principal task, and through a
clustering module, we obtain new labels. These can be used to create a new
classification task that can be combined with the principal task in any MTL
model.

The procedure from Paredes et al. (2012) exploits a linear classifier and
requires the knowledge of the labels for both the principal task and the
auxiliary tasks. With our method, we aim to follow a similar process giving up
on the supervision and fostering non-linear classifiers estimated by neural
networks. The proposed method generates auxiliary tasks in such a way that
their labels implicitly drive an MTL network to understand the unrelatedness
between the tasks. Our idea is to work in a specific representation space, a
product manifold, to unveil the auxiliary tasks for a given principal task. We
get inspiration from Fumero et al. (2021), who discovered the product manifold
as a convenient representation basis for disentanglement. In particular, as
depicted in Figure 1, we extract task-specific features using weakly
supervised disentanglement; then, we identify the most disentangled factor of
variation within a subspace, and finally, we generate new labels via a
clustering module to enable seamless integration with the primary task in any
MTL model.

This makes our proposed pipeline agnostic to the choice of the MTL model,
given that the latter acts directly on the primary and auxiliary labels. In
this way, any MTL model can be chosen, depending on several factors besides
performance, such as efficiency, scalability, and resource constraints. In the
experimental section, we utilize three different MTL models with Detaux,
revealing its flexibility.

##  2 Related Work

We organize this section into three different parts, each one providing an
overview of a topic related to our work: i) MTL and auxiliary learning; ii)
disentanglement; and iii) existing studies on the relationship between MTL and
disentanglement.

###  2.1 MTL and auxiliary learning

MTL, _i.e_., the procedure through which we can solve multiple learning
problems at the same time (Caruana, 1997), can help us reduce inference time,
reach improved accuracy, and increase data efficiency (Standley et al., 2020).
When the adopted dataset contains annotation for multiple tasks, the
challenges to face concern which tasks may work well together (Zamir et al.,
2018; Standley et al., 2020; Fifty et al., 2021) or how to weigh the losses of
different tasks (Kendall et al., 2018) to create a better joint optimization
objective. Numerous methods have recently emerged addressing the simultaneous
resolution of multiple tasks (Gao et al., 2019; Vandenhende et al., 2020).

A different problem arises when we would like to use one of these methods but
only one task is approachable, given the annotations in the considered
dataset. Auxiliary task learning aims at maximizing the prediction performance
on a principal task by supervising the model to additionally learn other
tasks, as shown in (Liu et al., 2019b; Navon et al., 2021). Therefore,
auxiliary tasks are tasks of minor interest, or even irrelevant compared to
the principal task we want to solve, and thus can be seen as regularizers if
learned simultaneously with the task of interest (Liebel and Körner, 2018).
For example, Paredes et al. (2012) suggests that using two unrelated groups of
tasks, where one of them is hosting the principal task, can lead to better
performance, where unrelated means that the two groups of tasks are defined by
an orthogonal set of features. Also in Liebel and Körner (2018) the authors
make use of seemingly unrelated tasks to help the learning on one principal
task, this time without imposing any constraint on the feature structure. In
our case, we are working in product manifold, which has been already shown by
Fumero et al. (2021) as effective for separating embedding subspaces that are
orthogonal by design.

Moreover, recent emerging techniques leverage meta-learning to effectively
select the most appropriate auxiliary tasks or even autonomously create novel
ones. Liu et al. (2019b) and Li and Shan (2021) both train two neural networks
simultaneously, a label-generation network to predict the auxiliary labels and
a multi-task network to train the primary task alongside the auxiliary task.
These, in contrast with our approach, require the a priori definition of a
hierarchy binding the auxiliary labels to the principal task labels and
present conflicting ideas on the possible semantic interpretation of the
generated labels. Furthermore, they are computationally inefficient: meta-
learning is a resource-intensive technique and requires the retraining of the
entire architecture to change the employed multi-task method.

Even more recently, Dery et al. (2022) propose to deconstruct existing
objectives for NLP within a unified taxonomy, identifying connections between
them, and generating new ones by selecting the best combinations from a
cartesian product of the available options. Furthermore, Nam et al. (2023)
also used meta-learning, presenting a novel framework for generating new
auxiliary objectives to address the niche problem of few-shot semi-supervised
tabular learning.

To the best of our knowledge, we are not aware of any other method that
proposes a systematic approach for generating new labels from a disentangled
latent space, in order to enable MTL classification when only the annotations
for one task are given in the considered dataset.

###  2.2 Learning disentangled representations

Representing data in a space where different components are independent is a
long-standing research topic in machine learning. The rise of deep learning in
recent years led to proposed learning disentangled representations as an
important aspect of unsupervised deep learning (Bengio et al., 2013).

Recent literature has proposed several characterizations of disentanglement,
whether that is in terms of group theory (Higgins et al., 2018), metric and
product spaces (Fumero et al., 2021), or permutations of element-wise,
nonlinear functions (Horan et al., 2021). The seminal paper of Higgins et al.
(2017) demonstrated that variational auto-encoders could learn to disentangle
by enforcing the ELBO objective, while Chen et al. (2016) relies on GANs and
an information-theoretic view of disentanglement. Later works, such as
Eastwood and Williams (2018); Singh et al. (2019); Ojha et al. (2020)
extensively explored different directions and use cases. Work by Locatello et
al. (2019a) showed that completely unsupervised disentanglement was not
possible due to the inability of the models to identify factors of variation.
Soon after, the authors proposed weak supervision and having access to few
labels as a way to bypass this limitation (Locatello et al., 2020a, b).

In Detaux, we place ourselves in the same setting of Fumero et al. (2021), but
control and force the disentanglement by supervision only on the known
(principal) task. We describe in detail the main differences between the
original method and our custom implementation in Section 4.

###  2.3 Relationship between MTL and disentanglement

Meng et al. (2019) report a connection between disentangled representations
and MTL, showing that disentangled features can improve the performance of
multi-task networks, especially on data with previously unseen properties.
Disentanglement is obtained by adversarial learning, forcing the encoded
features to be minimally informative about irrelevant tasks. In this case, the
tasks to be disentangled are known a priori, while in our case only the
principal task task is known.

Yang et al. (2022) propose a novel concept called “Knowledge Factorization”.
Exploiting the knowledge contained in a pre-trained multi-task network (called
teacher), the idea is to train disentangled single-task networks (called
students) to reduce the computational effort required by the final single-task
network. The factorization of the teacher knowledge is dual: they provide
structural factorization and representation factorization. In structural
factorization, they split the net into a common-knowledge network and a task-
specific network, based on mutual information.

Finally, Maziarka et al. (2023) propose a disentanglement analysis of MTL
models by creating a semi-synthetic dataset based on latent information in
simple datasets. The authors run the latent information through randomly
initialized fully-connected layers to create tasks that are harder than just
recovering the simple factors. A CNN is then trained to produce a
representation that fits these auxiliary tasks. The reported results may be
seen as inconclusive, as they do not provide a clear indication of how
disentangled representations directly impact MTL performance.

In our proposal, we show that disentanglement in a representation space can be
used as a general prior for MTL, _i.e_., by using disentanglement to mine for
auxiliary tasks, an MTL model extracts a model-specific embedding which
exploits the combination of the principal and the newly discovered labels,
thus improving the performance on the principal task.

##  3 Mathematical Background

To understand the core concepts of our research, let us delve into the
mathematical background presented in Fumero et al. (2021). Due to lack of
space, we only provide a compact overview of the key concepts of this idea and
refer the readers to the original paper for more details regarding the loss
formulations and training process.

Given as input a collection of data, such as a set of labeled images, a
disentanglement procedure should output a representation of these data,
separating the different generative factors that produce all the variations
observed in the data. The method proposed by Fumero et al. (2021) is based on
the manifold hypothesis: high-dimensional data lies near a lower-dimensional
manifold. Fostering this idea, the authors claim that, if independent factors
generate the data, then this manifold is a product manifold:
$\mathcal{M}=\mathcal{M}_{1}\times{}\mathcal{M}_{2}\times{}\ldots{}\times{}\mathcal{M}_{k}$,
where each $\mathcal{M}_{i},i\in{}\\{1\ldots{}k\\}$ is orthogonal to the
others and represents (in the ideal case) at most one generative factor of the
data. Given a pair of data $(x_{1},x_{2})$ that differ in the $h$-th
generative factor only, in Fumero et al. (2021), their learned representations
are considered fully disentangled if they are equal for all projections in the
submanifolds $\\{\mathcal{M}_{i}\\}_{i=1}^{k}$, except for the $h$-th.

In practice, the authors consider a finite-dimensional normed vector space
$\mathbf{Z}$ containing the disentangled latent representation, obtained as
the output of an encoder network ($\mathbf{Z}$ is indeed a special case of a
manifold). Without loss of generality, considering $\mathbf{Z}$ over the field
of reals, we can state that $\mathbf{Z}\subseteq{}\mathcal{R}^{d}$. Under
perfect disentanglement, _i.e_., a fully minimized loss term $\mathcal{L}$
from Equation 2, the latent disentangled representation takes the form of a
Cartesian product space
$\mathbf{Z}=\mathcal{S}_{1}\times{}\mathcal{S}_{2}\times{}\ldots{}\times{}\mathcal{S}_{k}$,
such that for all
$(i,j)\in{}\\{1{}\ldots{}k\\},\mathcal{S}_{i}\cap{}\mathcal{S}_{j}=\\{0\\}$.
Intuitively, each subspace encodes an “axis” of variation. Finally, an
aggregation step restores the complete latent information, and a decoder $g$
maps the resulting vectors back to the input data space.

To learn the global manifold structure, a standard autoencoder architecture is
used, with $f$ being an encoder that receives non-i.i.d data pairs
$(x_{1},x_{2})$ and produces the latent representations $(z_{1},z_{2})$, and
the decoder $g$ that approximates the inverse of $f$. The pair sampling
procedure is designed to induce weak supervision, requiring a pair of images
to vary only in one (or a few) generative factors. Additionally, a set of $k$
neural networks $p_{i},i\in{}\\{1\ldots{}k\\}$ called projectors, are trained
simultaneously and guided by an unsupervised oracle $\mathcal{O}$, to map the
latent codes in the subspaces $\\{\mathcal{S}_{i}\\}_{i=1}^{k}$, each of which
contains the corresponding submanifold $\\{\mathcal{M}_{i}\\}_{i=1}^{k}$. To
wrap up, the representation framework operates in the following way:

$x\xrightarrow{f}z\xrightarrow[i=1\ldots{}k]{\\{p_{i}\\}}\\{s_{i}\\}\xrightarrow[i=1\ldots{}k]{\sum_{i}}\tilde{z}\xrightarrow{g}\tilde{x}\;,$
(1)

where $\tilde{z}$ and $\tilde{x}$ are the aggregated latent representation and
the reconstructed input, respectively. Notably, $f$ and $g$ are initially
trained only to minimize the reconstruction error, which is needed to generate
the global structure of the manifold $\mathcal{M}$. After a warm-up period,
three constraints are added to the optimization problem, each with its own
non-learned weight (_i.e_., $\beta_{1}$, $\beta_{2}$, and $\beta_{3}$) to
disentangle the latent code:

$\mathcal{L}=\mathcal{L}_{rec}+\beta_{1}(\mathcal{L}_{dist}+\mathcal{L}_{spar})+\beta_{2}\mathcal{L}_{cons}+\beta_{3}\mathcal{L}_{reg}\;.$
(2)

$\mathcal{L}_{rec}$ and $\mathcal{L}_{spar}$ correspond to L2 reconstruction
and L1 sparsity loss terms, respectively. $\mathcal{L}_{cons}$, namely the
consistency loss, forces each projector $p_{i}$ to be invariant to changes in
subspaces $\mathcal{S}_{j},j\neq{}i$. The distance loss $\mathcal{L}_{dist}$
is a contrastive loss term that follows the oracle $\mathcal{O}$, which
calculates the subspace $\mathcal{S}_{i}$ where the projections of the images
in the pair $(x_{1},x_{2})$ differ the most, and forces the projection
representation of the two input images onto the subspaces not selected by
$\mathcal{O}$ to be as close as possible, while the projections in the
selected $\mathcal{S}_{i}$ to differ the most. $\mathcal{L}_{cons}$ and
$\mathcal{L}_{dist}$ thus realize the orthogonality of the discovered
subspaces. Finally, the regularization loss $\mathcal{L}_{reg}$ introduces a
penalty that ensures the choice of the oracle $\mathcal{O}$ is evenly
distributed among the subspaces.

##  4 Methodology

In this Section, after introducing the setting and the adopted notation, we
will describe the fundamental contributions of our work: the principal task-
based oracle 4.1, and the auxiliary task discovery procedure 4.2.

##### Setting and notation

We assume the existence of a labeled image dataset
$D=\\{\,(x^{(i)},y^{(i)})\,|\,\forall{i}\in{}\\{1\dots
N\\},\,x^{(i)}\in{}\mathcal{R}^{w\times{}h\times{}c},\,y^{(i)}\in{}\mathbb{N}\\}$,
where $w$ is the width, $h$ is the height, and $c$ is the number of channels,
and $N$ is the number of (image, label) tuples. We consider the classification
task whose fundamental objective is to learn a mapping from the image space
$\\{x^{(i)}|\forall{i}\in{}\\{1\dots N\\}\\}$ to the corresponding label
$\\{y^{(i)}|\forall{i}\in{}\\{1\dots N\\}\\}$.

###  4.1 The principal task-based oracle

A major issue of the procedure proposed by Fumero et al. (2021) in our setting
is that the oracle will assign the variation given by the principal task label
to a random subspace. In order to facilitate the automatic discovery of
auxiliary tasks, we must have a way to accommodate the known variation of the
principal task in an arbitrary subspace and fix it there, constraining the
representation learning. To achieve this, we define a masking procedure that
creates the principal task oracle $\hat{\mathcal{O}}$, such that the
$\alpha{}$-th subspace contains all the variation in the data corresponding to
pairs $(x^{(1)},x^{(2)})$ whose elements differ in their label. This implies
that we do not inject direct knowledge of the known label, but only whether or
not it differs between images of the pair. Thus, in contrast to Fumero et al.
(2021), where all labels are required, we only need to utilize the labels
associated with the principal task. Given an (arbitrary) subspace
$\alpha{}\in{}\\{1\ldots{}k\\}$ (by default we set $\alpha{}=1$) where we wish
to force the variation of the principal task labels, we define
$\hat{\mathcal{O}}$ as:

$\small{\hat{\mathcal{O}}}(z^{(1)},z^{(2)})=\begin{cases}\alpha&\text{if
}y^{(1)}\neq{}y^{(2)}\\\
\operatorname*{arg\,max}_{i}d(s_{i}^{(1)},s_{i}^{(2)}),\forall{}i\neq{}\alpha{}&\text{otherwise}\end{cases}\;,$
(3)

where $d(s_{i}^{(1)},s_{i}^{(2)})$, $i\in\\{1...k\\}$ is the distance between
the projections of the pair $(z^{(1)},z^{(2)})$, in each i-th subspace
$\mathcal{S}_{i}$.

Our new oracle implies that the distance and regularization losses will always
force the variation in the data to be encoded in $\mathcal{S}_{\alpha{}}$ if
$y^{(1)}\neq{}y^{(2)}$, and in a different subspace otherwise (if
$y^{(1)}=y^{(2)}$). Thanks to the consistency loss, the remaining subspaces
can encode other variations while remaining invariant to the ones related to
the principal task and contained in $\mathcal{S}_{\alpha{}}$. This constraint
imposed through the consistency loss will lead us to discover a proper
representation where unknown tasks will correspond to (possibly) multiple
subspaces, orthogonal to the ones of the known task.

###  4.2 Auxiliary task discovery

In the disentangled representation of the input data, where the known
principal task variation is encoded into a specific subspace, we look to find
new, auxiliary tasks in the remaining subspaces.

Intuitively, we wish to have a disentangled subspace that exhibits a
clustering tendency over the projected data. Therefore, set $p_{j}$ to be the
projector that has maximized the overall distance $d$ of Equation 3 after
training and apply a clustering algorithm to the corresponding subspace
$\mathcal{S}_{j}$ to obtain weak labels, determining the new auxiliary
classification task. Under the assumption that tasks living in orthogonal
spaces help increase MTL performance (Paredes et al., 2012), we now show why
our method regularizes the learning procedure and implicitly guides it towards
orthogonal feature spaces for each task.

Let $\mathcal{S}_{\alpha}$ be the subspace that contains the variation of the
principal task, forced by Equation 3, and
$\mathbf{Z}\in{}\mathcal{R}^{N\times{}d}$ be the latent representation of the
input data $x$. Then, for any two vectors
$\\{z^{(1)},z^{(2)}\\}\in{}\mathbf{Z}$, the functions $p_{\alpha{}}$ and
$p_{j}$ will lead to the latent representations
$\\{s^{(1)}_{\alpha{}},s^{(1)}_{j},s^{(2)}_{\alpha{}},s^{(2)}_{j}\\}$, such
that the metric $d$ induced by the norm of the space acts independently on
$d(s^{(1)}_{\alpha{}},s^{(2)}_{\alpha{}})$ from $d(s^{(1)}_{j},s^{(2)}_{j})$,
due to the orthogonality of the basis vectors. In simpler terms, the
relationship between $d(s^{(1)}_{\alpha{}},s^{(2)}_{\alpha{}})$ will not
influence the one between $d(s^{(1)}_{j},s^{(2)}_{j})$, given that the
information encoded in each subspace is different. This can also be verified
in terms of covariance, where due to the orthogonality of subspaces
$\langle{}p_{\alpha}(\mathbf{Z}),p_{j}(\mathbf{Z})\rangle{}=0$, which would
lead to a covariance (given centered data) of 0. To summarize, the labels of
the new auxiliary tasks contain discriminative information that cannot be
reduced to the set of the principal task labels. In this way, the auxiliary
task will not provide redundant information.

While it is possible to use an arbitrary clustering algorithm, we would like
for it to support clusters of arbitrary shapes and for the number of clusters
to not be directly specified (_e.g_., K-Means). Therefore, we utilize the
HDBSCAN algorithm introduced by Campello et al. (2013), due to its ability to
cluster data points based on their proximity and density without the need to
specify the number of clusters explicitly. In the case HDBSCAN finds just one
cluster, no auxiliary task can be found, and the procedure stops. Otherwise,
we have discovered a novel task and its corresponding labels
$y^{\prime}\in{}\mathbb{N}$, which can be used as input to any MTL model. In
this work, we limit ourselves to finding one additional task only. Scaling on
more tasks, _i.e_., investigating additional subspaces, is the subject of
future work.

At this stage, we have access to the new dataset
$D^{\prime}=\\{\,(x_{i},y_{i},y^{\prime}_{i})\,|\,\forall{}i\in{}\\{1\dots
N\\},\,x_{i}\in{}\mathcal{R}^{w\times{}h\times{}c},\,y_{i},y^{\prime}_{i}\in{}\mathbb{N}\\}$.
We are now ready to learn on $D^{\prime}$ using multi-task classification.

##  5 Experiments

We start by providing a motivating toy example on the 3D Shapes (Burgess and
Kim, 2018) synthetic dataset in 5.1. Next, 5.2 explains the experiments on
real-world image datasets (_i.e_., FACES (Ebner et al., 2010), CIFAR-10
(Krizhevsky et al., 2009), SVHN (Netzer et al., 2011), and Cars (Krause et
al., 2013)), to cope with real and complex use cases. Finally, in 5.3 we
discuss some additional experiments and research questions that pinpoint the
advantages of our solution.

For all our experiments, we fix the batch size to 32, the learning rate to
$0.0005$, AdamW (Loshchilov and Hutter, 2018) as optimizer, within the PyTorch
Lightning framework, on an NVIDIA RTX 3090. We train our disentanglement model
for 40 epochs on 3D Shapes and 400 epochs on the other datasets. Instead, all
the MTL models were trained for 150 epochs.

###  5.1 Synthetic data

To showcase the capabilities of our methodology, we begin our experiments with
the 3D Shapes dataset, a widely used benchmark in the disentanglement
literature (Kim and Mnih, 2018; Locatello et al., 2019b; Fumero et al., 2021).
3D Shapes is composed of six generative factors: floor hue, wall hue, object
hue, scale, shape, and orientation, resulting in 480,000 images. To adapt it
to our specific case, we treat the classification of one generative factor as
the principal task and pretend to have no knowledge of the others.

Due to the synthetic nature of the images in 3D Shapes, solving classification
tasks with a neural network can be excessively easy, leaving limited
possibility for improvement through MTL. Thus, to render this setting slightly
more complicated, we add salt-and-pepper noise to 15% of the image pixels.
With the presence of noise, the classification of the object scale (4 classes)
becomes challenging. Hence, we have chosen it as the primary task for our
experiments.

We sample pairs of images with $0.5$ probability of having the same principal
task label and fed these into the disentanglement model, where the encoder $f$
and the decoder $g$ are parametrized through a simple LeNet-like architecture.
The number of subspaces $k$ is set to 10 as used in Fumero et al. (2021).

As described in 4.2, we cluster the most disentangled subspace (not
considering the forced one) according to the disentanglement loss. In our
experiment, it coincides with the subspace which contains the information
regarding the object hue (10 classes). The minimum cluster size hyperparameter
of HDBSCAN is set to 2% of the number of data points $N$.

We feed the noisy 3D Shapes and the enriched label set into an MTL hard
parameter-sharing architecture with a VGG16 (Simonyan and Zisserman, 2015) as
the backbone and compare Single-Task Learning (STL) vs MTL (using,
respectively, one vs two classification heads). For this comparison, we need
to perform a train-test split on 3D Shapes, which is non-trivial since the
possible combinations of the latent factors in the dataset are present exactly
once. Therefore, we split the dataset based on the floor and wall hue labels,
allocating the images that contain 5 out of the 10 values for both factors
only to the testing set, resulting in a 75-25 train-test split. On the
principal task, MTL achieves an accuracy of 0.889, outperforming with a large
margin the 0.125 obtained by STL.

###  5.2 Real data

Table 1:  Classification accuracy on the FACES, CIFAR-10, SVHN, and Cars datasets. (*) indicates that the results are the ones reported in the original paper. In bold the best results, underlined the models that outperform STL. Learning Paradigm | FACES | CIFAR-10 | SVHN | Cars
---|---|---|---|---
STL | 0.915 | 0.844 | 0.956 | 0.711
Ours MTL-HPS | 0.951 | 0.848 | 0.954 | 0.789
Ours NDDR | 0.932 | 0.872 | 0.952 | 0.712
Ours MTI | 0.978 | 0.910 | 0.961 | 0.807
MAXL | 0.933 | 0.868 | 0.953 | 0.638
AuxiLearn | 0.915 | 0.811 | 0.943 | 0.644*

As in the toy example, during the disentanglement procedure, pairs of images
are sampled only based on the principal task labels. In FACES this corresponds
with the person’s facial expression. In CIFAR-10, SVHN, and Cars with the only
annotated labels. To obtain a higher fidelity reconstruction, we utilize a
ResNet-18 encoder-decoder architecture. The number of subspaces $k$ is set to
10.

During the auxiliary task discovery, we set the minimum cluster size
hyperparameter of HDBSCAN to 1% of the number of data points $N$ for all the
datasets.

We compare our approach to two different auxiliary learning methods, _i.e_.,
MAXL (Liu et al., 2019b) and AuxiLearn (Navon et al., 2021). Unlike these
auxiliary learning architectures, our discovered auxiliary task can be
exploited interchangeably with any MTL model. Therefore, we select three
different multi-task models: i) HPS (weighing the losses to give more
importance to the main task, as explained in Kendall et al. (2018)), ii) NDDR
(Gao et al., 2019), and iii) MTI (Vandenhende et al., 2020), to foster this
property of our approach. MTI is always pre-trained due to its inherent nature
of performing high-resolution operations, which is unfeasible on
$64\times{}64$ images.

Table 1 summarizes the results. MTI with our generated labels displays the
best performance. Furthermore, even simple CNN-based architecture like HPS and
NDDR, achieve superior results compared to MAXL and AuxiLearn. Most notably,
we outperform STL with at least one of the MTL models in all the datasets,
whereas MAXL and AuxiLearn have significant performance discrepancies between
the datasets.

We would like to report, for the sake of completeness, that on the CARS
dataset, which contains very complex images, we exploit pre-trained MTL
models. For the disentanglement phase, we change the structure of the encoder
$f$ such that it does not produce a dense representation in the bottleneck
layer but a compressed feature map. The latent space projectors are then
learned using $1\times{}1$ convolution, and the disentanglement losses are
applied to the flattened feature map. Furthermore, in Table 1, with *, we
denote the result we take from the original paper, as we encountered
challenges in replicating the performance using the available code and
information.

###  5.3 Research questions

#### Is disentanglement useful for discovering new auxiliary tasks?

Figure 2: 3D visualization (via PCA) of the discovered auxiliary task in the
entangled autoencoder feature space (a) and the most disentangled subspace (b)
on FACES. Learning a disentangled representation leads to a subspace that
separates the data into two major groups, which correspond to the labels of
the new auxiliary task. Instead, using only a reconstruction loss leads to an
entangled representation from which it is not beneficial to extract auxiliary
tasks. Different colors mean different clusters found by HDBSCAN, which are
subsequently projected by PCA in 3 dimensions. Best viewed in color.

With this ablation, we wish to quantitatively and qualitatively show how
disentanglement is effective in extracting task labels from the underlying
data structure. On the FACES dataset, we compare the auxiliary task generated
by Detaux with the auxiliary task resulting from the clustering on the latent
space of an autoencoder that only learns to reconstruct. Without the
disentanglement, HPS can only reach 0.9 accuracy, worse than the 0.915
obtained by STL, revealing that performing auxiliary task mining on the
entangled autoencoder space does not provide helpful information. In Figure 2,
we compare the projected messy clusters created from the entangled
representation (a) and the clear grouping obtained in the disentangled
representation space (b).

#### Is it necessary to return to the image space for MTL?

One may ask why we did not work directly in the latent feature space found by
the disentanglement procedure. We did some preliminary experiments in this
direction, but they yielded inconclusive results and raised implementation
issues that are out of the scope of the contribution of this paper. A reason
is that most of the MTL frameworks (_e.g_., MTL-HPS, NDDR, and MTI) require
convolution, which is not well defined in the feature space. Another reason is
that Detaux works at a representation level, regardless of any classification
aim induced by a specific classification framework. Its sole purpose is to
reveal, together with the principal subspace determined by the initial labels,
other orthogonal complementary subspaces, which can be assumed as tasks if
they admit clustering. The output of Detaux is an enriched set of labels, that
can be exploited with any MTL model. In addition, Detaux enables us to
visualize and interpret the disentangled subspaces since it reconstructs the
images. This procedure allowed us to understand that, in the toy example, the
additional task corresponds to the object’s hue. Unfortunately, in the more
complex real cases, clear interpretation becomes more challenging, barely
disclosing in the FACES benchmark the gender as an additional task. In the
other cases, we had no clue. Anyway, it is worth noting that we focused on
producing a framework that transforms a single-task classification problem
into an MTL one, and we let the interpretability be in the background, leaving
it as future work.

##  6 Conclusion

In this paper, we propose a novel outlook on the utility of disentangled
representations, utilizing them as a proxy for auxiliary learning in order to
improve the accuracy of a principal task originally solvable only in a single-
task fashion. Our proposed pipeline facilitates the unsupervised discovery of
new tasks from a factorized representation. These newly discovered tasks can
be readily incorporated into any MTL framework. We demonstrate empirically
that this approach offers advantageous performance, and we analyze various
aspects using ablation studies. Our implementation and analysis shed light on
the potential of combining disentanglement and MTL for improved performance
and generalizability.

## Acknowledgements

This work was partially supported by the MUR under the grant “Dipartimenti di
Eccellenza 2023-2027” of the Department of Informatics, Systems and
Communication of the University of Milano-Bicocca, Italy. We gratefully
acknowledge the support of NVIDIA Corporation with the RTX A5000 GPUs granted
through the Academic Hardware Grant Program to the University of Milano-
Bicocca for the project “Learned representations for implicit binary
operations on real-world 2D-3D data”. Furthermore, this study was also carried
out within the PNRR research activities of the consortium iNEST
(Interconnected North-Est Innovation Ecosystem) funded by the European Union
Next-GenerationEU (Piano Nazionale di Ripresa e Resilienza (PNRR) – Missione 4
Componente 2, Investimento 1.5 – D.D. 1058 23/06/2022, ECS_00000043). This
manuscript reflects only the Authors’ views and opinions, neither the European
Union nor the European Commission can be considered responsible for them.

## References

  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * Burgess and Kim (2018) Chris Burgess and Hyunjik Kim.  3d shapes dataset, 2018.
  * Campello et al. (2013) Ricardo JGB Campello, Davoud Moulavi, and Jörg Sander.  Density-based clustering based on hierarchical density estimates.  In _Pacific-Asia conference on knowledge discovery and data mining_ , pages 160–172. Springer, 2013.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 28:41–75, 1997.
  * Chen et al. (2016) Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  _Advances in neural information processing systems_ , 29, 2016.
  * Dery et al. (2022) Lucio M Dery, Paul Michel, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar.  Aang: Automating auxiliary learning.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Eastwood and Williams (2018) Cian Eastwood and Christopher KI Williams.  A framework for the quantitative evaluation of disentangled representations.  In _International Conference on Learning Representations_ , 2018.
  * Ebner et al. (2010) Natalie C Ebner, Michaela Riediger, and Ulman Lindenberger.  Faces—a database of facial expressions in young, middle-aged, and older women and men: Development and validation.  _Behavior research methods_ , 42:351–362, 2010.
  * Fifty et al. (2021) Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn.  Efficiently identifying task groupings for multi-task learning.  _Advances in Neural Information Processing Systems_ , 34:27503–27516, 2021.
  * Fumero et al. (2021) Marco Fumero, Luca Cosmo, Simone Melzi, and Emanuele Rodolà.  Learning disentangled representations via product manifold projection.  In _International conference on machine learning_ , pages 3530–3540. PMLR, 2021.
  * Gao et al. (2019) Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L Yuille.  Nddr-cnn: Layerwise feature fusing in multi-task cnns by neural discriminative dimensionality reduction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3205–3214, 2019.
  * Higgins et al. (2017) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  beta-vae: Learning basic visual concepts with a constrained variational framework.  In _International conference on learning representations_ , 2017.
  * Higgins et al. (2018) Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner.  Towards a definition of disentangled representations.  _arXiv preprint arXiv:1812.02230_ , 2018.
  * Horan et al. (2021) Daniella Horan, Eitan Richardson, and Yair Weiss.  When is unsupervised disentanglement possible?  _Advances in Neural Information Processing Systems_ , 34:5150–5161, 2021.
  * Jayaraman et al. (2014) Dinesh Jayaraman, Fei Sha, and Kristen Grauman.  Decorrelating semantic visual attributes by resisting the urge to share.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1629–1636, 2014.
  * Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7482–7491, 2018.
  * Kim and Mnih (2018) Hyunjik Kim and Andriy Mnih.  Disentangling by factorising.  In _International Conference on Machine Learning_ , pages 2649–2658. PMLR, 2018.
  * Krause et al. (2013) Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In _Proceedings of the IEEE international conference on computer vision workshops_ , pages 554–561, 2013.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.
  * Li and Shan (2021) Yong Li and Shiguang Shan.  Meta auxiliary learning for facial action unit detection.  _IEEE Transactions on Affective Computing_ , 2021.
  * Liebel and Körner (2018) Lukas Liebel and Marco Körner.  Auxiliary tasks in multi-task learning.  _arXiv preprint arXiv:1805.06334_ , 2018.
  * Liu et al. (2019a) Cheng Liu, Chu-Tao Zheng, Sheng Qian, Si Wu, and Hau-San Wong.  Encoding sparse and competitive structures among tasks in multi-task learning.  _Pattern Recognition_ , 88:689–701, 2019a.
  * Liu et al. (2019b) Shikun Liu, Andrew Davison, and Edward Johns.  Self-supervised generalisation with meta auxiliary learning.  _Advances in Neural Information Processing Systems_ , 32, 2019b.
  * Locatello et al. (2019a) Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In _international conference on machine learning_ , pages 4114–4124. PMLR, 2019a.
  * Locatello et al. (2019b) Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem.  Disentangling factors of variation using few labels.  _arXiv preprint arXiv:1905.01258_ , 2019b.
  * Locatello et al. (2020a) Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen.  Weakly-supervised disentanglement without compromises.  In _International Conference on Machine Learning_ , pages 6348–6359. PMLR, 2020a.
  * Locatello et al. (2020b) Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem.  Disentangling factors of variation using few labels.  _arXiv 1905.01258_ , 2020b.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2018.
  * Maziarka et al. (2023) Łukasz Maziarka, Aleksandra Nowak, Maciej Wołczyk, and Andrzej Bedychaj.  On the relationship between disentanglement and multi-task learning.  In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part I_ , pages 625–641. Springer, 2023.
  * Meng et al. (2019) Qingjie Meng, Nick Pawlowski, Daniel Rueckert, and Bernhard Kainz.  Representation disentanglement for multi-task learning with application to fetal ultrasound.  In _Smart Ultrasound Imaging and Perinatal, Preterm and Paediatric Image Analysis_ , pages 47–55. Springer, 2019.
  * Nam et al. (2023) Jaehyun Nam, Jihoon Tack, Kyungmin Lee, Hankook Lee, and Jinwoo Shin.  STUNT: Few-shot tabular learning with self-generated tasks from unlabeled tables.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Navon et al. (2021) Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya.  Auxiliary learning by implicit differentiation.  In _International Conference on Learning Representations_ , 2021.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In _NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011_ , 2011.
  * Ojha et al. (2020) Utkarsh Ojha, Krishna Kumar Singh, Cho-Jui Hsieh, and Yong Jae Lee.  Elastic-infogan: Unsupervised disentangled representation learning in class-imbalanced data.  _Advances in neural information processing systems_ , 33:18063–18075, 2020.
  * Paredes et al. (2012) Bernardino Romera Paredes, Andreas Argyriou, Nadia Berthouze, and Massimiliano Pontil.  Exploiting unrelated tasks in multi-task learning.  In _Artificial intelligence and statistics_ , pages 951–959. PMLR, 2012.
  * Simonyan and Zisserman (2015) K Simonyan and A Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _3rd International Conference on Learning Representations (ICLR 2015)_ , 2015.
  * Singh et al. (2019) Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee.  Finegan: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6490–6499, 2019.
  * Standley et al. (2020) Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese.  Which tasks should be learned together in multi-task learning?  In _International Conference on Machine Learning_ , pages 9120–9132. PMLR, 2020.
  * Vandenhende et al. (2020) Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool.  Mti-net: Multi-scale task interaction networks for multi-task learning.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 527–543. Springer, 2020.
  * Wang et al. (2003) Hongcheng Wang et al.  Facial expression decomposition.  In _Proceedings ninth IEEE international conference on computer vision_ , pages 958–965. IEEE, 2003.
  * Yang et al. (2022) Xingyi Yang, Jingwen Ye, and Xinchao Wang.  Factorizing knowledge in neural networks.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIV_ , pages 73–91. Springer, 2022.
  * Zamir et al. (2018) Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese.  Taskonomy: Disentangling task transfer learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3712–3722, 2018.
  * Zheng et al. (2019) Yu Zheng, Jianping Fan, Ji Zhang, and Xinbo Gao.  Exploiting related and unrelated tasks for hierarchical metric learning and image classification.  _IEEE Transactions on Image Processing_ , 29:883–896, 2019.
  * Zhou et al. (2011) Dengyong Zhou, Lin Xiao, and Mingrui Wu.  Hierarchical classification via orthogonal transfer.  In _Proceedings of the 28th International Conference on International Conference on Machine Learning_ , pages 801–808, 2011.


# MedLane: A Benchmark Dataset for Understandable
Medical Language Translation

Junyu Luo  Zifei Zheng  Hanzhong Ye  Muchao Ye
PennState University
Yaqing Wang
SUNY Buffalo
&Quanzeng You
Microsoft Research
&Cao Xiao
IQVIA
&Fenglong Ma
PennState University

###### Abstract

With the popularity of digital health, it allows patients to easily access
their health data such as clinical notes from doctors. However, digital health
data with professional clinical jargon are incomprehensible to most patients
without professional knowledge, which impedes effective communication between
patients and doctors. A possible solution to bridge this gap is “translating”
professional medical language to layperson-understandable language. Towards
this end, current works try to create a mapping dictionary for the replacement
of professional medical words or directly employ unsupervised neural machine
translation models. However, due to the lack of adequately aligned data, the
performance of existing models is unsatisfactory. Moreover, how to
automatically and fairly evaluate the results is still an open challenge. To
address these challenges, in this paper, we first introduce MedLane – a new
human-annotated Medical Language translation dataset, to align professional
medical sentences with layperson-understandable expressions. The dataset
contains 12,801 training samples, 1,015 validation samples, and 1,016 testing
samples. We then evaluate one naive and six deep learning-based approaches on
the MedLane dataset, including directly copying, a statistical machine
translation approach Moses, four neural machine translation approaches (i.e.,
the proposed PMBERT-MT model, Seq2Seq and its two variants), and a modified
text summarization model PointerNet. To compare the results, we utilize eleven
metrics, including three new measures specifically designed for this task.
Finally, we discuss the limitations of MedLane and baselines, and point out
possible research directions for this task.

##  1 Introduction

The increasing accessibility of healthcare data makes patients understand
their current health conditions more convenient, which may improve the
effectiveness of communication between patients and clinicians. However, many
notes and records are usually written in professional clinical jargon and
abbreviations to achieve efficiency and preciseness, which are difficult to be
understood for patients without medical knowledge and clinical experience. To
fill such a gap, clinicians have to be responsible for translating the
professional medical information to patients in plain languages. Obviously, it
occupies lots of valuable medical resources and further increases burdens for
clinicians. However, the translating work cannot be avoided because the lack
of doctor-patient communication may lead to a tense doctor-patient
relationship Ha and Longnecker (2010). Therefore, there is a great need for
automatically translating professional clinical language to layperson-
understandable language.

Towards this aim, researchers have proposed to use expert-curated dictionaries
Kandula et al. (2010); Zeng and Tse (2006); Zeng-Treitler et al. (2007)or
pattern-based techniques Vydiswaran et al. (2014) to achieve the auto-
translation. However, creating and managing an up-to-date professional
dictionary is not only labor-intensive but also needs the participation of
domain experts. Pattern-based techniques require the high-quality of input
data, which significantly limits their applicability. Recently, researchers
cast the medical language understanding task as a neural machine translation
task Weng et al. (2019). Due to the lack of aligned sentences, the existing
study takes it as an unsupervised translation. However, without any labeled
information and sufficient training data, it is extremely hard for
unsupervised approaches to learn an accurate, usable, and reliable translation
model.

To address the aforementioned challenges, in this paper, we introduce a human-
annotated dataset, named MedLane, which aligns professional-to-customer
sentences extracted from clinical notes in the MIMIC-III v1.4 database Johnson
et al. (2016). We evaluate seven approaches on this dataset with eleven
metrics to show the usability of MedLane. To sum up, our major contributions
are listed as follows:

  * •

A New Dataset. To the best of our knowledge, MedLane is the first new,
publicly available, human-annotated dataset for dealing with the
understandable medical language translation task (UMLT), which consists of
12,801 training samples, 1,015 validation samples, and 1,016 testing samples.

  * •

Novel Measurements. Different from bilingual translation tasks, our task is to
translate professional medical jargon to layperson-understandable language. We
not only require the translated results readable but also accurate and easily
understandable. Thus, we design three new yet general evaluation metrics,
which can also be used for other tasks such as medical report generation.

  * •

Strong Baselines. We propose a straightforward but effective solution, named
PMBERT-MT, which is built upon the pre-trained language model PubMedBERT Gu et
al. (2020). Besides, we implement six other baselines, including directly
copying, a statistical machine translation approach
Moses111http://www.statmt.org/moses/, three neural machine translation
approaches (i.e., Seq2Seq Bahdanau et al. (2014) and its two variants), and a
modified text summarization model PointerNet Vinyals et al. (2015), for
validating the usability of the MedLane dataset.

  * •

Comprehensive Evaluation. We conduct comprehensive experiments by running
seven approaches, evaluating on eleven metrics including eight traditional
machine translation metrics and three new designed ones, and analyzing their
performance in terms of both quality and quantity.

##  2 Related Work

####  2.0.1 Medical Language Translation

There are mainly three types of approaches for making professional medical
language accessible to ordinary customers: dictionary-based, pattern-based and
neural machine translation. All approaches are unsupervised, mainly due to
lacking of the aligned training data.

The dictionary-based approaches rely on the expert-curated dictionary for
transferring professional words Kandula et al. (2010); Zeng and Tse (2006);
Zeng-Treitler et al. (2007). However, they are over-mechanized and cannot deal
with the case of polysemant, i.e., a term can refer to different explanations
according to the context.

The pattern-based approaches aim to identify pairs of professional and
consumer terms from a large corpus like Wikipedia Vydiswaran et al. (2014).
Although they perform better than dictionary-based models, their performance
highly depends on the quality of the input data and the existence of related
corpus.

The neural machine translation models have been leveraged mainly for word
embedding alignment Kang et al. (2016); Weng and Szolovits (2018), and
especially focusing on unsupervised machine translation Weng et al. (2019).
Compared with previous methods, neural machine translation models can generate
sentences with better readability and deal with the polysemant issue. However,
it is extremely hard for unsupervised approaches to learn an accurate, usable,
and reliable translation model without any labeled information and sufficient
training data.

####  2.0.2 Medical Natural Language Processing Datasets

Below we review existing medical natural language datasets. We omit the non-
English ones such as the UFAL Medical Corpus
v.1.0222https://ufal.mff.cuni.cz/ufal_medical_corpus and only focus on the
English based datasets.

The MIMIC-III database Johnson et al. (2016) contains de-identified data from
58,976 ICU patient admissions, which includes several types of medical
information such as demographics, medications, clinical notes, and so on. Most
existing works mainly leverage medical codes and biosignals for disease
detection Ma et al. (2018), readmission prediction Du et al. (2020), mortality
prediction Caicedo-Torres and Gutierrez (2019). There are some studies
focusing on using the clinical notes for automated ICD coding Li and Yu
(2020); Xie and Xing (2018); Cao et al. (2020), natural language inference
Shivade (2018) and unsupervised medical language translation Weng et al.
(2019).

National NLP Clinical Challenges
(n2c2)333https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/ (the original
i2b2 NLP datasets) provide unstructured notes from the Research Patient Data
Repository at Partners Healthcare System in Boston. The challenges include
clinical trial cohort selection, adverse drug events and medication
extraction, clinical concept normalization, clinical semantic textual
similarity, etc., which are totally different from our task.

Though the MIMIC-III and n2c2 datasets contain clinical notes, none of them
provides publicly available human-annotated and aligned data used for
translating professional clinical jargon to layperson-understandable language.
Thus, the MedLane dataset is unique yet necessary.

##  3 The MedLane Dataset

There are two key steps when we create the MedLane dataset. The first step is
to recognize suitable professional medical sentences (named source sentences).
The second is to translate those sentences to the aligned plain customer
understandable sentences by annotators, which are further checked by experts.
We will illustrate the details of each step in the following sub-sections.

###  3.1 Source Sentence Collection

####  3.1.1 Section Selection

We extract professional clinical notes from the NOTEEVENTS table of the MIMIC-
III v1.4 dataset444https://mimic.physionet.org/mimictables/noteevents/. We
mainly focus on the following three sections: (1) History of present illness,
(2) Brief summary of hospital course, and (3) Brief hospital course. These
three sections contain thoughts and reasoning for the communication between
clinicians, which are usually written with professional medical jargon. We
first recognize those sections from the whole clinical notes using the regular
expression technique and then select candidate sentences.

####  3.1.2 Sentence Selection

Although most sentences in the three selected sections contain many
professional clinical jargon words, there are still some sentences written
with plain language. For example, this is a sentence extracted from one of the
MIMIC-III clinical notes, which is _“She now also reports of being hunger.”_.
For such sentences, we do not need to translate them again. To filter those
sentences, we design a heuristic feature-based sentence selection algorithm.

We first tokenize each sentence into a set of words and then use a dictionary-
based approach to identity medical abbreviations. We also count the number of
commonly-used/top-3000 English words555https://www.ef.com/wwen/english-
resources/english-vocabulary/top-3000-words/. Based on the length of the
sentences, the number of medical abbreviations, and the number of top-3000
words, we can automatically select candidate sentences.

Algorithm 1 Sentence Selection Algorithm

0: Target sentence $s$, top-3000 word set $T$, medical abbreviation set $A$

1: Tokenize $s$ into words $[w_{1},...,w_{n}]$

2: for $i=1$ to $n$ do

3: if $w_{i}\in A$ then

4: $abb$ = $abb$ \+ 1;

5: end if

6: $w_{i}^{\prime}=lemmatize(w_{i})$;

7: if $w_{i}^{\prime}\notin T$ then

8: $unc=unc+1$;

9: end if

10: end for

11: if $n<10$ or $\frac{unc+abb}{n}>0.5$ or $\frac{unc+abb}{n}<0.1$ then

12: return False;

13: else

14: return True;

15: end if

The sentence selection algorithm is summarized in Algorithm 1. For each
tokenized word $w_{i}$, we first check whether it belongs to the medical
abbreviation set $A$, and $abb$ in line 4 denotes the current number of
medical abbreviations in the target sentence $s$. If $w_{i}$ is not a medical
abbreviation, then we will check whether the lemmatized $w_{i}$ belongs to the
top-3000 word set $T$. $unc$ in line 8 represents the current number of words
that are neither medical abbreviations nor commonly-used words. Finally, based
on the predefined criteria, the algorithm can automatically decide to keep or
remove the target sentence.

###  3.2 Data Annotation

After we collect a set of source sentences, the next step is to annotate them.
However, annotating medical sentences is different from creating a parallel
bilingual translation corpus. The medical language translation task aims to
“translate” professional and clinical jargon to layperson-understandable
language, which is still from the same language but using different
expressions. Besides, annotating bilingual translation data focuses on
readability and accuracy. Except for those two perspectives, annotating
medical sentences also considers understandability, which is from the
perspective of patients or customers. Based on the above guidance, we hire
workers to translate the source sentences.

The purpose of this work is to create a benchmark for the understandable
medical language translation (UMLT) task, which is not only used for training
translation models but also for fairly evaluating different approaches. Thus,
we set different requirements for workers when annotating the training data
and validation/testing data. In general, they use two steps when annotating
each source sentence. The first step is to paraphrase the abbreviations with
the whole words or phrases. For each abbreviation, there may be several full
forms. For example, “ _TLC_ ” has two full forms666https://medical-
dictionary.thefreedictionary.com/TLC. One is “ _thin-layer chromatography_ ”,
and the other is “ _total lung capacity_ ”. Therefore, it is important for
workers to understand the context in which the abbreviation or term has been
used. The second step is to use simple words to replace professorial medical
expressions. Take the word “ _hematocrit_
”777https://www.mayoclinic.org/tests-procedures/hematocrit/about/pac-20384728
as an example, which means the ratio of the volume of red blood cells to the
volume of the whole blood. If we use the expression, _“the proportion of red
blood cells in the blood”_ , it is much understandable for patients compared
with directly using professional clinical jargon. An example in Figure 1
illustrates the annotating procedure.

When annotating the training data, workers are asked to return the final
understandable sentences, i.e., the simplified ones, which will be checked by
experts again to guarantee the annotation quality. For validation and testing
data, we require workers to return both rephrasing and simplifying forms for
each source sentence.

Figure 1: An example of annotating a source sentence by a work using two
steps, i.e., rephrasing and simplifying. In the rephrasing step, three
abbreviations are replaced by full forms. In the simplifying step, the full
form “nasal cannula” is replaced by “tube insertion on nose”.

Note that for all the training, validation, and testing data, there is a
special case that we do not need to translate the source sentence. For
example, it is easy to understand the sentence “ _He had a set of surveillance
blood cultures drawn last week, which were negative._ ”. These sentences are
extremely useful when training an understandable translation model because
they can be considered as important indicators for guiding model learning. In
the validation and testing data, another special case is that the sentence may
not be simplified any more. For example, the source sentence is “ _She also
had subjective SOB with CXR suggesting fluid overload._ ”, and the rephrased
and simplified sentences are the same, which is “ _She also had subjective
[shortness of breath] with [chest x-ray] suggesting fluid overload._ ”.

# of tokens in the source sentences | 14,780
---|---
# of tokens in the target sentences | 14,278
# of overlapped tokens between source and target | 12,501
Avg. length of the source sentences | 20.6
Avg. length of the target sentences | 24.0
Avg. # of abbreviations in validation and testing data | 1.2
Table 1: MedLane data statistics.

###  3.3 Dataset Statistics

The MedLane dataset contains 12,801 sentences for training, 1,015 sentences
for validation, and 1,016 sentences for testing. Table 1 shows the statistical
characteristics of the MedLane dataset. Compared with traditional machine
translation datasets, MedLane is small. However, our dataset is different from
those datasets. First, the way of annotation is different as we discussed in
the previous section. Second, there are a large number of overlapped tokens
between source and target sentences, which is also different from traditional
machine translation. These differences make our task unique and challenging.

##  4 Approaches

In this section, we present an easy and straightforward solution (PMBERT-MT)
for solving the challenges of the understandable medical language translation
task. Besides, six baseline approaches are introduced to evaluate the
feasibility and utility of the annotated MedLane dataset.

###  4.1 The PMBERT-MT Model

The pre-trained model BERT Devlin et al. (2018) has shown its superior
effectiveness to improve the performance of different downstream applications,
such as question answering Reddy et al. (2019), natural language understanding
Liu et al. (2019), and image captioning Yang et al. (2019). In the medical
domain, there are several models developed based on BERT, including
ClinicalBERT Alsentzer et al. (2019), BioBERT Lee et al. (2020), and
PubMedBERT Gu et al. (2020). Among these three models, PubMedBERT can achieve
the best performance according to the BLURB
leaderboard888https://microsoft.github.io/BLURB/leaderboard.html. Thus, we
utilize PubMedBERT as the pre-trained model to conduct the translation task,
and the designed PMBERT-MT model is shown in Figure 2.

Figure 2: Overview of the proposed PMBERT-MT model. $\mathbf{v}_{*}$ is a
special token used in the PMBERT-MT model to represent the global information
of a sentence.

Input & Output. Let $[w_{1},w_{2},\cdots,w_{n}]$ denote the tokenized
professional medical sentence $\mathbf{s}$. For each sentence, a special word
$<$_SOS_ $>$ is added to indicate the beginning of sentence $s$. Also, an
ending word $<$_EOS_ $>$ is appended to each ground truth sentence
$\mathbf{y}=[y_{1},y_{2},\cdots,y_{m}]$. Thus, the input of the PMBERT-MT
model is $[<$_SOS_ $>$, $w_{1},w_{2},\cdots,w_{n}]$, and the corresponding
ground truth is $[y_{1},y_{2},\cdots,y_{m},<$_EOS_ $>]$.

Encoder. We first learn an embedding vector for each word $w_{i}\in[<$_SOS_
$>$, $w_{1},w_{2},\cdots,w_{n}]$, i.e., $\mathbf{e}_{i}=embedding(w_{i}),$
where $embedding()$ is the PubMedBERT embedding layer. Then, a transformer
unit Vaswani et al. (2017) is employed to learn the representation of each
input word using:
$\mathbf{v}_{i}=transformer(\mathbf{e}_{i}|\mathbf{e}_{*},\mathbf{e}_{1},\cdots,\mathbf{e}_{n}),$
where $\mathbf{e}_{*}$ is the embedding of $<$_SOS_ $>$. $\mathbf{v}_{*}$ is
the representation of $<$_SOS_ $>$, which is also considered as the context
vector of the input sentence.

Decoder. The decoder aims to generate the target sentence based on both the
input source sentence and the outputs of the encoder. In particular, we
utilize an attention-based LSTM model as the decoder. The decoder is often
trained to generate the $t$-th word $y_{t}$ given the global representation
vector $\mathbf{v}_{*}$ and all the generated words from $y_{1}$ to $y_{t-1}$.
Let $\mathbf{h}_{t}$ represent the hidden state outputted by the decoder LSTM,
which is denoted as:
$\mathbf{h}_{t}=LSTM(\mathbf{h}_{t-1},y_{t-1}|\mathbf{h}_{0}=\mathbf{v}_{*},y_{0}=\mbox{$<$\emph{SOS}$>$}).$

Then we can obtain the weighted context vector $\mathbf{c}_{t}$ using the
attention mechanism as follows:

$\mathbf{c}_{t}=\sum_{i=1}^{n}\alpha_{ti}\mathbf{v}_{i},\alpha_{ti}=\frac{\exp({o}_{ti})}{\sum_{j=1}^{n}(\exp({o}_{tj}))},{o}_{ti}=\mathbf{h}_{t}\mathbf{v}_{i}^{\top}.$

Finally, we can obtain the probability of the $t$-th word:
$p(y_{t}|\\{y_{1},\cdots,y_{t-1}\\},\\{w_{1},\cdots,w_{n}\\})=g(\mathbf{h}_{t},\mathbf{c}_{t}),$
where
$g(\mathbf{h}_{t},\mathbf{c}_{t})=softmax(\mathbf{W}[\mathbf{h}_{t};\mathbf{c}_{t}]+\mathbf{b})$,
$\mathbf{W}$ and $\mathbf{b}$ are two parameters.

###  4.2 Baselines

Naive Translation Approach. As we discussed in the Data Annotation Section,
the understandable medical language translation is different from the
traditional bilingual translation task, which aims to make professional
medical jargon understandable to patients or customers. Besides, even though
some sentences contain a few medical words, they may not require to be
translated. Thus, _directly copying (Copy)_ can be considered as a naive
baseline.

Statistical Machine Translation. We select a widely-used statistical machine
translation (SMT) system _Moses_ 999http://www.statmt.org/moses/ as a
baseline. SMT methods Koehn et al. (2007) follow the same principle, i.e.,
finding the maximum probability models from learning the aligned corpus, but
less flexible, because it is limited by statistical $n$-gram language model
Marino et al. (2006). Thus, the learned word relations are generally limited
within five words. However, the required learning parameters of SMT models are
less compared with advanced neural machine translation methods, which may give
them extra advantages in dealing with small datasets.

Neural Machine Translation. With the recent fast development of deep neural
models and computing ability, neural machine translation has become more and
more popular in the natural language processing area. The adopted PMBERT-MT
model belongs to this category. Besides, we use the representative neural
machine translation model _Seq2Seq_ Bahdanau et al. (2014) as a baseline.
Different from PMBERT-MT, the encoder of Seq2Seq is a bidirectional LSTM
model, but the network structure of the decoder is the same.

In addition, we use two variants of Seq2Seq as baselines. One is the reduced
version of Seq2Seq denoted as _Seq2Seq_ $-$, which does not use the attention
mechanism in the decoder. The other is denoted as _Seq2Seq-S_ , which tries to
evaluate the idea of sharing the embedding space of encoder and decoder
models. In other words, they use the same parameter to embed words. The
remaining components of both Seq2Seq$-$ and Seq2Seq-S are the same as those of
Seq2Seq.

Text Summarization Model. For the medical language translation task, many
words can be directly referred from the source sentences. Thus, the setting is
very similar to the text summarization task Narayan et al. (2017), where the
abstract is generated by selecting proper words from the original text. The
difference is that for our task, some new words may need to be added. Towards
this end, we simply redesign the pointer network Vinyals et al. (2015), called
_PointerNet_ , by adding a generating/referring option to the model. The model
first decides which mode to use for the current output. In the referring mode,
the model is acting as a general pointer network. However, in the generating
mode, the model is acting like a normal Seq2Seq model.

##  5 Experiments

In this section, we conduct experiments on the MedLane dataset to evaluate the
performance of different baselines. We first introduce the experimental
settings for baselines, then describe evaluation metrics, and finally, analyze
the experimental results.

###  5.1 Experimental Settings

For the statistical model Moses, we follow the training procedure listed on
the User Manual and Code Guide
file101010http://www.statmt.org/moses/manual/manual.pdf. For neural machine
translation models and text summarization baseline, we all conduct a grid
search to find the optimal parameters. Finally, for Seq2Seq, Seq2Seq$-$,
Seq2Seq-S, and PointerNet, the hidden size is set to 256 for both encoder and
decoder by greedy search, the learning rate is set to $1e-3$. We use Adam
Kingma and Ba (2014) as the optimizer. Tokenization is performed using NLTK
word tokenizer Bird et al. (2009). Early stop is also applied by checking the
BLEU score Papineni et al. (2002) on the validation set, and the training
batch size is set to 30.

Since PMBERT-MT is developed based on PubMedBERT, the hidden size is the same
as that of PubMedBERT, which is 786. We also use the default PubMedBERT Adam
optimizer with the learning rate as $5e-5$, the warm-up method, the default
PubMedBERT vocabulary, and tokenization are applied. In the evaluation stage,
the same NLTK word tokenizer is applied as baselines to break the sentences
into words for calculating the scores for a fair comparison. Early stop is
also applied with the same method, and the training batch size is set to 30.
All models are trained on Ubuntu 16.04 with 128GB memory and an Nvidia Tesla
P100 GPU.

###  5.2 Evaluation Metrics

####  5.2.1 Existing Machine Translation Metrics

BLEU Papineni et al. (2002) is an automatic algorithm for evaluating the
machine-translated results. BLEU focuses on the word similarity between the
target and candidate sentences, which is helpful in general machine
translation task. In our experiments, we report BLEU-1, BLEU-2, BLEU-3,
BLEU-4, and the averaged BLEU score (BLEU for short). Besides, METEOR Lavie
and Agarwal (2007), ROUGE-L Lin (2004), CIDEr Vedantam et al. (2015) are also
used as the evaluation metrics.

Source: vascular saw the pt and did not feel that there was an acute need for
an invasive procedure .
Target: vascular saw the patient and did not feel that there was an acute need
for an invasive procedure .

Figure 3: Example of the failure of existing metrics.

Model | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | BLEU | METEOR | ROUGE-L | CIDEr | HIT | CWR | AScore
---|---|---|---|---|---|---|---|---|---|---|---
Copy | 0.7898 | 0.7495 | 0.7147 | 0.6826 | 0.7342 | 0.4601 | 0.8496 | 5.9432 | 0.5400 | 0.7001 | 0.6323
Moses | 0.7880 | 0.7130 | 0.6530 | 0.6016 | 0.6889 | 0.4237 | 0.8188 | 5.1046 | 0.6823 | 0.7543 | 0.6859
Seq2seq | 0.7136 | 0.6322 | 0.5969 | 0.5160 | 0.6147 | 0.3533 | 0.7609 | 4.1299 | 0.7388 | 0.7980 | 0.6648
Seq2seq- | 0.5066 | 0.3315 | 0.2373 | 0.1787 | 0.3135 | 0.1859 | 0.4948 | 1.2670 | 0.6427 | 0.8367 | 0.4070
Seq2seq-S | 0.7180 | 0.6386 | 0.5778 | 0.5267 | 0.6153 | 0.3604 | 0.7683 | 4.2635 | 0.7331 | 0.7953 | 0.6630
PointerNet | 0.6870 | 0.5904 | 0.5158 | 0.4541 | 0.5618 | 0.3338 | 0.7285 | 3.9458 | 0.6414 | 0.7555 | 0.5949
PMBERT-MT | 0.8003 | 0.7428 | 0.6952 | 0.6531 | 0.7228 | 0.4566 | 0.8218 | 5.3293 | 0.7808 | 0.7358 | 0.7477
Rank | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 2 | 1 | 6 | 1

Table 2: Performance evaluation of all the baselines with eleven metrics.

####  5.2.2 Task-specific Metrics

Since our task is different from traditional machine translation tasks,
directly applying existing evaluation metrics cannot fairly evaluate the
performance of different models. Here, we use an example in Figure 3 to
demonstrate the failure of existing evaluation metrics, such as BLEU score. If
we directly copy the original source sentence as the answer, a very high BLEU
score can be obtained, which is 0.85. However, the critical term “ _pt_ ” is
not translated. Without translating this term, patients or customers may not
totally understand the meaning of this sentence. Thus, it is necessary to
design task-specific metrics.

  * •

_Hit Ration (HIT)_. A key challenge of medical language translation is to
translate professional medical jargon to layperson-understandable words. Let
$N_{p}$ denote the number of professional medical terms in a source sentence
and $N_{t}$ be the number of correctly translated terms in the target. We then
have the HIT ration, which is $HIT=\frac{N_{t}}{N_{p}}$.

  * •

_Common Word Ration (CWR)_. To evaluate the simplicity of the translated
sentences, we calculate the common word ratio for each output sentence. We
first lemmatize each word of the output. If the lemmatized form is in the
top-3000 commonly-used English words, then it is a common word. Otherwise, it
is not a common word. Let $N_{c}$ denote the number of common words in the
translated sentence and $N$ represents the length of the translated sentence.
The common word ratio score is $CWR=\frac{N_{c}}{N}$.

  * •

_Aggregation Score (AScore)_. The quality of the translated sentences is not
only decided by the readability (BLEU) but also related to the correctness
(HIT) and simplicity (CWR). Among these three perspectives, readability and
correctness should be more important than simplicity. Thus, we design a new
score to model them at the same time, which is

$Ascore=\frac{1+\alpha^{2}+\beta^{2}}{\frac{\alpha^{2}}{BLEU}+\frac{\beta^{2}}{HIT}+\frac{1}{CWR}},$

where $\alpha$ and $\beta$ are parameters for controlling the importance of
BLEU and HIT scores. If any of the three metrics is 0, then it will be added
to a very small number such as $10^{-8}$ to avoid AScore to be 0. We take
$\alpha=2$ and $\beta=1.5$ in our experiments.

###  5.3 Experimental Results

Performance Analysis. Table 2 shows the experimental results of all baselines
on eleven evaluation metrics. We can observe that the naive Copy approach
actually outperforms other baselines with traditional machine translation
evaluation metrics, which is in accord with our discussion in the Evaluation
Metrics section. Thus, it is necessary to introduce new metrics for a fair
evaluation. On the three newly designed metrics, the Copy approaches perform
the worst on the HIT score, and its AScore is not high, which is reasonable.

The statistical method Moses has fewer parameters in its language translation
model, and thus, can achieve a stable performance among all the evaluation
metrics. General neural network-based approaches, including Seq2Seq, its
variants, and PointerNet, have a relatively low BLEU score. The reason is that
the labeled data is insufficient for them to train a powerful translation
model from scratch. PMBERT-MT is also a neural network-based method, but
conducting pre-training on a large medical language corpus significantly
increases the ability of model learning. Hence, the BLEU score of PMBERT-MT is
very high. In fact, the performance of PMBERT-MT is the best among all the
baselines.

From the view of the HIT score, we can find a sufficient gap between
generation-based methods (including Seq2Seq, Seq2Seq-S, and PMBERT-MT) and
other methods. To achieve a high HIT score, the accurate recognition of
abbreviations is necessary. Moreover, the models should understand the context
correctly, which is an advantage of neural network-based models. Besides,
another requirement is the generation ability. The use of reference mechanism
probably limits the generation ability of the PointerNet model, and thus, it
does not achieve a high score in the HIT metric.

The CWR score can reflect the simplicity of sentences in a scene. However, we
should notice that the higher CWR scores do not mean better performance. The
reason is that translating professional medical terms will inevitably result
in some uncommon words. Thus, we should attach less importance to the CWR
score when calculating the comprehensive rank.

The top 3 models in the view of AScore are PMBERT-MT, Moses, and Seq2Seq,
which is reasonable. The AScore attaches the highest importance to BLEU score,
followed by the HIT and CWR scores. Using the harmonic mean can make sure that
we punish the tendency of going overboard on one subject and guarantee good
general performance.

Source: |  she was taken to ir where 3 peripheral branches of the right renal artery was embolized , and a hd line was placed under sterile procedure in the rij .
---|---
Reference 1: |  she was taken to [interventional radiology] where 3 peripheral branches of the right kidney artery was embolized , and a [hemodialysis] line was placed under sterile procedure in the [right internal jugular] .
Reference 2: |  she was taken to [interventional radiology] where 3 peripheral branches of the right kidney artery was embolized , and a blood filtering line was placed under sterile procedure in the right internal neck .
PMBERT-MT |  she was taken to interventional radiology where 3 peripheralial of the right renal artery was embolized , and a blood filtering line was placed under sterile in the right internal jugular .
Seq2Seq |  she was taken to interventional radiology where 3 3 of the right kidney artery kidney was was , and a blood filtering was placed under cholangiography cholangiography procedure in the the
Seq2Seq$-$ |  she was taken to the right coronary artery 3 3 , right groin artery was placed in the right femoral artery and right blood pressure of 3 .
PointerNet |  she was taken to radiology where she where she the of the right right artery artery artery artery she artery artery was placed under , the and she was under
Moses |  she was taken to today where 3 peripheral branches of the right kidney artery was embolized , and a blood filtering was placed under sterile procedure in the rij .

Table 3: An example that PMBERT-MT outperforms other baselines. We can observe
that PMBERT-MT misses some words in the translated sentence, but all the
prefession terms are translated. The BLEU score of PMBERT-MT is 0.8311, and
the HIT score is 1.000, which are much higher than those of other baselines.

Source: |  # cirrhosis : patient with history of alcoholic vs nash cirrhosis complicated by esophagel , gastric , and rectal varices
---|---
Reference 1: |  # [chronic disease of the liver] : patient with history of alcoholic vs [non-alcoholic steatohepatitis] [chronic disease of the liver] complicated by esophagel , gastric , and rectal varices .
Reference 2: |  # [chronic disease of the liver] : patient with history of alcoholic vs liver inflammation and damage complicated by esophagel , gastric , and rectal varices .
PMBERT-MT |  cirrhosis cirrhosis : patient with history of alcoholic vs late disease disease complicated by food , , , , , , , eseseseseseseseseseseseseses .
Seq2Seq |  # cirrhosis : patient with history of alcoholic vs cirrhosis cirrhosis , gastric , gastric , and , , , varices . 22
Seq2Seq$-$ |  # surgical history of patient with history by , history , patient , by surgical by and by surgical tract .
PointerNet |  # cirrhosis : patient with history of painful cell function cirrhosis complicated by , , , , , , and rectal rectal in rectal varices .
Moses |  # cirrhosis : patient with history of alcoholic cirrhosis , complicated by nash esophagel , acid , and rectal and .

Table 4: A hard example that all the baselines cannot translate accurately.

Case Studies. We provide two examples to demonstrate the effectiveness of
PMBERT-MT and its drawback. In Table 3, we can observe that PMBERT-MT performs
the best. It can correctly translate professional medical terms as well as in
a simplified manner. However, for the hard example in Table 4, almost all the
baselines cannot achieve better performance. Only Moses keeps a relatively
complete sentence structure. A possible reason is that other baselines may be
unfamiliar with the word “cirrhosis”. From these results, we can find that it
is challenging to accurately translate medical jargon to layperson-
understandable language.

Insight Analysis. In our dataset, the length of different sentences varies a
lot. To analyze the relationship between the length and the performance, we
conduct the following experiment. We first divide source sentences of the
testing data into five groups based on their length and then calculate the
values of BLEU, HIT, CWR and AScore on the proposed PMBERT-MT model. The
results are shown in Figure 4.

Figure 4: Sentence length v.s. performance.

From Figure 4, we can observe that with the increase of the sentence length,
the values of BLEU, HIT, and AScore drop, which is in accord with traditional
machine translation tasks. However, the trend of CWR score is different from
that of other three metrics, which sightly increases. The reason is that for
the long sentences, they may contain more common words and do not need to be
translated, which leads to the increase of the CWR score. These results also
confirm the reasonableness of the design of AScore, which assigns more
importance weights to the BLEU and HIT scores compared with the CWR score.
Also, we can conclude that using the CWR metric as the only criteria may not
really reveal the performance of baselines.

##  6 Discussions

Form the experimental results, we can find that supervised machine translation
methods generally achieve acceptable performance on this task. Using a pre-
trained language model is an effective approach to reduce the learning
difficulty, which leads to better performance. Next, we discuss the limitation
of the MedLane dataset and the baselines. Also, we point out possible
solutions to improve the performance of understandable medical language
translation.

Limitations. When annotating the dataset, we remove the extremely hard
instances, which contains many professional medical terms and abbreviations.
In the current annotation procedure, workers need a dictionary to check the
abbreviations and find their full forms. However, one abbreviation may have
several full forms, and choosing the correct one may also need domain
knowledge. There may be some out-of-dictionary terms in the dataset. Workers
may ignore these words when conducting annotation. Besides, in the current
dataset, all the sentences are written in lowercase. Some professional terms
with the lowercase may be the same as common words, which may mislead all the
baselines. Finally, the readability of training sentences needs to be further
enhanced.

Future Work. Though the PMBERT-MT model can achieve better performance, it is
still not a perfect solution. To further improve the quality of translation, a
possible solution may incorporate extra knowledge, such as medical knowledge
graph, which may help models to understand the UMLT task more accurately.
Another possible way may introduce information retrieval techniques into this
task for generating understandable sentences. For example, Wikipedia is a good
resource to extract some plain expressions for explaining professional medical
jargon.

Since only a small portion of sentences are selected from clinical notes in
the MIMIC-III dataset, the remaining ones still contain a lot of useful
information. How to use such unlabeled data to improve the performance should
be further studied. There are two possible ways to use unlabeled data. The
first one is to conduct pre-training with unlabeled data and then fine-tune
the current baselines with labeled data. The other way is to conduct semi-
supervised learning to directly use the unlabeled data.

From the experimental results, we can observe that the traditional statistical
machine translation model performs better for some hard examples in Table 4.
Thus, combining neural machine translation models with traditional statistical
models may improve the results.

##  7 Conclusions

In this paper, we introduce MedLane, which is a new human-annotated medical
language translation dataset for solving the task of automatically translating
professional medical sentences for ordinary users. In particular, we run seven
baseline models, including directly copying, a statistical machine translation
model, four neural machine translation models, and a text summarization model.
Besides, three new metrics are designed according to the special requirements
of medical language translation. From the experimental results, we find that
the pre-trained model PMBERT-MT is a practical solution to address the
challenges of the UMLT task. A success case and a failure case are given to
show the translated results of existing models. Finally, a discussion section
points out the limitations of the current dataset and the baselines, and we
also discuss different research directions to further improve the performance
of understandable medical language translation.

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural language processing with Python: analyzing text with the natural language toolkit_.  ” O’Reilly Media, Inc.”.
  * Caicedo-Torres and Gutierrez (2019) William Caicedo-Torres and Jairo Gutierrez. 2019.  Iseeu: Visually interpretable deep learning for mortality prediction inside the icu.  _Journal of biomedical informatics_ , 98:103269.
  * Cao et al. (2020) Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Shengping Liu, and Weifeng Chong. 2020\.  Hypercore: Hyperbolic and co-graph representation for automatic icd coding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3105–3114.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Du et al. (2020) Guodong Du, Jia Zhang, Zhiming Luo, Fenglong Ma, Lei Ma, and Shaozi Li. 2020.  Joint imbalanced classification and feature selection for hospital readmissions.  _Knowledge-Based Systems_ , 200:106020.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.  _arXiv preprint arXiv:2007.15779_.
  * Ha and Longnecker (2010) Jennifer Fong Ha and Nancy Longnecker. 2010.  Doctor-patient communication: a review.  _Ochsner Journal_ , 10(1):38–43.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9.
  * Kandula et al. (2010) Sasikiran Kandula, Dorothy Curtis, and Qing Zeng-Treitler. 2010.  A semantic and syntactic text simplification tool for health content.  In _AMIA annual symposium proceedings_ , volume 2010, page 366. American Medical Informatics Association.
  * Kang et al. (2016) Hong Jin Kang, Tao Chen, Muthu Kumar Chandrasekaran, and Min Yen Kan. 2016.  A comparison of word embeddings for english and cross-lingual chinese word sense disambiguation.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, and Evan Herbst. 2007.  Statistical machine translation.
  * Lavie and Agarwal (2007) Alon Lavie and Abhaya Agarwal. 2007.  Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments.  In _Proceedings of the second workshop on statistical machine translation_ , pages 228–231.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Li and Yu (2020) Fei Li and Hong Yu. 2020.  Icd coding from clinical text using multi-filter residual convolutional neural network.  In _AAAI_ , pages 8180–8187.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2019) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019.  Multi-task deep neural networks for natural language understanding.  _arXiv preprint arXiv:1901.11504_.
  * Ma et al. (2018) Fenglong Ma, Quanzeng You, Houping Xiao, Radha Chitta, Jing Zhou, and Jing Gao. 2018\.  Kame: Knowledge-based attention model for diagnosis prediction in healthcare.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , pages 743–752.
  * Marino et al. (2006) José B Marino, Rafael E Banchs, Josep M Crego, Adrià de Gispert, Patrik Lambert, José AR Fonollosa, and Marta R Costa-jussà. 2006.  N-gram-based machine translation.  _Computational linguistics_ , 32(4):527–549.
  * Narayan et al. (2017) Shashi Narayan, Nikos Papasarantopoulos, Shay B Cohen, and Mirella Lapata. 2017\.  Neural extractive summarization with side information.  _arXiv preprint arXiv:1704.04530_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D Manning. 2019.  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Shivade (2018) Chaitanya Shivade. 2018.  Lessons from natural language inference in the clinical domain.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575.
  * Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015.  Pointer networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 2692–2700. Curran Associates, Inc.
  * Vydiswaran et al. (2014) V. G. Vinod Vydiswaran, Qiaozhu Mei, David A Hanauer, and Kai Zheng. 2014.  Mining consumer health vocabulary from community-generated text.  _AMIA … Annual Symposium proceedings / AMIA Symposium. AMIA Symposium_ , 2014:1150.
  * Weng et al. (2019) Wei-Hung Weng, Yu-An Chung, and Peter Szolovits. 2019.  Unsupervised clinical language translation.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3121–3131.
  * Weng and Szolovits (2018) Wei-Hung Weng and Peter Szolovits. 2018.  Mapping unparalleled clinical professional and consumer languages with embedding alignment.
  * Xie and Xing (2018) Pengtao Xie and Eric Xing. 2018.  A neural architecture for automated icd coding.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1066–1076.
  * Yang et al. (2019) Xu Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. 2019.  Auto-encoding scene graphs for image captioning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 10685–10694.
  * Zeng and Tse (2006) Qing T Zeng and Tony Tse. 2006.  Exploring and developing consumer health vocabularies.  _Journal of the American Medical Informatics Association_ , 13(1):24–29.
  * Zeng-Treitler et al. (2007) Qing Zeng-Treitler, Sergey Goryachev, Hyeoneui Kim, Alla Keselman, and Douglas Rosendale. 2007.  Making texts in electronic health records comprehensible to consumers: a prototype translator.  In _AMIA Annual Symposium Proceedings_ , volume 2007, page 846. American Medical Informatics Association.