#  Universal Domain Adaptation for Robust Handling of
Distributional Shifts in NLP

Hyuhng Joon Kim1, Hyunsoo Cho1,2, Sang-Woo Lee2,3,4, Junyeob Kim1,
Choonghyun Park1, Sang-goo Lee1,5, Kang Min Yoo1,2,3∗, Taeuk Kim6∗
1Seoul National University, 2NAVER Cloud, 3NAVER AI Lab,
4KAIST, 5IntelliSys, 6Hanyang University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

When deploying machine learning systems to the wild, it is highly desirable
for them to effectively leverage prior knowledge to the unfamiliar domain
while also firing alarms to anomalous inputs. In order to address these
requirements, Universal Domain Adaptation (UniDA) has emerged as a novel
research area in computer vision, focusing on achieving both adaptation
ability and robustness (i.e., the ability to detect out-of-distribution
samples). While UniDA has led significant progress in computer vision, its
application on language input still needs to be explored despite its
feasibility. In this paper, we propose a comprehensive benchmark for natural
language that offers thorough viewpoints of the model’s generalizability and
robustness. Our benchmark encompasses multiple datasets with varying
difficulty levels and characteristics, including temporal shifts and diverse
domains. On top of our testbed, we validate existing UniDA methods from
computer vision and state-of-the-art domain adaptation techniques from NLP
literature, yielding valuable findings: We observe that UniDA methods
originally designed for image input can be effectively transferred to the
natural language domain while also underscoring the effect of adaptation
difficulty in determining the model’s performance.

**footnotetext: Co-corresponding authors.

##  1 Introduction

Figure 1:  The model trained with formal language (source domain) will likely
face spoken language (target domain) in the real world. The model is expected
to properly handle such transferable input despite the distributional shift.
(middle) At the same time, the model should discern unprocessable inputs
(bottom) from the target domain.

Deep learning models demonstrate satisfactory performance when tested on data
from the training distribution. However, real-world inputs encounter novel
data ceaselessly that deviate from the trained distribution, commonly known as
distributional shift. When confronted with such inputs, machine learning
models frequently struggle to differentiate them from regular input.
Consequently, they face challenges in adapting their previously acquired
knowledge to the new data distribution, resulting in a significant drop in
performance (Ribeiro et al., 2020; Miller et al., 2020; Hendrycks et al.,
2020). The aforementioned phenomenon represents a longstanding challenge
within the machine learning community, wherein even recent cutting-edge
language models (OpenAI, 2023; Touvron et al., 2023; Chowdhery et al., 2022;
Brown et al., 2020) do not serve as an exception to this predicament (Wang et
al., 2023).

In response to these challenges, existing literature proposes two distinct
approaches. The first approach, known as Domain Adaptation (DA) (Blitzer et
al., 2006; Ganin et al., 2016b; Karouzos et al., 2021; Wu and Shi, 2022),
endeavors to establish alignment between a new set of data from an unknown
distribution and the model’s prior knowledge distribution. The objective is to
enhance the model’s generalization capability and reduce performance drop
springing from the distributional shift. In parallel, a distinct line of work,
referred to as out-of-distribution (OOD) detection (Aggarwal, 2017; Hendrycks
and Gimpel, 2017; Hendrycks et al., 2019; Cho et al., 2021), focuses on
discerning inputs originating from dissimilar distributions. They opt to
circumvent potential risks or disruptions arising from shifted inputs, thereby
enriching system robustness and resilience.

While both approaches offer unique advantages addressing specific
distributional shifts, integrating their merits could substantially enhance
robustness. In pursuit of this objective, a novel field called Universal
Domain Adaptation (UniDA) (You et al., 2019) has emerged, aiming to harness
the synergies of both OOD detection and DA when confronted with distributional
shifts. UniDA leverages the best of the two worlds and offers comprehensive
perspectives that integrate the merits of these two research areas. The
essence of UniDA lies in measuring the uncertainty of the data from the
shifted distribution precisely. Then, we can enhance the model’s
transferability by distinguishing portions of low-uncertainty inputs that can
be adequately handled with the current model’s knowledge. Simultaneously, we
enrich the robustness of the model to OOD inputs by discerning the remaining
samples that cannot be processed normally. However, distinguishing between
these inputs and properly processing them becomes increasingly challenging
without explicit supervision.

Despite the versatility of UniDA, this topic has yet to be explored in the
Natural Language Processing (NLP) literature. As a cornerstone in enhancing
reliability against distributional shifts in NLP, we introduce a testbed for
evaluating the model’s robustness in a holistic view. First, we construct
various adaptation scenarios in NLP, utilizing an array of thoughtfully
selected datasets. To discern the degree to which our proposed datasets
incorporate the various degree of challenges in UniDA, we define two novel
metrics: Performance Drop Rate (PDR) and Distinction Difficulty Score (DDS).
Using these metrics, We verify that our testbed captures a broad spectrum of
distributional shifts. Finally, based on the suggested setting, we
systematically compare several UniDA methods inherently designed for the task,
against heuristic combinations of previous approaches for the parts of the
problem, i.e., OOD detection and DA.

Our empirical results show that UniDA methods are fully transferable in the
NLP domain and can robustly respond to various degrees of shift. Moreover, we
find out that the adaptation difficulty notably affects the performance of the
methods. In certain circumstances, DA methods display comparable or even
better performance. We release our dataset, encouraging future research on
UniDA in NLP to foster the development of more resilient and domain-specific
strategies.222The dataset is available at
https://github.com/heyjoonkim/universal_domain_adaptation_for_nlp

##  2 Universal Domain Adaptation

###  2.1 Problem Formulation

Distributional shift refers to a situation where the joint distribution $P$
estimated from the training dataset fails to adequately represent the wide
range of diverse and complex test inputs. More formally, a distributional
shift arises when the test input $\bm{x}_{\text{test}}$ originates from a
distant distribution $Q$, which is not effectively encompassed by the current
trained distribution $P$.

One of the most prevalent research areas addressing this distributional shift
includes OOD detection and DA. OOD detection aims to strictly detect all
inputs from $Q$ to enhance the model’s reliability. Although distribution $Q$
demonstrates a discernibly different distribution from the distribution $P$,
the trained model can still transfer a subset of instances from $Q$,
overcoming the inherent discrepancy between $P$ and $Q$. This particular
capability serves as a fundamental motivation underlying the pursuit of DA.
UniDA endeavours to integrate the merits of both fields, thereby enhancing
both the generalizability and reliability of the model. Specifically, let us
divide the target distribution $Q$ into disjoint subsets $H$, which share the
same label space with source distribution $P$ and its complement $I$ ($Q=H\cup
I$). The objective of UniDA is to enrich the robustness of the model by
flexibly transferring existing knowledge to transferable samples from $H$
while firing alarms to unknown samples from $I$.

###  2.2 Challenges in UniDA

UniDA models should be capable of accurately capturing the underlying reasons
behind the shift, thereby enabling the discrimination between transferable
samples and unknown samples. Among the diverse categories of causes, the
domain gap and the category gap (You et al., 2019) emerge as pivotal factors,
each exerting a substantial impact on the overall complexity of the UniDA
problem. While these concepts have previously been defined in a rather vague
manner, we deduced the necessity for a more explicit definition. Thus, we set
forth to redefine the concepts of domain and category gap more explicitly.

Domain gap refers to the performance drop when a model trained on $P$ fails to
correctly process transferable inputs due to the fundamental discrepancy
between $P$ and $H$, i.e., a domain shift. A dataset with a higher domain gap
amplifies the problem’s difficulty as the trained model becomes more
susceptible to misaligning transferable samples.

A category shift, characterized by the disparity in the class sets considered
by $P$ and $I$, causes a category gap. Category gap represents the performance
drop that arises for inputs from $I$, which cannot be processed properly due
to differing class sets between $P$ and $I$, which are erroneously handled. A
larger category gap makes distinguishing unknown samples from transferable
samples harder, thereby worsening the robustness of the model.

From the perspective of addressing the domain gap and category gap, the main
goal of UniDA is to minimize both gaps simultaneously. This aims to ensure
transferable samples properly align with the source domain for adequate
processing, while handling unknown samples as unprocessable exceptions.

##  3 Testbed Design

The primary objective of our research is to construct a comprehensive
benchmark dataset that effectively captures the viewpoint of UniDA. To
accomplish our objective, we attempt to create a diverse dataset that
encompasses a range of difficulty levels and characteristics, such as domains,
sentiment, or temporal change. These variations are the fundamental elements
that can significantly influence the overall performance.

Specifically, we initially select datasets from multiple practical domains and
approximate the adaptation difficulty by quantifying different shifts with our
newly proposed metrics. In the following subsections, we provide an in-depth
explanation of our dataset along with the analysis of our benchmarks.

###  3.1 Quantifying Different Shifts

As the extent of both domain and category gaps significantly influences the
overall adaptation complexity, it is essential to quantify these gaps when
designing the dataset for evaluation. Unfortunately, existing literature has
not devised a clear-cut and quantitative measure for assessing domain and
category gaps. Therefore, we endeavoured to define measures that can aptly
approximate the two types of gaps.

Performance Drop Rate (PDR) measures the degree of domain gap by assessing the
absolute difficulty of the dataset itself and the performance drop caused by
the shift from $P$ to $H$. Specifically, we fine-tune bert-base-uncased on the
source train set and evaluate its test set accuracy $acc_{s}$ from the same
distribution. Leveraging the same model trained on the source domain, we then
measure the accuracy of the target test set $acc_{t}$. We measure the
performance degradation caused by the distributional shift by measuring
$acc_{s}-acc_{t}$. Since the significance of the performance drop may vary
depending on the source performance, we normalize the result with the source
performance and measure the proportion of the performance degradation. A more
significant drop rate indicates a greater decline in performance, considering
the source domain performance. Formally, PDR for a source domain $s$ and a
target domain $t$ can be measured as follows:

$\text{PDR}_{s,t}=100\times\frac{acc_{s}-acc_{t}}{acc_{s}}$ (1)

Distinction Difficulty Score (DDS) is measured to estimate the difficulty of
distinguishing between $H$ and $I$, which, in other words, measures the
difficulty of handling the category shift. We utilized the same model trained
on the source domain and extracted the [CLS] representations of the source
inputs. We estimated the source distribution, assuming the extracted
representations follow the multivariate normal distribution. We then extracted
[CLS] representations of target distribution inputs from the same model and
measured the Mahalanobis distance between the source distribution. Using the
distance, we measured the Area Under the ROC Curve (AUC) as a metric for
discerning $I$ and $H$. AUC values closer to 1 indicate the ease of discerning
unknown inputs from the transferable inputs. Since we focus on quantifying the
difficulty in distinguishing the two, we subtract the AUC from 1 to derive our
final measure of interest. For the source domain $s$, the target domain $t$,
and AUC as $\text{AUC}_{s,t}$, DDS can be measured as:

$\text{DDS}_{s,t}=100\times(1-\text{AUC}_{s,t})$ (2)

###  3.2 Implementation of Different Shifts

To construct a representative testbed for UniDA, it is essential to illustrate
domain and category shifts. To exhibit domain shift, we delineated domains
from various perspectives. This involves explicit factors such as temporal or
sentiment and implicit definitions based on the composition of the class set.
Detailed formation of domains for each dataset is stipulated in Section 3.3.

To establish category shifts, the source and the target domain must have a set
of common classes $C$ and a set of their own private classes, $\bar{C}_{s}$
and $\bar{C}_{t}$, respectively. We followed previous works (You et al., 2019;
Fu et al., 2020) by sorting the class name in alphabetical order and selecting
the first $|C|$ classes as common, the subsequent $|\bar{C}_{s}|$ as source
private, and the rest as target private classes. The class splits for each
dataset are stated as $|C|/|\bar{C}_{s}|/|\bar{C}_{t}|$ in the main
experiments.

###  3.3 Dataset Details

We focused on text classification tasks for our experiments. Four datasets
were selected from multiple widely used classification domains in NLP, such as
topic classification, sentiment analysis, and intent classification. We
reformulated the datasets so that our testbed could cover diverse adaptation
scenarios.

Huffpost News Topic Classification (Huffpost) (Misra, 2022) contains Huffpost
news headlines spanning from 2012 to 2022. The task is to classify news
categories given the headlines. Using the temporal information additionally
provided, we split the dataset year-wise from 2012 to 2017, treating each year
as a distinct domain. We selected the year 2012 as the source domain, with the
subsequent years assigned as the target domains, creating 5 different levels
of temporal shifts.

Multilingual Amazon Reviews Corpus (Amazon) (Keung et al., 2020) includes
product reviews that are commonly used to predict star ratings based on the
review, and additional product information is provided for each review. We
have revised the task to predict the product information given the reviews and
utilized the star ratings to define sentiment domains. Reviews with a star
rating of 1 or 2 are grouped as negative sentiment, and those with a rating of
4 or 5 are categorized as positive. We exclude 3-star reviews, considering
them neutral.

MASSIVE (FitzGerald et al., 2022) is a hierarchical dataset for intent
classification. The dataset consists of 18 first-level and 60 second-level
classes. Each domain is defined as a set of classes, including private classes
exclusive to a specific domain and common classes shared across domains. We
divided the common first-level class into two parts based on second-level
classes to simulate domain discrepancy. The half of the divided common class
represents the source domain while the other half represents the target
domain. We assume that the second-level classes within the same first-level
class share a common feature and thus can be adapted.

CLINC-150 (Larson et al., 2019) is widely used for intent classification in
OOD detection. The dataset consists of 150 second-level classes over 10 first
level-classes and a single out-of-scope class. The domain is defined in the
same way as MASSIVE.

Dataset |  | Huffpost
---
(2013)
| Huffpost
---
(2014)
| Huffpost
---
(2015)
| Huffpost
---
(2016)
PDR | 7.87 | 20.69 | 20.47 | 25.00
DDS | 19.67 | 27.79 | 25.71 | 31.19
Dataset |  | Huffpost
---
(2017)
CLINC-150 | MASSIVE | Amazon
PDR | 33.62 | 11.59 | 36.14 | 15.45
DDS | 31.74 | 13.45 | 23.92 | 36.04

Table 1:  PDR and DDS values of each datasets. The largest value of PDR and
DDS is highlighted in bold.

###  3.4 Dataset Analysis

In this section, we intend to validate whether our testbed successfully
demonstrates diverse adaptation difficulties, aligning with our original
motivation. We assess adaptation difficulty from domain and category gap
perspectives, each approximated by PDR and DDS, respectively.

The results of PDR and DDS are reported in Table 1. The result shows diverse
PDR values ranging from 7 to 36 points, indicating various degrees of domain
gap across the datasets. MASSIVE measured the most considerable domain gap,
while Huffpost (2013) demonstrated the most negligible domain gap among the
datasets. Additionally, our testbed covers a wide range of category gaps,
indicated by the broad spectrum of DDS values. Specifically, Amazon exhibits a
significantly high DDS value, representing an extremely challenging scenario
of differentiating the unknown samples from the transferable samples.

Figure 2:  The visualization of adaptation difficulty for each dataset in
terms of domain gap (PDR) and category gap (DDS). We categorized the dataset
into 4 distinct groups based on the adaptation complexity. Best viewed in
color.

We consolidate the two indicators to measure the coverage of different
adaptation complexity of our proposed testbed. We visualized the datasets
considering the PDR and DDS; metrics on the respective axes. Figure 2 is the
visualization of the adaptation complexity for each dataset factored by PDR
and DDS. We grouped the datasets into four distinct clusters based on the
plotted distribution. Datasets closer to the lower-left corner, CLINC-150 and
Huffpost (2013), are easy adaptation scenarios with minor domain and category
gaps. Datasets plotted in the center, Huffpost (2014, 2015, 2016), presents
moderate difficulty. Amazon suffers from a category gap significantly, while
Huffpost (2017) and MASSIVE demonstrate a notable domain gap, yielding high
adaptation complexity. The results validate that our testbed embodies a
diverse range of adaptation difficulties as intended.

##  4 Experimental Setting

###  4.1 Compared Methods

We compare several domain adaptation methods on our proposed testbed. We
selected two previous state-of-the-art closed-set Domain Adaptation (CDA)
methods, UDALM (Karouzos et al., 2021) and AdSPT (Wu and Shi, 2022), under the
assumption that all the inputs from the target domain are transferable without
considering unknown classes. Two previous state-of-the-art UniDA methods were
selected, OVANet (Saito and Saenko, 2021) and UniOT (Chang et al., 2022),
which are fully optimized to handle UniDA scenarios in the vision domain. We
also conducted experiments with additional baseline methods such as DANN
(Ganin et al., 2016a), UAN (You et al., 2019), and CMU (Fu et al., 2020).
However, the performance was subpar compared to the selected methods,
exhibiting a similar tendency. Hence, we report the additional results in
Appendix A. For the backbone of all the methods, we utilized bert-base-uncased
(Devlin et al., 2019) and used the [CLS] representation as the input feature.
Implementation details are stipulated in Appendix B.

###  4.2 Thresholding Method

Since CDA methods are not designed to handle unknown inputs, additional
techniques are required to discern them. A straightforward yet intuitive
approach to detecting unknown inputs is applying a threshold for the output of
the scoring function. The scoring function reflects the appropriateness of the
input based on the extracted representation. If the output of the scoring
function falls below the threshold, the instance is classified as unknown. We
sequentially apply thresholding after the adaptation process.333 In the case
of applying the thresholding first, all the inputs from the target domain
would be classified as OOD. If all the target inputs were classified as OOD,
the criteria for discerning transferable and unknown inputs become inherently
unclear. Therefore, we have only considered scenarios where thresholding is
applied after the adaptation.  Formally, for an input $x$, categorical
prediction $\hat{y}$, threshold value $w$, and a scoring function $f_{score}$,
the final prediction is made as:

$y(x)=\begin{cases}\text{argmax}(\hat{y}),&\text{if $f_{score}(x)>w$}\\\
\texttt{unknown},&\text{otherwise}.\end{cases}$ (3)

We utilize Maximum Softmax Probability (MSP) as the scoring function444In
addition to MSP, we have applied various scoring functions such as cosine
similarity and Mahalanobis distance, but using MSP achieved the best
performance. Results for other thresholding methods have been included in the
Appendix A. (Hendrycks and Gimpel, 2017). Following the OOD detection
literature, the value at the point of 95% from the sorted score values was
selected as the threshold.

###  4.3 Evaluation Protocol

The goal of UniDA is to properly process the transferable inputs and detect
the unknown inputs simultaneously, consequently making both the transferable
and the unknown accuracies crucial metrics. We applied H-score (Fu et al.,
2020) as the primary evaluation metric to integrate both evaluation metrics.
H-score is the harmonic mean between the accuracy of common class $acc_{C}$
and unknown class $acc_{\bar{C}_{t}}$, where $acc_{C}$ is the accuracy over
the common class set $C$ and $acc_{\bar{C}_{t}}$ is the accuracy predicting
the unknown class. The model with a high H-score is considered robust in the
UniDA setting, indicating its proficiency in both adaptation (high $acc_{C}$)
and OOD detection (high $acc_{\bar{C}_{t}}$). Formally, the H-score can be
defined as :

$H_{score}=2\cdot\frac{acc_{C}\cdot
acc_{\bar{C}_{t}}}{acc_{C}+acc_{\bar{C}_{t}}}$ (4)

Although the H-score serves as an effective evaluation criterion for UniDA, we
also report $acc_{C}$ and $acc_{\bar{C}_{t}}$ to provide a comprehensive
assessment. We report the averaged results and standard deviations over four
runs for all experiments.

##  5 Experimental Results

Figure 3:  H-score results of AdSPT and UniOT on all the datasets. The
preferred method varies depending on the adaptation complexity.

Huffpost (2012 $\rightarrow$ 2013) (3 / 4 / 4)
---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 52.74 $\pm 2.92$ | 58.55 $\pm 6.03$ | 55.15 $\pm 2.37$
AdSPT | 55.05 $\pm 2.11$ | 80.66 $\pm 2.99$ |  65.38 $\pm 1.00$
OVANet | 65.11 $\pm 0.60$ | 24.91 $\pm 6.75$ | 35.64 $\pm 6.70$
UniOT | 53.76 $\pm 1.10$ | 65.86 $\pm 3.57$ |  59.14 $\pm 1.14$
CLINC-150 (4 / 3 / 3)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 74.92 $\pm 6.11$ | 69.91 $\pm 13.98$ |  71.28 $\pm 4.62$
AdSPT | 3.96 $\pm 2.17$ | 98.69 $\pm 0.36$ | 7.55 $\pm 3.95$
OVANet | 83.49 $\pm 0.62$ | 31.24 $\pm 1.70$ | 45.45 $\pm 1.88$
UniOT | 64.14 $\pm 9.14$ | 77.36 $\pm 3.94$ |  69.88 $\pm 6.11$

Table 2:  Experimental results on CLINC-150 and Huffpost (2013), which is a
relatively easy adaptation scenario. The best method with the highest H-score
is in bold, and the second-best method is underlined.

Huffpost (2012 $\rightarrow$ 2014) (3 / 4 / 4)
---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 29.90 $\pm 1.89$ | 50.00 $\pm 17.49$ | 34.14 $\pm 5.07$
AdSPT | 29.64 $\pm 4.51$ | 80.03 $\pm 6.09$ |  42.93 $\pm 3.41$
OVANet | 45.85 $\pm 2.62$ | 33.96 $\pm 6.78$ | 36.02 $\pm 3.95$
UniOT | 33.49 $\pm 4.79$ | 71.44 $\pm 6.19$ |  45.23 $\pm 3.60$
Huffpost (2012 $\rightarrow$ 2015) (3 / 4 / 4)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 25.15 $\pm 2.60$ | 61.79 $\pm 8.88$ | 35.56 $\pm 3.15$
AdSPT | 29.08 $\pm 5.39$ | 73.04 $\pm 12.71$ |  40.78 $\pm 3.42$
OVANet | 43.91 $\pm 2.09$ | 35.70 $\pm 4.41$ | 39.21 $\pm 2.43$
UniOT | 27.16 $\pm 3.40$ | 75.50 $\pm 2.68$ |  39.82 $\pm 3.39$
Huffpost (2012 $\rightarrow$ 2016) (3 / 4 / 4)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 29.14 $\pm 1.59$ | 60.87 $\pm 6.87$ | 39.28 $\pm 1.44$
AdSPT | 28.84 $\pm 3.57$ | 82.52 $\pm 4.22$ |  42.55 $\pm 3.27$
OVANet | 44.30 $\pm 2.89$ | 33.49 $\pm 4.79$ | 38.03 $\pm 3.79$
UniOT | 33.09 $\pm 3.62$ | 69.27 $\pm 3.83$ |  44.60 $\pm 2.85$

Table 3:  Experimental results on Huffpost (2014, 2015, 2016), which has a
moderate complexity for adaptation. The best method with the highest H-score
is in bold, and the second-best method is underlined.

Amazon (11 / 10 / 10)
---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 47.69 $\pm 1.47$ | 15.10 $\pm 2.30$ | 22.85 $\pm 2.65$
AdSPT | 30.26 $\pm 3.52$ | 30.93 $\pm 26.14$ | 27.12 $\pm 11.03$
OVANet | 44.50 $\pm 0.84$ | 42.49 $\pm 3.95$ |  43.38 $\pm 1.83$
UniOT | 38.60 $\pm 1.08$ | 70.98 $\pm 7.76$ |  49.90 $\pm 2.08$

Table 4:  Experimental results on Amazon, which has a significant influence of
category gap. The best method with the highest H-score are in bold, and the
second-best method is underlined.

MASSIVE (8 / 5 / 5)
---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 38.33 $\pm 4.88$ | 41.29 $\pm 15.14$ | 39.28 $\pm 9.69$
AdSPT | 36.78 $\pm 2.65$ | 59.87 $\pm 13.47$ | 45.32 $\pm 6.20$
OVANet | 59.04 $\pm 0.69$ | 39.28 $\pm 2.14$ |  47.12 $\pm 1.47$
UniOT | 44.01 $\pm 0.20$ | 79.19 $\pm 3.52$ |  56.52 $\pm 3.49$
Huffpost (2012 $\rightarrow$ 2017) (3 / 4 / 4)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UDALM | 32.85 $\pm 1.05$ | 43.08 $\pm 6.76$ | 37.08 $\pm 2.82$
AdSPT | 35.74 $\pm 3.33$ | 53.04 $\pm 16.02$ |  41.36 $\pm 5.04$
OVANet | 48.37 $\pm 4.12$ | 35.03 $\pm 3.60$ | 40.46 $\pm 2.32$
UniOT | 31.57 $\pm 1.92$ | 75.85 $\pm 9.63$ |  44.36 $\pm 1.27$

Table 5:  Experimental results on MASSIVE and Huffpost (2017), which
demonstrates high domain gap. The best method with the highest H-score is in
bold, and the second-best method is underlined.

###  5.1 Overview

We conduct evaluations based on the clusters defined in Section 3.4 and
analyze how the results vary depending on the adaptation complexity. Figure 3
presents an overview of the H-score results for the best-performing method
from each CDA and UniDA approach: AdSPT representing CDA and UniOT
representing UniDA. Despite an outlier caused by unstable thresholding in
CLINC-150, the overall trend demonstrates that AdSPT manifests comparable
performance in less complex scenarios, while UniOT exhibits superior
performance towards challenging scenarios. These trends align with the
findings of other methods that are not depicted in the figure.

###  5.2 Detailed Results

Table 2 demonstrates the results of relatively easy adaptation scenarios. CDA
methods demonstrate performance that is on par with, or even superior to,
UniDA methods. The results appear counter-intuitive, as CDA methods are
designed without considering unknown samples. Specifically, UDALM outperforms
all the UniDA methods in CLINC-150 and performs comparable or even better in
Huffpost (2013). AdSPT exhibits the best performance in Huffpost (2013).
However, AdSPT suffers a significant performance drop in CLINC-150, as we
speculate this result is due to the inherent instability of the thresholding
method. The misguided threshold classifies the majority of the inputs as
unknown, which leads to a very high $acc_{\bar{C}_{t}}$, but significantly
reduces the $acc_{C}$. This inconsistency also leads to a high variance of
$acc_{\bar{C}_{t}}$ for all the CDA methods.

In the case of moderate shifts, no particular method decisively stands out, as
presented by Table 3. In all cases, AdSPT and UniOT present the best
performance with a marginal difference, making it inconclusive to determine a
superior approach. Despite the relatively subpar performance, UDALM and OVANet
also exhibit similar results. Still, it is notable that CDA methods, which are
not inherently optimized for UniDA settings, show comparable results.

The result of Amazon, in which the category gap is most prominent, is reported
in Table 4. UniDA methods exhibit substantially superior performance to CDA
methods. In particular, while the difference in $acc_{C}$ is marginal, there
exists a substantial gap of up to 55 points in $acc_{\bar{C}_{t}}$. As the
category gap intensifies, we observe the decline in the performance of CDA
methods, which are fundamentally limited by the inability to handle unknown
inputs.

Finally, the results of Huffpost (2017) and MASSIVE, which exhibit a high
domain gap, are reported in Table 5. The result indicates that UniDA methods
consistently display superior performance in most cases. However, the
divergence between the approaches is relatively small compared to Amazon.
UniOT demonstrates the best performance in all datasets, with OVANet’s
slightly lower performance. AdSPT demonstrates marginally better performance
than OVANet in Huffpost (2017), but the gap is marginal. Even though CDA
methods pose comparable performance, UniDA methods demonstrate better overall
performance.

###  5.3 Impact of Threshold Values

(a) CLINC-150

(b) Huffpost (2012 $\rightarrow$ 2013)

(c) Huffpost (2012 $\rightarrow$ 2014)

(d) Huffpost (2012 $\rightarrow$ 2015)

(e) Huffpost (2012 $\rightarrow$ 2016)

(f) Huffpost (2012 $\rightarrow$ 2017)

(g) MASSIVE

(h) Amazon

Figure 4:  H-score performance with different threshold values. Results of
UniDA methods are visualized as a horizontal line for comparison.

The selection of the threshold value considerably influences the performance
of CDA methods. In order to probe the impact of the threshold values on the
performance, we carry out an analysis whereby different threshold values are
applied to measure the performance of the methods.

The results are demonstrated in Figure 4. In cases of low or moderate
adaptation complexity, such as CLINC-150 and Huffpost (2013, 2014, 2015,
2016), CDA methods demonstrate the potential to outperform UniDA methods when
provided an appropriate threshold. However, as the adaptation complexity
intensifies, such as Huffpost (2017), MASSIVE, and Amazon, UniDA methods
outperform CDA methods regardless of the selected threshold. These
observations align seamlessly with the findings from Section 4.3 that
underscore the proficiency of UniDA methods in managing challenging adaptation
scenarios. Additionally, it should be noted that determining the optimal
threshold is particularly challenging in the absence of supervision from the
target domain. Therefore, the best performance should be considered upper-
bound of the CDA methods.

##  6 Related Work

###  6.1 Domain Adaptation

The studies in the field of DA in NLP primarily assumes a closed-set
environment, which the source and the target domain share the same label
space. CDA research predominantly concentrated on learning domain invariant
features (Blitzer et al., 2006; Pan et al., 2010; Ben-David et al., 2020;
Ganin and Lempitsky, 2015; Du et al., 2020) for effective adaptation. With the
advent of pre-trained language models (PLMs), CDA methods have evolved to
effectively leverage the capabilities of PLMs. Techniques such as masked
language modeling (Karouzos et al., 2021) or soft-prompt with adversarial
training (Wu and Shi, 2022) have shown promising results. However, the closed-
set assumption has a fundamental drawback as it may leave the models
vulnerable when exposed to data from an unknown class.

To mitigate such issue, a new line of work named UniDA (You et al., 2019) was
proposed which assumes no prior knowledge about the target domain. You et al.
(2019) quantifies sample-level transferability by using of uncertainty and
domain similarity. Following the work, Fu et al. (2020) calibrates multiple
uncertainty measures to handle such an issue. Saito and Saenko (2021) apply a
one-vs-all classifier to minimize inter-class distance and classify unknown
classes. More recently, Chang et al. (2022) applied Optimal Transport and
further expanded the task to discovering private classes. Other recent works
focus on utilizing mutually nearest neighbor samples (Chen et al., 2022a, c)
or leveraging source prototypes with target samples (Chen et al., 2022b; Kundu
et al., 2022). Despite the practicality of UniDA, its application in the NLP
domain has barely explored.

###  6.2 Out-of-Distribution Detection

The early exploration of OOD detection focused on training supervised
detectors (Dhamija et al., 2018; Lee et al., 2018a; Jiang et al., 2018).
However, since obtaining labeled OOD samples is impractical, recent OOD
detection research has shifted towards unsupervised methods, such as
generating pseudo-OOD data (Chen and Yu, 2021; Zheng et al., 2020), utilizing
self-supervised learning (Moon et al., 2021; Manolache et al., 2021; Li et
al., 2021; Zeng et al., 2021; Zhan et al., 2021; Cho et al., 2022), and
measuring uncertainty through scoring functions for input instances (Hendrycks
and Gimpel, 2017; Lee et al., 2018b; Liu et al., 2020; Tack et al., 2020).
While these methods have shown effectiveness, OOD detection is limited in that
it does not offer opportunities for adaptation.

##  7 Conclusion and Future Work

In this study, we present a testbed for evaluating UniDA in the field of NLP.
The testbed is designed to exhibit various levels of domain and category gaps
through different datasets. Two novel metrics, PDR and DDS, were proposed
which can measure the degree of domain and category gap, respectively. We
assessed UniDA methods and the heuristic combination of CDA and OOD detection
in our proposed testbed. Experimental results show that UniDA methods,
initially designed for the vision domain, can be effectively transferred to
NLP. Additionally, CDA methods, which are not fully optimized in UniDA
scenario, produce comparable results in certain circumstances.

Recent trends in NLP focus on Large Language Models (LLMs) of their
significant generalization abilities. However, the robustness of LLMs from the
perspective of UniDA remains uncertain. As part of our future work, we assess
the performance and the capabilities of LLMs from a UniDA viewpoint.

## Limitations

#### Limited coverage of the evaluated model sizes

The evaluation was conducted only with models of limited size. Moreover, there
is a lack of zero-shot and few-shot evaluations for large language models
(LLMs) that have recently emerged with remarkable generalization capabilities.
The evaluation of LLMs is currently being considered as a top priority for our
future work, and based on preliminary experiments, the results were somewhat
unsatisfactory compared to small models with basic tuning for classification
performance. In this regard, recent research that evaluated LLMs for
classification problems (such as GLUE) also reported that the performance is
not yet comparable to task-specifically tuned models. Considering the
limitations of LLMs in terms of their massive resource usage and the fact that
tuning small models still outperforms them in a task-specific manner, the
findings from this study are still considered highly valuable in the NLP
community.

#### Limited scope of the tasks

Our proposed testbed is restricted to text classification tasks only. The
majority of existing research on DA and OOD also focuses on classification.
This selective task preference is primarily due to the challenge of defining
concepts such as domain shifts and category shifts in generative tasks.
However, in light of recent advancements in the generative capabilities of
models, handling distributional shifts in generative tasks is indubitably an
essential problem that needs to be addressed in our future work.

## Acknowledgement

This work was mainly supported by SNU-NAVER Hyperscale AI Center and partly
supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government (MSIT)
[No.2020-0-01373, Artificial Intelligence Graduate School Program (Hanyang
University), NO.2021-0-02068, Artificial Intelligence Innovation Hub
(Artificial Intelligence Institute, Seoul National University)] and Korea
Evaluation Institute of Industrial Technology (KEIT) grant funded by the Korea
government (MOTIE).

## References

  * Aggarwal (2017) Charu C. Aggarwal. 2017.  _An Introduction to Outlier Analysis_, pages 1–34. Springer International Publishing, Cham.
  * Ben-David et al. (2020) Eyal Ben-David, Carmel Rabinovitz, and Roi Reichart. 2020.  Perl: Pivot-based domain adaptation for pre-trained deep contextualized embedding models.  _Transactions of the Association for Computational Linguistics_ , 8:504–521.
  * Blitzer et al. (2006) John Blitzer, Ryan McDonald, and Fernando Pereira. 2006.  Domain adaptation with structural correspondence learning.  In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_ , pages 120–128, Sydney, Australia. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chang et al. (2022) Wanxing Chang, Ye Shi, Hoang Duong Tuan, and Jingya Wang. 2022.  Unified optimal transport framework for universal domain adaptation.  In _Advances in Neural Information Processing Systems_.
  * Chen and Yu (2021) Derek Chen and Zhou Yu. 2021.  GOLD: Improving out-of-scope detection in dialogues using data augmentation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 429–442, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chen et al. (2022a) Liang Chen, Qianjin Du, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. 2022a.  Mutual nearest neighbor contrast and hybrid prototype self-training for universal domain adaptation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(6):6248–6257.
  * Chen et al. (2022b) Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. 2022b.  Evidential neighborhood contrastive learning for universal domain adaptation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(6):6258–6267.
  * Chen et al. (2022c) Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng. 2022c.  Geometric anchor correspondence mining with uncertainty modeling for universal domain adaptation.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16113–16122.
  * Cho et al. (2022) Hyunsoo Cho, Choonghyun Park, Jaewook Kang, Kang Min Yoo, Taeuk Kim, and Sang-goo Lee. 2022.  Enhancing out-of-distribution detection in natural language understanding via implicit layer ensemble.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 783–798, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Cho et al. (2021) Hyunsoo Cho, Jinseok Seol, and Sang-goo Lee. 2021.  Masked contrastive learning for anomaly detection.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , pages 1434–1441. International Joint Conferences on Artificial Intelligence Organization.  Main Track.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhamija et al. (2018) Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. 2018.  Reducing network agnostophobia.  _Advances in Neural Information Processing Systems_ , 31.
  * Du et al. (2020) Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, and Jianxin Liao. 2020.  Adversarial and domain-aware bert for cross-domain sentiment analysis.  In _Proceedings of the 58th annual meeting of the Association for Computational Linguistics_ , pages 4019–4028.
  * FitzGerald et al. (2022) Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. 2022.  Massive: A 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages.
  * Fu et al. (2020) Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. 2020.  Learning to detect open classes for universal domain adaptation.  In _European Conference on Computer Vision_.
  * Ganin and Lempitsky (2015) Yaroslav Ganin and Victor Lempitsky. 2015.  Unsupervised domain adaptation by backpropagation.  In _Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37_ , ICML’15, page 1180–1189. JMLR.org.
  * Ganin et al. (2016a) Yaroslav Ganin, E. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. 2016a.  Domain-adversarial training of neural networks.  In _Journal of machine learning research_.
  * Ganin et al. (2016b) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016b.  Domain-adversarial training of neural networks.  _The journal of machine learning research_ , 17(1):2096–2030.
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  In _International Conference on Learning Representations_.
  * Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020.  Pretrained transformers improve out-of-distribution robustness.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2744–2751, Online. Association for Computational Linguistics.
  * Hendrycks et al. (2019) Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. 2019.  Deep anomaly detection with outlier exposure.  In _International Conference on Learning Representations_.
  * Jiang et al. (2018) Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. 2018.  To trust or not to trust a classifier.  _Advances in neural information processing systems_ , 31.
  * Karouzos et al. (2021) Constantinos F. Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021.  Udalm: Unsupervised domain adaptation through language modeling.  _ArXiv_ , abs/2104.07078.
  * Keung et al. (2020) Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020.  The multilingual Amazon reviews corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4563–4568, Online. Association for Computational Linguistics.
  * Kundu et al. (2022) Jogendra Nath Kundu, Suvaansh Bhambri, Akshay Kulkarni, Hiran Sarkar, Varun Jampani, and R. Venkatesh Babu. 2022.  Subsidiary prototype alignment for universal domain adaptation.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  * Lee et al. (2018a) Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. 2018a.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  In _International Conference on Learning Representations_.
  * Lee et al. (2018b) Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018b.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * Li et al. (2021) Tian Li, Xiang Chen, Shanghang Zhang, Zhen Dong, and Kurt Keutzer. 2021.  Cross-domain sentiment classification with contrastive learning and mutual information maximization.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 8203–8207.
  * Liu et al. (2020) Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. 2020.  Energy-based out-of-distribution detection.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 21464–21475. Curran Associates, Inc.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Manolache et al. (2021) Andrei Manolache, Florin Brad, and Elena Burceanu. 2021.  DATE: Detecting anomalies in text via self-supervision of transformers.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 267–277, Online. Association for Computational Linguistics.
  * Miller et al. (2020) John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020.  The effect of natural distribution shift on question answering models.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 6905–6916. PMLR.
  * Misra (2022) Rishabh Misra. 2022.  News category dataset.  _arXiv preprint arXiv:2209.11429_.
  * Moon et al. (2021) Seung Jun Moon, Sangwoo Mo, Kimin Lee, Jaeho Lee, and Jinwoo Shin. 2021.  Masker: Masked keyword regularization for reliable text classification.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(15):13578–13586.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Pan et al. (2010) Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2010.  Cross-domain sentiment classification via spectral feature alignment.  In _Proceedings of the 19th International Conference on World Wide Web_ , WWW ’10, page 751–760, New York, NY, USA. Association for Computing Machinery.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Saito and Saenko (2021) Kuniaki Saito and Kate Saenko. 2021.  Ovanet: One-vs-all network for universal domain adaptation.  _arXiv preprint arXiv:2104.03344_.
  * Tack et al. (2020) Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. 2020.  Csi: Novelty detection via contrastive learning on distributionally shifted instances.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 11839–11852. Curran Associates, Inc.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Wang et al. (2023) Jindong Wang, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang, Haojun Huang, Wei Ye, Xiubo Geng, Binxin Jiao, Yue Zhang, and Xing Xie. 2023.  On the robustness of chatgpt: An adversarial and out-of-distribution perspective.  _arXiv preprint arXiv:2302.12095_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu and Shi (2022) Hui Wu and Xiaodong Shi. 2022.  Adversarial soft prompt tuning for cross-domain sentiment analysis.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2438–2447, Dublin, Ireland. Association for Computational Linguistics.
  * You et al. (2019) Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. 2019.  Universal domain adaptation.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zeng et al. (2021) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021.  Modeling discriminative representations for out-of-domain detection with supervised contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 870–878, Online. Association for Computational Linguistics.
  * Zhan et al. (2021) Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao-Ming Wu, and Albert Y.S. Lam. 2021.  Out-of-scope intent detection with self-supervision and discriminative training.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3521–3532, Online. Association for Computational Linguistics.
  * Zheng et al. (2020) Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020.  Out-of-domain detection for natural language understanding in dialog systems.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:1198–1209.

##  Appendix A Full Experimental Results

###  A.1 UniDA Results

Table 7 is the full results of UniDA methods in our proposed testbed. Baseline
methods such as UAN (You et al., 2019) and CMU (Fu et al., 2020) are included
in the results. We can observe that UniDA methods do not always retrain the
same level of applicability in NLP. Specifically, UAN and CMU utilize a fixed
threshold defined in the vision domain. While CMU remains fully compatible in
the NLP domain, UAN struggles to apply effectively, as it fails to detect
unknown samples.

###  A.2 CDA Results

In this section, we demonstrate the experimental results of CDA methods with
two additional scoring functions: cosine similarity and Mahalanobis distance.
The threshold value was selected based on the score from the scoring
functions, using the same approach as the main experiment. Also, we report the
results of DANN (Ganin et al., 2016a) and source-only fine-tuning which was
left out from the main experiment. In some cases, source-only fine-tuning
outperforms other adaptation methods, which is also observed in the vision
domain (You et al., 2019).

  1. 1.

Cosine Similarity (Tack et al., 2020) calculates the cosine similarity score
between the test input and the train input. The score is selected as the
cosine similarity between the input and the nearest neighbor. The results are
reported in Table 8.

  2. 2.

Mahalanobis Distance (Lee et al., 2018b) is the distance of the test sample to
each class distribution. The representation is assumed to follow the
multivariate normal distributions. The distance between the nearest class
distribution is used as the score. The results are demonstrated in Table 9

Additionally, the full experimental results of MSP thresholding are presented
in Table 10.

CDA Methods | BERT | DANN | UDALM | AdSPT
---|---|---|---|---
Huffpost (2013) | 5e-5 | 5e-5 | 1e-4 | 5e-5
Huffpost (2014) | 5e-5 | 5e-5 | 1e-4 | 5e-5
Huffpost (2015) | 5e-5 | 5e-5 | 1e-4 | 5e-5
Huffpost (2016) | 5e-5 | 5e-5 | 1e-4 | 5e-5
Huffpost (2017) | 5e-5 | 1e-4 | 1e-4 | 5e-5
CLINC-150 | 1e-4 | 5e-4 | 5e-4 | 1e-5
MASSIVE | 5e-5 | 5e-5 | 5e-5 | 5e-5
Amazon | 1e-5 | 5e-6 | 5e-5 | 1e-5
UniDA Methods | UAN | CMU | OVANet | UniOT
Huffpost (2013) | 5e-5 | 5e-5 | 5e-5 | 1e-4
Huffpost (2014) | 5e-5 | 5e-5 | 5e-5 | 1e-4
Huffpost (2015) | 1e-4 | 1e-4 | 5e-5 | 5e-5
Huffpost (2016) | 1e-4 | 5e-5 | 5e-5 | 1e-4
Huffpost (2017) | 1e-4 | 5e-5 | 5e-5 | 1e-4
CLINC-150 | 5e-6 | 1e-5 | 5e-6 | 5e-5
MASSIVE | 5e-5 | 5e-5 | 5e-5 | 1e-4
Amazon | 5e-5 | 5e-5 | 5e-5 | 5e-5

Table 6:  Learning rates for each methods. The learning rate was selected from
5e-4, 1e-4, 5e-5, 1e-5, and 1e-6 with the best evaluation performance from the
source domain.

Dataset | Huffpost (2012 $\rightarrow$ 2013) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2014) (3 / 4 / 4)
---|---|---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UAN | 73.58 $\pm 5.76$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 36.38 $\pm 3.04$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$
CMU | 58.18 $\pm 1.88$ | 26.89 $\pm 4.29$ |  36.76 $\pm 4.22$ | 33.13 $\pm 3.35$ | 39.98 $\pm 3.71$ | 36.02 $\pm 1.46$
OVANet | 65.11 $\pm 0.60$ | 24.91 $\pm 6.75$ | 35.64 $\pm 6.70$ | 45.85 $\pm 2.62$ | 33.96 $\pm 6.78$ |  38.56 $\pm 3.95$
UniOT | 53.76 $\pm 1.10$ | 65.86 $\pm 3.57$ |  59.14 $\pm 1.14$ | 33.49 $\pm 4.79$ | 71.44 $\pm 6.19$ |  45.23 $\pm 3.60$
Dataset | Huffpost (2012 $\rightarrow$ 2015) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2016) (3 / 4 / 4)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UAN | 34.17 $\pm 6.00$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 42.53 $\pm 16.52$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$
CMU | 39.57 $\pm 3.54$ | 17.98 $\pm 6.30$ | 24.02 $\pm 5.57$ | 41.80 $\pm 2.08$ | 30.09 $\pm 3.51$ | 34.84 $\pm 1.97$
OVANet | 43.91 $\pm 2.09$ | 35.70 $\pm 4.41$ |  39.21 $\pm 2.43$ | 44.30 $\pm 2.89$ | 33.49 $\pm 4.79$ |  38.03 $\pm 3.79$
UniOT | 27.16 $\pm 3.40$ | 75.50 $\pm 2.68$ |  39.82 $\pm 3.39$ | 33.09 $\pm 3.62$ | 69.27 $\pm 3.83$ |  44.60 $\pm 2.85$
Dataset | Huffpost (2012 $\rightarrow$ 2017) (3 / 4 / 4) | CLINC-150 (4 / 3 / 3)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UAN | 22.74 $\pm 10.63$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 48.99 $\pm 14.49$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$
CMU | 34.87 $\pm 3.14$ | 55.79 $\pm 2.71$ |  42.86 $\pm 2.72$ | 60.70 $\pm 1.02$ | 36.89 $\pm 1.52$ |  45.87 $\pm 1.09$
OVANet | 48.37 $\pm 4.12$ | 35.03 $\pm 3.60$ | 40.46 $\pm 2.32$ | 83.49 $\pm 0.62$ | 31.24 $\pm 1.70$ | 45.45 $\pm 1.88$
UniOT | 31.57 $\pm 1.93$ | 75.85 $\pm 9.63$ |  44.36 $\pm 1.27$ | 64.14 $\pm 9.14$ | 77.36 $\pm 3.94$ |  69.88 $\pm 6.11$
Dataset | MASSIVE (8 / 5 / 5) | Amazon (11 / 10 / 10)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
UAN | 13.00 $\pm 7.18$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 35.40 $\pm 3.44$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$
CMU | 37.38 $\pm 1.59$ | 10.33 $\pm 3.06$ | 16.03 $\pm 3.84$ | 44.30 $\pm 0.90$ | 39.95 $\pm 9.67$ | 41.48 $\pm 5.10$
OVANet | 59.04 $\pm 0.69$ | 39.25 $\pm 2.14$ |  47.12 $\pm 1.47$ | 44.50 $\pm 0.84$ | 42.49 $\pm 3.95$ |  43.38 $\pm 1.83$
UniOT | 44.01 $\pm 3.52$ | 79.19 $\pm 4.14$ |  56.52 $\pm 3.49$ | 38.60 $\pm 1.08$ | 70.98 $\pm 7.76$ |  49.90 $\pm 2.08$

Table 7:  Experimental results of UniDA methods in the proposed testbed. For
each dataset, the best method with the highest H-score is in bold and the
second-best method is underlined.

Dataset | Huffpost (2012 $\rightarrow$ 2013) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2014) (3 / 4 / 4)
---|---|---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 64.06 $\pm 2.13$ | 28.02 $\pm 3.03$ |  38.86 $\pm 2.63$ | 40.63 $\pm 3.55$ | 37.41 $\pm 4.81$ |  38.65 $\pm 1.34$
DANN | 58.92 $\pm 1.62$ | 1.69 $\pm 0.57$ | 3.28 $\pm 1.07$ | 33.08 $\pm 2.94$ | 2.01 $\pm 0.82$ | 3.76 $\pm 1.48$
UDALM | 60.12 $\pm 2.56$ | 23.94 $\pm 3.80$ |  34.12 $\pm 4.00$ | 37.97 $\pm 1.69$ | 29.27 $\pm 6.15$ |  32.84 $\pm 4.22$
AdSPT | 62.71 $\pm 0.88$ | 10.89 $\pm 3.04$ | 18.41 $\pm 4.38$ | 39.57 $\pm 3.81$ | 23.88 $\pm 3.44$ | 29.74 $\pm 3.59$
Dataset | Huffpost (2012 $\rightarrow$ 2015) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2016) (3 / 4 / 4)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 40.75 $\pm 2.12$ | 30.90 $\pm 1.55$ |  35.13 $\pm 1.53$ | 47.07 $\pm 2.46$ | 26.15 $\pm 3.73$ |  33.42 $\pm 2.43$
DANN | 31.71 $\pm 2.49$ | 3.39 $\pm 0.92$ | 6.11 $\pm 1.57$ | 36.68 $\pm 3.86$ | 2.16 $\pm 0.80$ | 4.05 $\pm 1.44$
UDALM | 37.06 $\pm 1.74$ | 24.79 $\pm 4.83$ |  29.59 $\pm 3.89$ | 44.10 $\pm 1.52$ | 23.26 $\pm 2.48$ |  30.38 $\pm 2.02$
AdSPT | 39.92 $\pm 4.75$ | 15.01 $\pm 3.85$ | 21.74 $\pm 4.84$ | 41.44 $\pm 3.63$ | 12.66 $\pm 3.31$ | 19.14 $\pm 3.97$
Dataset | Huffpost (2012 $\rightarrow$ 2017) (3 / 4 / 4) | CLINC-150 (4 / 3 / 3)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 48.84 $\pm 3.16$ | 29.52 $\pm 4.67$ |  36.62 $\pm 3.91$ | 74.53 $\pm 3.02$ | 71.37 $\pm 6.54$ |  72.69 $\pm 2.48$
DANN | 31.30 $\pm 2.34$ | 9.32 $\pm 1.63$ | 14.31 $\pm 2.02$ | 60.21 $\pm 9.33$ | 49.65 $\pm 14.72$ | 52.74 $\pm 10.15$
UDALM | 41.08 $\pm 3.92$ | 29.10 $\pm 7.01$ | 33.48 $\pm 3.83$ | 75.99 $\pm 3.63$ | 78.11 $\pm 5.41$ |  76.90 $\pm 2.35$
AdSPT | 43.02 $\pm 0.74$ | 31.07 $\pm 4.77$ |  35.93 $\pm 3.37$ | 42.94 $\pm 5.59$ | 6.96 $\pm 3.44$ | 11.57 $\pm 4.59$
Dataset | MASSIVE (8 / 5 / 5) | Amazon (11 / 10 / 10)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 51.21 $\pm 3.62$ | 62.96 $\pm 2.06$ |  56.44 $\pm 2.69$ | 44.88 $\pm 2.10$ | 14.49 $\pm 1.25$ |  21.90 $\pm 1.68$
DANN | 41.55 $\pm 9.00$ | 13.74 $\pm 3.52$ | 20.62 $\pm 5.04$ | 44.58 $\pm 0.97$ | 19.61 $\pm 1.08$ |  27.21 $\pm 0.93$
UDALM | 65.22 $\pm 2.67$ | 64.25 $\pm 2.17$ |  64.67 $\pm 0.76$ | 46.65 $\pm 0.47$ | 13.58 $\pm 2.23$ | 20.96 $\pm 2.79$
AdSPT | 36.78 $\pm 2.65$ | 59.87 $\pm 13.47$ | 45.32 $\pm 6.20$ | 30.26 $\pm 1.45$ | 6.45 $\pm 4.55$ | 10.15 $\pm 5.65$

Table 8:  Experimental results of CDA methods with cosine similarity as the
scoring function. For each dataset, the best method with the highest H-score
is in bold and the second-best method is underlined.

Dataset | Huffpost (2012 $\rightarrow$ 2013) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2014) (3 / 4 / 4)
---|---|---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 10.40 $\pm 12.11$ | 35.69 $\pm 5.72$ |  13.78 $\pm 12.38$ | 11.28 $\pm 8.86$ | 42.44 $\pm 2.47$ |  16.41 $\pm 9.26$
DANN | 18.55 $\pm 9.65$ | 1.50 $\pm 0.30$ | 2.69 $\pm 0.62$ | 16.78 $\pm 7.67$ | 1.48 $\pm 0.67$ | 2.70 $\pm 1.22$
UDALM | 7.39 $\pm 10.12$ | 27.73 $\pm 4.25$ | 9.26 $\pm 10.18$ | 4.92 $\pm 4.14$ | 34.04 $\pm 5.66$ | 8.15 $\pm 6.52$
AdSPT | 18.07 $\pm 10.66$ | 7.70 $\pm 0.36$ |  10.21 $\pm 2.49$ | 17.29 $\pm 8.06$ | 18.80 $\pm 3.47$ |  16.87 $\pm 5.39$
Dataset | Huffpost (2012 $\rightarrow$ 2015) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2016) (3 / 4 / 4)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 11.28 $\pm 11.10$ | 37.71 $\pm 4.18$ |  15.38 $\pm 11.27$ | 10.34 $\pm 9.85$ | 31.43 $\pm 5.76$ |  14.52 $\pm 10.93$
DANN | 20.09 $\pm 5.97$ | 2.93 $\pm 1.23$ | 4.88 $\pm 1.59$ | 20.79 $\pm 10.71$ | 1.92 $\pm 0.32$ | 3.44 $\pm 0.41$
UDALM | 4.19 $\pm 2.98$ | 31.19 $\pm 6.15$ | 7.06 $\pm 4.84$ | 6.23 $\pm 5.09$ | 31.52 $\pm 5.19$ | 9.49 $\pm 6.74$
AdSPT | 19.20 $\pm 7.31$ | 14.55 $\pm 3.02$ |  16.30 $\pm 4.47$ | 17.78 $\pm 9.51$ | 9.73 $\pm 4.76$ |  12.10 $\pm 6.48$
Dataset | Huffpost (2012 $\rightarrow$ 2017) (3 / 4 / 4) | CLINC-150 (4 / 3 / 3)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 10.31 $\pm 13.20$ | 34.04 $\pm 5.04$ |  13.10 $\pm 13.18$ | 16.67 $\pm 21.48$ | 72.56 $\pm 5.44$ |  21.87 $\pm 25.69$
DANN | 16.79 $\pm 0.94$ | 7.48 $\pm 1.62$ | 10.29 $\pm 1.63$ | 1.87 $\pm 1.36$ | 41.89 $\pm 20.06$ | 3.25 $\pm 1.89$
UDALM | 7.65 $\pm 5.29$ | 30.08 $\pm 8.33$ | 10.93 $\pm 6.62$ | 5.18 $\pm 8.80$ | 80.15 $\pm 3.95$ |  8.73 $\pm 14.40$
AdSPT | 19.77 $\pm 8.68$ | 30.93 $\pm 3.52$ |  23.30 $\pm 7.35$ | 6.59 $\pm 7.97$ | 29.85 $\pm 29.28$ | 7.41$\pm 5.75$
Dataset | MASSIVE (8 / 5 / 5) | Amazon (11 / 10 / 10)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 5.62 $\pm 9.41$ | 68.53 $\pm 4.54$ |  9.07 $\pm 14.57$ | 4.66 $\pm 2.59$ | 15.97 $\pm 1.59$ |  6.85 $\pm 3.45$
DANN | 4.59 $\pm 3.59$ | 9.73 $\pm 3.60$ | 5.26 $\pm 2.70$ | 8.62 $\pm 7.31$ | 17.74 $\pm 3.69$ |  9.75 $\pm 3.92$
UDALM | 2.96 $\pm 4.90$ | 62.80 $\pm 5.02$ | 5.17 $\pm 8.36$ | 2.68 $\pm 2.80$ | 19.01 $\pm 5.49$ | 4.48 $\pm 4.22$
AdSPT | 5.70 $\pm 6.41$ | 54.61 $\pm 12.44$ |  9.60 $\pm 10.75$ | 4.63 $\pm 4.66$ | 9.38 $\pm 6.68$ | 5.61 $\pm 5.65$

Table 9:  Experimental results of CDA methods with Mahalanobis distance as the
scoring function. For each dataset, the best method with the highest H-score
is in bold and the second-best method is underlined.

Dataset | Huffpost (2012 $\rightarrow$ 2013) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2014) (3 / 4 / 4)
---|---|---
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 51.47 $\pm 3.72$ | 72.07 $\pm 10.30$ |  59.64 $\pm 3.28$ | 26.71 $\pm 1.99$ | 78.98 $\pm 8.23$ |  39.75 $\pm 1.51$
DANN | 49.35 $\pm 6.58$ | 77.79 $\pm 25.05$ | 57.83 $\pm 6.49$ | 14.94 $\pm 3.26$ | 95.24 $\pm 0.70$ | 25.71 $\pm 4.48$
UDALM | 52.74 $\pm 2.92$ | 58.25 $\pm 6.03$ | 55.15 $\pm 2.37$ | 29.90 $\pm 1.89$ | 50.00 $\pm 17.49$ | 36.14 $\pm 5.07$
AdSPT | 55.05 $\pm 2.11$ | 80.66 $\pm 2.99$ |  65.38 $\pm 1.00$ | 29.64 $\pm 4.51$ | 80.03 $\pm 6.09$ |  42.93 $\pm 3.71$
Dataset | Huffpost (2012 $\rightarrow$ 2015) (3 / 4 / 4) | Huffpost (2012 $\rightarrow$ 2016) (3 / 4 / 4)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 27.35 $\pm 2.64$ | 77.13 $\pm 2.20$ |  40.30 $\pm 2.60$ | 33.12 $\pm 1.85$ | 63.65 $\pm 3.63$ |  43.53 $\pm 1.96$
DANN | 16.48 $\pm 7.90$ | 89.93 $\pm 11.51$ | 26.81 $\pm 9.33$ | 11.23 $\pm 4.13$ | 95.41 $\pm 1.23$ | 19.89 $\pm 6.52$
UDALM | 25.15 $\pm 2.60$ | 61.79 $\pm 8.88$ | 35.56 $\pm 3.15$ | 29.14 $\pm 1.59$ | 60.87 $\pm 6.87$ | 39.28 $\pm 1.44$
AdSPT | 29.08 $\pm 5.39$ | 73.04 $\pm 12.71$ |  40.78 $\pm 3.42$ | 28.84 $\pm 3.57$ | 82.52 $\pm 4.22$ |  42.55 $\pm 3.27$
Dataset | Huffpost (2012 $\rightarrow$ 2017) (3 / 4 / 4) | CLINC-150 (4 / 3 / 3)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 33.07 $\pm 3.90$ | 65.82 $\pm 7.47$ |  43.64 $\pm 1.71$ | 64.69 $\pm 3.68$ | 72.08 $\pm 9.82$ |  67.82 $\pm 4.15$
DANN | 22.84 $\pm 2.60$ | 82.46 $\pm 4.39$ | 35.66 $\pm 2.85$ | 48.02 $\pm 4.93$ | 67.48 $\pm 19.86$ | 54.90 $\pm 9.17$
UDALM | 32.85 $\pm 1.05$ | 43.08 $\pm 6.76$ | 37.08 $\pm 2.82$ | 74.92 $\pm 6.11$ | 69.91 $\pm 13.98$ |  71.28 $\pm 4.62$
AdSPT | 35.74 $\pm 3.33$ | 53.04 $\pm 16.02$ |  41.36 $\pm 5.04$ | 3.96 $\pm 2.17$ | 98.69 $\pm 0.36$ | 7.55 $\pm 3.95$
Dataset | MASSIVE (8 / 5 / 5) | Amazon (11 / 10 / 10)
Method | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score | $acc_{C}$ | $acc_{\bar{C}_{t}}$ | H-score
BERT | 30.04 $\pm 2.74$ | 68.95 $\pm 5.53$ |  41.79 $\pm 3.20$ | 44.98 $\pm 2.32$ | 12.66 $\pm 1.47$ | 19.70 $\pm 1.69$
DANN | 20.37 $\pm 11.32$ | 88.69 $\pm 5.27$ | 31.77 $\pm 13.32$ | 45.85 $\pm 0.49$ | 12.52 $\pm 0.71$ | 19.66 $\pm 0.87$
UDALM | 38.33 $\pm 4.88$ | 41.29 $\pm 15.14$ | 39.28 $\pm 9.69$ | 47.69 $\pm 1.47$ | 15.10 $\pm 2.30$ |  22.85 $\pm 2.65$
AdSPT | 36.78 $\pm 2.65$ | 59.87 $\pm 13.47$ |  45.32 $\pm 6.20$ | 30.26 $\pm 3.52$ | 30.93 $\pm 26.14$ |  27.12 $\pm 11.03$

Table 10:  Experimental results of CDA methods with Maximum Softmax
Probability as the scoring function. For each dataset, the best method with
the highest H-score is in bold and the second-best method is underlined.

##  Appendix B Implementation Details

For the experiments, we adopt a 12-layer pre-trained language model bert-base-
uncased (Devlin et al., 2019) as the backbone of all the methods. We utilized
the [CLS] representation as the input feature. AdamW optimizer (Loshchilov and
Hutter, 2019) was used for all the experiments with a batch size of 32. We
selected the best learning rate among 5e-4, 1e-4, 5e-5, 1e-5, and 5e-6. The
learning rate for each method is reported in Table 6. The model was trained
for 10 epochs with an early stopping on the accuracy of the source domain’s
evaluation set. All the experiments were implemented with Pytorch (Paszke et
al., 2019) and Huggingface Transformers library (Wolf et al., 2020). The
experiments take an hour on a single Tesla V100 GPU.

Dataset | CLINC-150 (4 / 3 / 3) | MASSIVE (8 / 5 / 5)
---|---|---
Runs | 1 | 2 | 3 | 1 | 2 | 3
BERT | 44.59 $\pm 6.01$ | 48.08 $\pm 2.90$ | 52.25 $\pm 3.54$ | 51.68 $\pm 1.62$ | 46.68 $\pm 1.90$ |  51.56 $\pm 0.81$
DANN | 47.91 $\pm 9.96$ | 53.23 $\pm 7.92$ |  60.42 $\pm 7.00$ | 47.16 $\pm 3.79$ | 31.39 $\pm 3.93$ | 45.21 $\pm 8.93$
UDALM |  49.17 $\pm 8.75$ |  55.04 $\pm 5.31$ | 52.42 $\pm 8.20$ | 55.52 $\pm 3.00$ | 42.34 $\pm 5.88$ | 51.15 $\pm 3.62$
AdSPT | 31.81 $\pm 8.63$ | 48.62 $\pm 1.85$ | 46.20 $\pm 5.22$ | 45.41 $\pm 9.04$ | 38.61 $\pm 3.90$ | 46.95 $\pm 5.22$
UAN | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$ | 0.00 $\pm 0.00$
CMU | 40.32 $\pm 2.32$ | 38.79 $\pm 2.19$ | 40.92 $\pm 1.77$ | 21.46 $\pm 1.62$ | 18.84 $\pm 5.26$ | 28.33 $\pm 4.29$
OVANet | 41.77 $\pm 1.90$ | 35.89 $\pm 3.86$ | 29.15 $\pm 2.17$ |  63.91 $\pm 3.41$ |  54.31 $\pm 0.63$ | 39.85 $\pm 3.74$
UniOT |  50.94 $\pm 7.20$ |  61.66 $\pm 1.70$ |  56.51 $\pm 3.53$ |  62.07 $\pm 2.89$ |  55.11 $\pm 2.31$ |  63.59 $\pm 3.15$
Table 11:  H-score results of UniDA methods in CLINC-150 and MASSIVE with
different class splits. The best method with the highest H-score is in bold
and the second-best method is underlined.

##  Appendix C Ablation on Different Class Splits

For the main experiment, we utilized class names as the criterion to implement
the category gap. However, this may only show the specific scenario of the
category gap. To provide a more comprehensive analysis, we also report the
results when the class set is randomly split. We utilized CLINC-150 and
MASSIVE dataset for the ablation study, and MSP thresholding was applied for
CDA methods. We conducted three experiments, each with a different class
split, and for every split, we reported the average results of three different
runs.

Table 11 is the results of the experiments. Due to the changes in the class
set to be predicted, the task difficulty varies, resulting in differences in
absolute performances. However, when comparing the relative performance
between different methods, we can observe that they exhibit consistent trends
regardless of the class split.

##  Appendix D Receiver Operating Characteristic (ROC) Curve

To measure Distinction Difficulty Score (DDS), we calculated the AUROC and
subtracted from 1. Figure 5 is the ROC curve of discerning unknown inputs from
the transferable inputs for our proposed datasets. The closer the ROC curve is
to the upper-left corner, it indicates that it is easier to distinguish
between unknown and transferable inputs.

(a) Huffpost (2013)

(b) Huffpost (2014)

(c) Huffpost (2015)

(d) Huffpost (2016)

(e) Huffpost (2017)

(f) CLINC-150

(g) MASSIVE

(h) Amazon

Figure 5:  ROC curve of discerning unknown samples from the transferable
samples. The closer the ROC curve is to the upper-left corner, it becomes
easier to distinguish between the them.


# Simultaneous Machine Translation with Large Language Models

###### Abstract

Large language models (LLM) have demonstrated their abilities to solve various
natural language processing tasks through dialogue-based interactions. For
instance, research indicates that LLMs can achieve competitive performance in
offline machine translation tasks for high-resource languages. However,
applying LLMs to simultaneous machine translation (SimulMT) poses many
challenges, including issues related to the training-inference mismatch
arising from different decoding patterns. In this paper, we explore the
feasibility of utilizing LLMs for SimulMT. Building upon conventional
approaches, we introduce a simple yet effective mixture policy that enables
LLMs to engage in SimulMT without requiring additional training. Furthermore,
after Supervised Fine-Tuning (SFT) on a mixture of full and prefix sentences,
the model exhibits significant performance improvements. Our experiments,
conducted with Llama2-7B-chat on nine language pairs from the MUST-C dataset,
demonstrate that LLM can achieve translation quality and latency comparable to
dedicated SimulMT models.111We will release our code, weights, and data with
publication.

Index Terms—  Simultaneous Machine Translation, Large Language Model,
Incremental Decoding

##  1 Introduction

Fig. 1: The illustration of the pipeline of our framework where the
incremental source text is colored blue, and incremental target text is
colored pink. RALCP denotes the Relaxed Agreement Longest Common Prefix
algorithm proposed by us (§2.2.2).

With the advent of ChatGPT, Large Language Models (LLMs) have emerged as a
focal point of research within the broader NLP academic community. Their
formidable ability to adhere to instructions enables them to address various
conventional NLP problems through conversational interactions. This trend, in
turn, motivates researchers to adapt a wider array of traditional NLP tasks to
LLMs, with the expectation that LLMs can achieve performance on par with, or
even surpass, dedicated specialized models.

Machine translation (MT), as a crucial generative NLP task, typically demands
models to possess robust multilingual capabilities. Moreover, achieving high-
quality translations often requires models to have a substantial amount of
commonsense knowledge. Numerous studies [1, 2] have already demonstrated that
LLMs perform comparably to dedicated machine translation models in high-
resource languages. However, there have been no successful instances of LLMs
employed in the branch of machine translation known as Simultaneous Machine
Translation (SimulMT) [3]. Unlike offline translation, in SimulMT, the source
text accumulates incrementally over time, and the translation model needs to
provide translations incrementally and synchronously. During this process, the
model requires to have a certain policy to make decisions on taking READ or
WRITE actions [3]. Existing approaches, including fixed policy like “wait-k”
[4, 5], adaptive policies such as monotonic attention and imitation learning
[6, 7], and incremental decoding with offline model [8, 9, 10], have already
been successfully applied to sequence-to-sequence models like Transformer
[11]. This leads us to pose the research question: How can we transform an LLM
into a simultaneous translation model? To adapt LLMs to this task, we need to
address the following key challenges:

  * •

To effectively handle accumulating source context and ensure that the target
is generated incrementally under the decoder-only architecture of LLM.

  * •

To design a reading and writing policy for LLM to achieve a well-balanced
performance and latency.

  * •

To bridge the discrepancy between an LLM’s standard pre-training data and the
SimulMT’s incremental nature (i.e., the training data of LLMs assumes that
user instructions and context are complete, but during the inference process
of SimulMT, the source context is partial and incremental).

In this paper, we have leveraged the insights from conventional simultaneous
translation models and combined the “wait-k” [4, 5] policy with the
incremental decoding approach [8, 9, 10] to design a mixture policy that
aligns with LLMs (See Figure 1). This policy enables the adaptation of LLMs to
SimulMT tasks without the need for any specialized training on the learning of
such policy. After subjecting the model to a single epoch of SFT using limited
multilingual data, it achieves performance on par with dedicated simultaneous
translation models. To address the challenge of suboptimal translation quality
when the model encounters partial source inputs as context, we further
incorporated a small amount of prefix data generated by ChatGPT (1000 examples
per language). These prefix data were combined with full-sentence pairs in
SFT. Experimental results demonstrate that the inclusion of prefix data can
lead to improved performance in certain language pairs.

##  2 Method

0: LLM : $g_{\text{LLM}}$, Cumulative Source Content: $S_{t}$, Cumulative
Target Content: $T_{t}$, Variables Definition: Read-$n$: $n$, Wait-$k$: $k$,
Beam-size: $B$, Agreement-degree: $\gamma$, Time step: $t$ { $t$ start from 1}

1: if NOT_FINISHED$(S_{t})$ then

2: if $t<=k$ or $t\mod n>0$ then

3: return $\mathbb{R}$

4: end if

5: end if

6: $X_{t}\leftarrow\phi(S_{t},T_{t})$

7: //LLM only returns new tokens after $X_{t}$

8: $\textbf{C}_{t},\textbf{Pr}_{t}\leftarrow g_{\text{LLM}}(X_{t},B)$

9: //$\textbf{C}_{t}$ and $\textbf{Pr}_{t}$ are sets of beam candidates and
probabilities.

10: if NOT_FINISHED$(S_{t})$ then

11: $P_{t}\leftarrow\text{RALCP}(\textbf{C}_{t},B,\gamma)$

12: else

13: $b^{*}\leftarrow\arg\max_{b}\textbf{Pr}_{t}$

14: $P_{t}\leftarrow C_{t}^{b^{*}},C_{t}^{b^{*}}\in\textbf{C}_{t}$

15: end if

16: if $P_{t}=\emptyset$ then

17: return $\mathbb{R}$

18: end if

19: return $\mathbb{W}(P_{t})$

Algorithm 1 Mixture Policy

###  2.1 Prompt Design of Incremental States

While there are significant differences in the decoding process between
SimulMT models and offline MT models, the fundamental approach to guiding LLMs
in translation remains consistent. This approach continues to rely on
constructing prompts composed of instructions + context as input, prompting
LLMs to perform text completion. To elaborate further, in offline translation,
we usually construct a prompt as follows: “[INST] Translate the following
sentence from English to German: $S$ [/INST]”, where $S$ is the source
sentence. LLM then provides the translation in the content completed after
“[/INST]”. The completed translation can be denoted as $T$.

In simultaneous translation, we keep the content of the instruction unchanged
and consider the source text as a time-dependent variable-length sequence
$S_{t}$. Additionally, we treat the accumulated translation content as another
variable-length sequence $T_{t}$. At this point, the model’s input is time-
dependent, and we define $X_{t}$ as the input to the model at time step $t$.
$X_{t}$ can be obtained through the prompting function
$X_{t}=\phi(S_{t},T_{t})$, which puts $S_{t}$ and $T_{t}$ in the same sequence
starting with the instruction: “[INST] Translate the following sentence from
English to German: $S_{t}$ [/INST] $T_{t}$”. By employing this approach, we
can effectively manage the ongoing source and target content separately and
structure them into standardized prompts (line 6 in Algo 1).

###  2.2 Mixture Policy

Regarding the policy for reading and writing, we introduce a hybrid policy
based on “wait-k” [4] and incremental decoding [8, 9, 10]. Formally, we define
the policy function and action as $a_{t}=\pi(S_{t},T_{t},t),$ where
$a_{t}\in\\{\mathbb{R},\mathbb{W}\\}$. When $a_{t}=\mathbb{R}$, the system
reads the latest source token and appends it to $S_{t}$. When
$a_{t}=\mathbb{W}$, the system returns the latest $l$ target tokens and
appends them to $T_{t}$. A detailed illustration is shown in Algo 1 and Figure
1.

####  2.2.1 Reading Policy

The decision-making process for the read action primarily depends on two
hyperparameters, $k$ and $n$. $k$ represents the initial waiting steps,
indicating that the system needs to read at least $k$ source tokens before
commencing translation [4]. $n$ represents the number of tokens the model has
to read at once before it is allowed to take write action (line 2 in Algo 1).
Given that LLMs typically require more computational resources for inference
compared to traditional simultaneous translation models, we opt to reduce
computational latency by reading $n$ tokens consecutively at once, thus
minimizing the number of model invocations.

####  2.2.2 Writing policy

Regarding the decision-making process for the write action, we employ the
incremental decoding approach and utilize the Longest Common Prefix (LCP)
algorithm to identify a sufficiently confident translation prefix. This
approach has been used in [8, 9, 10] and evolved into multiple variants, which
has been proved to have promising performance. Our approach is mainly based on
[10]. Specifically, we employ beam search to allow the model to generate $B$
candidate translations and subsequently use the LCP algorithm to find a common
prefix with the local agreement (LA) as the translation output for this write
action.

However, candidates generated by LLMs during beam search decoding may still
exhibit diversity, making it challenging for the LCP algorithm to identify a
prefix with LA, resulting in significant latency. To address this problem, we
optimize the LCP algorithm and introduce the Relaxed Agreement Longest Common
Prefix (RALCP) algorithm . RALCP employs a voting mechanism to relax the
constraints on identifying the common prefix. For example, if 80% of the
candidates can propose the same token, then that token is accepted as a part
of the prefix. We denote $\gamma$ as the agreement threshold, which is
considered as the threshold of accepting the most frequent token at the
certain position. Specifically, in conventional LCP, the prefix with local
agreement is located by matching the token at the same position $i$ for all
candidate sequences, if they are holding the same token, the token will be
gathered into the prefix. In RALCP, we relaxed the criteria of selecting the
token by employing the voting mechanism, i.e. if the token at $i$ has the
normalized votes (frequency) larger than $\gamma$, it will be accepted in the
prefix. In our experiments, we explored $\gamma$ ranging from 0.1 to 1.0 and
found that 0.6 is an empirically balanced value.

###  2.3 SFT and Prefix Training

Although the mixture policy already equips the LLM with the ability to perform
simultaneous translation. To achieve even better results, we can explore the
use of SFT to enhance the performance further. Thus, we finetune the LLM with
LoRA [12] in a manner exactly the same as offline translation. Specifically,
we put the source and target sentence into the prompt mentioned in §2.1, and
only compute loss on target text to avoid catastrophic forgetting. Meanwhile,
we only tune 1 epoch on the combined training set of all nine languages.

By analyzing the model’s output we observed that current policy can, to some
extent, mitigate hallucinations caused by incomplete source contexts, but the
model may still produce local agreed incorrect translations. This often
manifests as the model attempting to complete the target to form a complete
sentence, even if the completed part was not mentioned in the source sentence.

Inspired by [8], we constructed a small amount of prefix-to-prefix data,
aiming to mitigate the model’s tendency to attempt completion by employing
them in SFT. Specifically, we randomly sampled 1000 source sentences from the
training set of each language pair and truncated them into 20% to 80% of the
full length uniformly, resulting in 9000 source prefixes. We then used ChatGPT
to translate these source prefixes, thus obtaining target prefixes. These
prefix pairs are mixed into the combined multilingual training set with full
sentences inside.

##  3 Experiment

###  3.1 Experimental Setup

Model | en-cs | en-de | en-es | en-fr | en-it | en-nl | en-pt | en-ro | en-ru
---|---|---|---|---|---|---|---|---|---
Offline-Transformer (b=5) | 22.29 | 30.65 | 35.08 | 42.91 | 31.46 | 34.91 | 38.05 | 29.58 | 20.09
LLM-One-Shot (b=5) | 10.37 | 21.79 | 27.40 | 31.25 | 19.71 | 23.80 | 23.87 | 15.44 | 13.40
LLM-SFT (b=5) | 20.47 | 30.73 | 36.43 | 42.77 | 32.05 | 34.51 | 37.58 | 27.45 | 20.65
LLM-PFX-SFT (b=5) | 20.73 | 30.93 | 36.47 | 42.89 | 31.91 | 33.87 | 37.66 | 27.15 | 21.02
Wait-5-Transformer (k=5) | 10.77 (7.59) | 15.44 (6.17) | 18.94 (6.10) | 24.45 (6.75) | 16.12 (6.41) | 18.69 (6.35) | 19.87 (6.64) | 15.26 (7.66) | 10.22 (6.96)
Incre-Dec-Trans (b=5, k=3, n=3) | 17.33 (4.89) | 26.47 (5.26) | 31.64 (6.22) | 38.85 (5.93) | 28.30 (6.35) | 31.14 (6.11) | 33.80 (6.34) | 26.62 (6.89) | 16.81 (5.54)
Incre-Dec-Trans (b=10, k=6, n=6) | 19.82 (7.32) | 28.63 (7.73) | 33.16 (8.48) | 41.53 (8.34) | 29.70 (8.68) | 33.31 (8.54) | 36.17 (9.06) | 28.51 (8.95) | 18.48 (8.03)
LLM-One-Shot (b=5, k=3, n=3) | 10.63 (4.07) | 19.10 (3.81) | 24.48 (3.92) | 28.57 (4.03) | 17.12 (4.03) | 20.89 (3.71) | 21.86 (4.03) | 14.21 (4.08) | 12.63 (4.12)
LLM-SFT (b=5, k=3, n=3) | 19.09 (4.02) | 28.31 (4.07) | 33.82 (4.15) | 41.23 (4.19) | 29.46 (4.24) | 30.87 (3.92) | 35.05 (4.38) | 25.67 (4.30) | 18.29 (4.05)
LLM-PFX-SFT (b=5, k=3, n=3) | 19.80 (4.21) | 28.80 (4.15) | 33.86 (4.40) | 41.34 (4.29) | 29.07 (4.36) | 31.46 (3.99) | 34.87 (4.41) | 25.89 (4.40) | 19.21 (4.29)
LLM-PFX-SFT (b=10, k=6, n=6) | 21.31 (7.38) | 31.06 (7.31) | 36.34 (7.72) | 42.59 (7.61) | 31.53 (7.72) | 33.92 (7.08) | 37.56 (8.03) | 27.03 (7.91) | 20.66 (7.82)

Table 1: This table presents the overall result of our experiments (b=beam
size, k=wait-k, n=read-n). The first group is the performance of offline
setting for Transformer and LLM (Llama-2-7B-chat) with one-shot, with SFT
only, and with SFT+Prefix training (PFX). The second group contains baseline
simultaneous NMT models including a wait-5 Transformer [4] and the offline
Transformer applied with incremental decoding [10]. The third group presents
the simultaneous decoding results evaluated with our framework applied with
LLM, including the one-shot, SFT only and SFT+prefix training (PFX) setting.
The metrics are annotated as BLEU for offline results and BLEU (LAAL) for
streaming results. Best performed (in terms of BLEU) settings are bolded.

We selected nine language pairs from the MUST-C [13] dataset, which has been
commonly used in the evaluation of the performance of speech and text
translation systems. These nine language pairs all have English as the source
language and consist of TED talk speech utterances. Each language pair
contains between 100k to 200k training samples and over 2,000 test samples.
During training, the combined training set has a total number of 1.9M samples
(with an additional 9000 prefix samples for prefix training). We used the tst-
COMMON test set for evaluation. For evaluation metrics, BLEU [14] is used for
evaluating quality and LAAL [15] is used for evaluating latency. All
evaluations are conducted with the SimulEval toolkit [16].

We used Llama2-7B-chat as our LLM [17]. During SFT, LoRA adapters were
configured with $r=64$ and $\alpha=16$, thus having the total trainable
parameters to be 33M. We set the learning rate to 2e-4, the batch size to 48,
and employed 4-bit quantization. A single A100 GPU is used to perform SFT for
all settings only for one epoch.

We established two baseline models: 1) An offline Transformer [11] model
trained (complete sentence pair training set same with LLM, but with source
sentences prepended with a language tag) for 300k step with 16k tokens per
batch on 4 A40 GPUs. We followed the implementation of [10] for incremental
decoding inference but changed the reading policy proposed in our paper for
text input. 2) A “wait-k” Transformer [4] with a fixed $k$ set as 5 trained
with the same configuration to (1). We followed the implementation of [5] for
inference. To ensure a fair comparison, both baseline models contained 48M
parameters, matching the learnable parameter size to LLM’s LoRA adapters.

###  3.2 Experimental Results

Fig. 2: The correlation between BLEU and LAAL under different scale of
$\gamma$ in RALCP.

Our experiments are divided into three groups: 1) We evaluated the performance
of our proposed approach for the LLM under the one-shot setting (We found that
LLM under zero-shot setting often generates unexpected format in the response,
thus chose to use the one-shot setting), serving as the baseline performance
for this framework. 2) We conducted SFT on Llama2-7B-chat models using
complete sentence pairs to assess the improvement brought by SFT. 3) We
performed SFT using a mixture of ChatGPT-generated prefix pairs and complete
sentence pairs to examine the effectiveness of prefix training.

From Table 1, the following findings are evident: 1) In the offline scenario,
LLM exhibits a considerable gap compared to specially trained NMT models in
the one-shot setting. However, after SFT, this gap is narrowed, and in some
languages, LLM even surpasses NMT models. The introduction of prefix data does
not lead to a significant change in performance. 2) In the streaming scenario,
the one-shot performance is similar to its offline counterpart. Upon analyzing
the content of the model’s output, we observed that this may be due to LLM
attempting to generate responses learned from chat tasks, such as “Here is the
translation”, which can affect the final BLEU score. 3) LLM after SFT
outperforms baseline models using “wait-k” and incremental decoding in most
languages, with performance close to offline decoding. 4) The addition of
prefix data results in an average 1.3% improvement in BLEU but leads to an
average 3.2% increase in latency. 5) When using larger beam sizes, $n$, and
$k$, the model significantly outperforms incremental decoding baseline in
languages other than en-ro and exhibits lower latency. In terms of the
agreement threshold $\gamma$ in RALCP, we studied the influence of it to the
BLEU and LAAL with the en-de validation set, as shown in Figure 2. We found
that 0.6 is an empirically balanced choice, thus used under all settings in
our experiments. Clearly, compared with conventional LCP ($\gamma=1.0$), RALCP
can have better balancing on the performance and latency. However,
theoretically, the chosen of $\gamma$ should be language correlated, but due
to the limitation of computational resources, we use 0.6 for all language
pairs.

##  4 Conclusion

In this paper, we introduced the Mixture policy that enables seamless
adaptation of LLMs like Llama-2-7B-chat to simultaneous translation tasks.
Experimental results demonstrate that this policy allows LLMs to achieve their
intrinsic one-shot offline translation performance during simultaneous
decoding. After performing SFT, the models can outperform other dedicated
simultaneous translation models while exhibiting lower latency. By employing
prefix training, the model can achieve slight performance improvements in low-
latency scenarios. In future work, we plan to validate this approach across a
wider range of LLMs and languages and explore its integration with speech
modalities.

## References

  * [1] Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla,  “How good are GPT models at machine translation? A comprehensive evaluation,”  CoRR, vol. abs/2302.09210, 2023.
  * [2] Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang,  “Multilingual machine translation with large language models: Empirical results and analysis,”  CoRR, vol. abs/2304.04675, 2023.
  * [3] Jiatao Gu, Graham Neubig, Kyunghyun Cho, and Victor O. K. Li,  “Learning to translate in real-time with neural machine translation,”  in EACL, 2017, pp. 1053–1062.
  * [4] Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang,  “STACL: simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework,”  in ACL, 2019, pp. 3025–3036.
  * [5] Minghan Wang, Jiaxin Guo, Yinglu Li, Xiaosong Qiao, Yuxia Wang, Zongyao Li, Chang Su, Yimeng Chen, Min Zhang, Shimin Tao, Hao Yang, and Ying Qin,  “The hw-tsc’s simultaneous speech translation system for IWSLT 2022 evaluation,”  in IWSLT@ACL, 2022, pp. 247–254.
  * [6] Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu,  “Monotonic multihead attention,”  in ICLR, 2020.
  * [7] Philip Arthur, Trevor Cohn, and Gholamreza Haffari,  “Learning coupled policies for simultaneous machine translation using imitation learning,”  in EACL, 2021, pp. 2709–2719.
  * [8] Danni Liu, Gerasimos Spanakis, and Jan Niehues,  “Low-latency sequence-to-sequence speech recognition and translation by partial hypothesis selection,”  in Interspeech, 2020, pp. 3620–3624.
  * [9] Peter Polák, Ngoc-Quan Pham, Tuan-Nam Nguyen, Danni Liu, Carlos Mullov, Jan Niehues, Ondrej Bojar, and Alexander Waibel,  “CUNI-KIT system for simultaneous speech translation task at IWSLT 2022,”  in IWSLT@ACL, 2022, pp. 277–285.
  * [10] Jiaxin Guo, Daimeng Wei, Zhanglin Wu, Zongyao Li, Zhiqiang Rao, Minghan Wang, Hengchao Shang, Xiaoyu Chen, Zhengzhe Yu, Shaojun Li, Yuhao Xie, Lizhi Lei, and Hao Yang,  “The hw-tsc’s simultaneous speech-to-text translation system for IWSLT 2023 evaluation,”  in IWSLT@ACL, 2023, pp. 376–382.
  * [11] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  in NeurlPS, 2017, pp. 5998–6008.
  * [12] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen,  “Lora: Low-rank adaptation of large language models,”  in ICLR, 2022.
  * [13] Mattia Antonino Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi,  “Must-c: a multilingual speech translation corpus,”  in NAACL-HLT, 2019, pp. 2012–2017.
  * [14] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu,  “Bleu: a method for automatic evaluation of machine translation,”  in ACL, 2002, pp. 311–318.
  * [15] Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi,  “Over-generation cannot be rewarded: Length-adaptive average lagging for simultaneous speech translation,”  CoRR, vol. abs/2206.05807, 2022.
  * [16] Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, and Juan Miguel Pino,  “SIMULEVAL: an evaluation toolkit for simultaneous translation,”  in EMNLP, 2020, pp. 144–150.
  * [17] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom,  “Llama 2: Open foundation and fine-tuned chat models,”  CoRR, vol. abs/2307.09288, 2023.