# New Intent Discovery with Pre-training and Contrastive Learning

Yuwei Zhang2 Haode Zhang1 Li-Ming Zhan1 Xiao-Ming Wu1
Albert Y.S. Lam3
Department of Computing, The Hong Kong Polytechnic University, Hong Kong
S.A.R.1
University of California, San Diego2
Fano Labs, Hong Kong S.A.R.3
<EMAIL_ADDRESS>
{haode.zhang<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Work done while the author was with HK PolyU. Corresponding author.

###### Abstract

New intent discovery aims to uncover novel intent categories from user
utterances to expand the set of supported intent classes. It is a critical
task for the development and service expansion of a practical dialogue system.
Despite its importance, this problem remains under-explored in the literature.
Existing approaches typically rely on a large amount of labeled utterances and
employ pseudo-labeling methods for representation learning and clustering,
which are label-intensive, inefficient, and inaccurate. In this paper, we
provide new solutions to two important research questions for new intent
discovery: (1) how to learn semantic utterance representations and (2) how to
better cluster utterances. Particularly, we first propose a multi-task pre-
training strategy to leverage rich unlabeled data along with external labeled
data for representation learning. Then, we design a new contrastive loss to
exploit self-supervisory signals in unlabeled data for clustering. Extensive
experiments on three intent recognition benchmarks demonstrate the high
effectiveness of our proposed method, which outperforms state-of-the-art
methods by a large margin in both unsupervised and semi-supervised scenarios.
The source code will be available at https://github.com/zhang-yu-wei/MTP-CLNN.

##  1 Introduction

Why Study New Intent Discovery (NID)? Recent years have witnessed the rapid
growth of conversational AI applications. To design a natural language
understanding system, a set of expected customer intentions are collected
beforehand to train an intent recognition model. However, the pre-defined
intents cannot fully meet customer needs. This implies the necessity of
expanding the intent recognition model by repeatedly integrating new intents
discovered from unlabeled user utterances (Fig. 1). To reduce the effort in
manually identifying unknown intents from a mass of utterances, previous works
commonly employ clustering algorithms to group utterances of similar intents
(Cheung and Li, 2012; Hakkani-Tür et al., 2015; Padmasundari, 2018). The
cluster assignments thereafter can either be directly used as new intent
labels or as heuristics for faster annotations.

Figure 1: New Intent Discovery.

Research Questions (RQ) and Challenges. Current study of NID centers around
two basic research questions: _1) How to learn semantic utterance
representations to provide proper cues for clustering? 2) How to better
cluster the utterances?_ The study of the two questions are often interwoven
in existing research. Utterances can be represented according to different
aspects such as the style of language, the related topics, or even the length
of sentences. It is important to learn semantic utterance representations to
provide proper cues for clustering. Simply applying a vanilla pre-trained
language model (PLM) to generate utterance representations is not a viable
solution, which leads to poor performance on NID as shown by the experimental
results in Section 4.2. Some recent works proposed to use labeled utterances
of known intents for representation learning (Forman et al., 2015; Haponchyk
et al., 2018; Lin et al., 2020; Zhang et al., 2021c; Haponchyk and Moschitti,
2021), but they require a substantial amount of known intents and sufficient
labeled utterances of each intent, which are not always available especially
at the early development stage of a dialogue system. Further, pseudo-labeling
approaches are often exploited to generate supervision signals for
representation learning and clustering. For example, Lin et al. (2020)
finetune a PLM with an utterance similarity prediction task on labeled
utterances to guide the training of unlabeled data with pseudo-labels. Zhang
et al. (2021c) adopt a deep clustering method (Caron et al., 2018) that uses
$k$-means clustering to produce pseudo-labels. However, pseudo-labels are
often noisy and can lead to error propagation.

Our Solutions. In this work, we propose a simple yet effective solution for
each research question. Solution to RQ 1: multi-task pre-training. We propose
a multi-task pre-training strategy that takes advantage of both external data
and internal data for representation learning. Specifically, we leverage
publicly available, high-quality intent detection datasets, following Zhang et
al. (2021d), as well as the provided labeled and unlabeled utterances in the
current domain, to fine-tune a pre-trained PLM to learn task-specific
utterance representations for NID. The multi-task learning strategy enables
knowledge transfer from general intent detection tasks and adaptation to a
specific application domain. Solution to RQ 2: contrastive learning with
nearest neighbors. We propose to use a contrastive loss to produce compact
clusters, which is motivated by the recent success of contrastive learning in
both computer vision (Bachman et al., 2019; He et al., 2019; Chen et al.,
2020; Khosla et al., 2020) and natural language processing (Gunel et al.,
2021; Gao et al., 2021; Yan et al., 2021). Contrastive learning usually
maximizes the agreement between different views of the same example and
minimize that between different examples. However, the commonly used instance
discrimination task may push away false negatives and hurts the clustering
performance. Inspired by a recent work in computer vision (Van Gansbeke et
al., 2020), we introduce neighborhood relationship to customize the
contrastive loss for clustering in both unsupervised (i.e., without any
labeled utterances of known intents) and semi-supervised scenarios.
Intuitively, in a semantic feature space, neighboring utterances should have a
similar intent, and pulling together neighboring samples makes clusters more
compact.

Our main contributions are three-fold.

  * •

We show that our proposed multi-task pre-training method already leads to
large performance gains over state-of-the-art models for both unsupervised and
semi-supervised NID.

  * •

We propose a self-supervised clustering method for NID by incorporating
neighborhood relationship into the contrastive learning objective, which
further boosts performance.

  * •

We conduct extensive experiments and ablation studies on three benchmark
datasets to verify the effectiveness of our methods.

##  2 Related Works

New Intent Discovery. The study of NID is still in an early stage. Pioneering
works focus on unsupervised clustering methods. Shi et al. (2018) leveraged
auto-encoder to extract features. Perkins and Yang (2019) considered the
context of an utterance in a conversation. Chatterjee and Sengupta (2020)
proposed to improve density-based models. Some recent works Haponchyk et al.
(2018); Haponchyk and Moschitti (2021) studied supervised clustering
algorithms for intent labeling, yet it can not handle new intents. Another
line of works Forman et al. (2015); Lin et al. (2020); Zhang et al. (2021c)
investigated a more practical case where some known intents are provided to
support the discovery of unknown intents, which is often referred to as semi-
supervised NID.

To tackle semi-supervised NID, Lin et al. (2020) proposed to first perform
supervised training on known intents with a sentence similarity task and then
use pseudo labeling on unlabeled utterances to learn a better embedding space.
Zhang et al. (2021c) proposed to first pre-train on known intents and then
perform $k$-means clustering to assign pseudo labels on unlabeled data for
representation learning following Deep Clustering (Caron et al., 2018). They
also proposed to align clusters to accelerate the learning of top layers.
Another approach is to first classify the utterances as known and unknown and
then uncover new intents with the unknown utterances (Vedula et al., 2020;
Zhang et al., 2021b). Hence, it relies on accurate classification in the first
stage.

In this work, we address NID by proposing a multi-task pre-training method for
representation learning and a contrastive learning method for clustering. In
contrast to previous methods that rely on ample annotated data in the current
domain for pre-training, our method can be used in an unsupervised setting and
work well in data-scarce scenarios (Section 4.3).

Pre-training for Intent Recognition. Despite the effectiveness of large-scale
pre-trained language models (Radford and Narasimhan, 2018; Devlin et al.,
2019; Liu et al., 2019; Brown et al., 2020), the inherent mismatch in
linguistic behavior between the pre-training datasets and dialogues encourages
the research of continual pre-training on dialogue corpus. Most previous works
proposed to pre-train on open domain dialogues in a self-supervised manner
Mehri et al. (2020); Wu et al. (2020); Henderson et al. (2020); Hosseini-Asl
et al. (2020). Recently, several works pointed out that pre-training with
relavant tasks can be effective for intent recognition. For example, Zhang et
al. (2020) formulated intent recognition as a sentence similarity task and
pre-trained on natural language inference (NLI) datasets. Vulić et al. (2021);
Zhang et al. (2021e) pre-trained with a contrastive loss on intent detection
tasks. Our multi-task pre-training method is inspired from Zhang et al.
(2021d) which leverages publicly available intent datasets and unlabeled data
in the current domain for pre-training to improve the performance of few-shot
intent detection. However, we argue that the method is more suitable to be
applied for NID due to the natural existence of unlabeled utterances.

Contrastive Representation Learning. Contrastive learning has shown promising
results in computer vision (Bachman et al., 2019; Chen et al., 2020; He et
al., 2019; Khosla et al., 2020) and gained popularity in natural language
processing. Some recent works used unsupervised contrastive learning to learn
sentence embeddings (Gao et al., 2021; Yan et al., 2021; Kim et al., 2021;
Giorgi et al., 2021). Specifically, Gao et al. (2021); Yan et al. (2021)
showed that contrastive loss can avoid an anisotropic embedding space. Kim et
al. (2021) proposed a self-guided contrastive training to improve the quality
of BERT representations. Giorgi et al. (2021) proposed to pre-train a
universal sentence encoder by contrasting a randomly sampled text segment from
nearby sentences. Zhang et al. (2021e) demonstrated that self-supervised
contrastive pre-training and supervised contrastive fine-tuning can benefit
few-shot intent recognition. Zhang et al. (2021a) showed that combining a
contrastive loss with a clustering objective can improve short text
clustering. Our proposed contrastive loss is tailored for clustering, which
encourages utterances with similar semantics to group together and avoids
pushing away false negatives as in the conventional contrastive loss.

Figure 2: The left part shows the overall workflow of our method where the
training order is indicated by the red arrow. The datasets and corresponding
loss functions used in each training stage are indicated by the black arrows.
The right part illustrates a simple example of CLNN. A batch of four training
instances $\\{x_{i}\\}_{i=1}^{4}$ (solid markers) and their respective
neighborhoods $\\{\mathcal{N}_{i}\\}_{i=1}^{4}$ are plotted (hollow markers
within large circles). Since $x_{2}$ falls within $\mathcal{N}_{1}$, $x_{2}$
along with its neighbors are taken as positive instance for $x_{1}$ (but not
vice versa since $x_{1}$ is not in $\mathcal{N}_{2}$). We also show an example
of adjacency matrix $\bf{A}^{\prime}$ and augmented batch
$\mathcal{B}^{\prime}$. The pairwise relationships with the first instance in
the batch are plotted with solid lines indicating positive pairs and dashed
lines indicating negative pairs.

##  3 Method

Problem Statement. To develop an intent recognition model, we usually prepare
a set of expected intents $\mathcal{C}_{k}$ along with a few annotated
utterances
$\mathcal{D}^{\text{labeled}}_{\text{known}}=\\{(x_{i},y_{i})|y_{i}\in\mathcal{C}_{k}\\}$
for each intent. After deployed, the system will encounter utterances
$\mathcal{D}^{\text{unlabeled}}=\\{x_{i}|y_{i}\in\\{\mathcal{C}_{k},\mathcal{C}_{u}\\}\\}$
from both pre-defined (known) intents $\mathcal{C}_{k}$ and unknown intents
$\mathcal{C}_{u}$. The aim of new intent discovery (NID) is to identify the
emerging intents $\mathcal{C}_{u}$ in $\mathcal{D}^{\text{unlabeled}}$. NID
can be viewed as a direct extension of out-of-distribution (OOD) detection,
where we not only need to identify OOD examples but also discover the
underlying clusters. NID is also different from zero-shot learning in that we
do not presume access to any kind of class information during training. In
this work, we consider both unsupervised and semi-supervised NID, which are
distinguished by the existence of
$\mathcal{D}_{\text{known}}^{\text{labeled}}$, following Zhang et al. (2021c).

Overview of Our Approach. As shown in Fig. 2, we propose a two-stage framework
that addresses the research questions mentioned in Sec. 1. In the first stage,
we perform multi-task pre-training (MTP) that jointly optimizes a cross-
entropy loss on external labeled data and a self-supervised loss on target
unlabeled data (Sec. 3.1). In the second stage, we first mine top-$K$ nearest
neighbors of each training instance in the embedding space and then perform
contrastive learning with nearest neighbors (CLNN) (Sec. 3.2). After training,
we employ a simple non-parametric clustering algorithm to obtain clustering
results.

###  3.1 Stage 1: Multi-task Pre-training (MTP)

We propose a multi-task pre-training objective that combines a classification
task on external data from publicly available intent detection datasets and a
self-supervised learning task on internal data from the current domain.
Different from previous works (Lin et al., 2020; Zhang et al., 2021c), our
pre-training method does not rely on annotated data
($\mathcal{D}^{\text{labeled}}_{\text{known}}$) from the current domain and
hence can be applied in an unsupervised setting.

Specifically, we first initialize the model with a pre-trained BERT encoder
(Devlin et al., 2019). Then, we employ a joint pre-training loss as in Zhang
et al. (2021d). The loss consists of a cross-entropy loss on external labeled
data and a masked language modelling (MLM) loss on all available data from the
current domain:

$\mathcal{L}_{\text{stg1}}=\underbrace{\mathcal{L}_{\text{ce}}(\mathcal{D}^{\text{labeled}}_{\text{external}};\theta)}_{\text{supervised}}+\underbrace{\mathcal{L}_{\text{mlm}}(\mathcal{D}^{\text{all}}_{{\text{internal}}};\theta)}_{\text{self-
supervised}},$ (1)

where $\theta$ are model parameters. For the supervised classification task,
we leverage an external public intent dataset with diverse domains (e.g.,
CLINC150 (Larson et al., 2019)), denoted as
$\mathcal{D}^{\text{labeled}}_{\text{external}}$, following Zhang et al.
(2021d). For the self-supervised MLM task, we use all available data (labeled
or unlabeled) from the current domain, denoted as
$\mathcal{D}^{\text{all}}_{\text{internal}}$.

Intuitively, the classification task aims to learn general knowledge of intent
recognition with annotated utterances in external intent datasets, while the
self-supervised task learns domain-specific semantics with utterances
collected in the current domain. Together, they enable learning semantic
utterance representations to provide proper cues for the subsequent clustering
task. As will be shown in Sec. 4.3, both tasks are essential for NID.

For semi-supervised NID, we can further utilize the annotated data in the
current domain to conduct continual pre-training, by replacing
$\mathcal{D}^{\text{labeled}}_{\text{external}}$ in Eq. 1 to
$\mathcal{D}^{\text{labeled}}_{\text{known}}$. This step is not included in
unsupervised NID.

###  3.2 Stage 2: Contrastive Learning with Nearest Neighbors (CLNN)

In the second stage, we propose a contrastive learning objective that pulls
together neighboring instances and pushes away distant ones in the embedding
space to learn compact representations for clustering. Concretely, we first
encode the utterances with the pre-trained model from stage 1. Then, for each
utterance $x_{i}$, we search for its top-$K$ nearest neighbors in the
embedding space using inner product as distance metric to form a neighborhood
$\mathcal{N}_{i}$. The utterances in $\mathcal{N}_{i}$ are supposed to share a
similar intent as $x_{i}$. During training, we sample a minibatch of
utterances $\mathcal{B}=\\{x_{i}\\}_{i=1}^{M}$. For each utterance
$x_{i}\in\mathcal{B}$, we uniformly sample one neighbor $x^{\prime}_{i}$ from
its neighborhood $\mathcal{N}_{i}$. We then use data augmentation to generate
$\tilde{x}_{i}$ and $\tilde{x}^{\prime}_{i}$ for $x_{i}$ and $x^{\prime}_{i}$
respectively. Here, we treat $\tilde{x}_{i}$ and $\tilde{x}^{\prime}_{i}$ as
two views of $x_{i}$, which form a positive pair. We then obtain an augmented
batch
$\mathcal{B}^{\prime}=\\{\tilde{x}_{i},\tilde{x}^{\prime}_{i}\\}_{i=1}^{M}$
with all the generated samples. To compute contrastive loss, we construct an
adjacency matrix $\bm{A}^{\prime}$ for $\mathcal{B}^{\prime}$, which is a
$2M\times 2M$ binary matrix where $1$ indicates positive relation (either
being neighbors or having the same intent label in semi-supervised NID) and 0
indicates negative relation. Hence, we can write the contrastive loss as:

$l_{i}=-\frac{1}{|\mathcal{C}_{i}|}\sum_{j\in\mathcal{C}_{i}}\log\frac{\exp(\text{sim}(\tilde{h}_{i},\tilde{h}_{j})/\tau)}{\sum_{{k\neq
i}}^{2M}\exp(\text{sim}(\tilde{h}_{i},\tilde{h}_{k})/\tau)},$ (2)
$\mathcal{L}_{\text{stg2}}=\frac{1}{2M}\sum_{i=1}^{2M}l_{i},$ (3)

where $\mathcal{C}_{i}\equiv\\{\bm{A}^{\prime}_{i,j}=1|j\in\\{1,...,2M\\}\\}$
denotes the set of instances having positive relation with $\tilde{x}_{i}$ and
$|\mathcal{C}_{i}|$ is the cardinality. $\tilde{h}_{i}$ is the embedding for
utterance $\tilde{x}_{i}$. $\tau$ is the temperature parameter.
$\text{sim}(\cdot,\cdot)$ is a similarity function (e.g., dot product) on a
pair of normalized feature vectors. During training, the neighborhood will be
updated every few epochs. We implement the contrastive loss following Khosla
et al. (2020).

Notice that the main difference between Eq. 2 and conventional contrastive
loss is how we construct the set of positive instances $\mathcal{C}_{i}$.
Conventional contrastive loss can be regarded as a special case of Eq. 2 with
neighborhood size $K=0$ and the same instance is augmented twice to form a
positive pair (Chen et al., 2020). After contrastive learning, a non-
parametric clustering algorithm such as $k$-means can be applied to obtain
cluster assignments.

Data Augmentation. Strong data augmentation has been shown to be beneficial in
contrastive learning (Chen et al., 2020). We find that it is inefficient to
directly apply existing data augmentation methods such as EDA (Wei and Zou,
2019), which are designed for general sentence embedding. We observe that the
intent of an utterance can be expressed by only a small subset of words such
as “suggest restaurant” or “book a flight”. While it is hard to identify the
keywords for an unlabeled utterance, randomly replacing a small amount of
tokens in it with some random tokens from the library will not affect intent
semantics much. This approach works well in our experiments (See Table 5 RTR).

Advantages of CLNN. By introducing the notion of neighborhood relationship in
contrastive learning, CLNN can 1) pull together similar instances and push
away dissimilar ones to obtain more compact clusters; 2) utilize proximity in
the embedding space rather than assigning noisy pseudo-labels (Van Gansbeke et
al., 2020); 3) directly optimize in the feature space rather than clustering
logits as in Van Gansbeke et al. (2020), which has been proven to be more
effective by Rebuffi et al. (2020); and 4) naturally incorporate known intents
with the adjacency matrix.

##  4 Experiment

###  4.1 Experimental Details

Datasets. We evaluate our proposed method on three popular intent recognition
benchmarks. BANKING (Casanueva et al., 2020) is a fine-grained dataset with
$77$ intents collected from banking dialogues, StackOverflow (Xu et al., 2015)
is a large scale dataset collected from online queries, M-CID (Arora et al.,
2020) is a smaller dataset collected for Covid-19 services. We choose CLINC150
(Larson et al., 2019) as our external public intent dataset in stage 1 due to
its high-quality annotations and coverage of diverse domains. The dataset
statistics are summarized in Table 1. We use the same splits of BANKING and
StackOverflow as in Zhang et al. (2021b). Details about dataset splitting are
provided in the Appendix.

Dataset | domain | #Intents | #Utterances
---|---|---|---
CLINC150 | general | 120 | 18,000
BANKING | banking | 77 | 13,083
StackOverflow | questions | 20 | 20,000
M-CID | covid-19 | 16 | 1,745
Table 1: Dataset statistics.

| BANKING | StackOverflow | M-CID
---|---|---|---
| Methods | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI | ACC
unsupervised | GloVe-KM | 48.75 | 12.74 | 27.92 | 21.79 | 4.54 | 24.26 | 46.40 | 35.57 | 46.99
GloVe-AG | 52.76 | 14.41 | 31.18 | 23.45 | 4.85 | 24.48 | 51.23 | 32.57 | 42.35
SAE-KM | 60.12 | 24.00 | 37.38 | 48.72 | 23.36 | 37.16 | 51.03 | 43.51 | 52.95
SAE-DEC | 62.92 | 25.68 | 39.35 | 61.32 | 21.17 | 57.09 | 50.69 | 44.52 | 53.07
SAE-DCN | 62.94 | 25.69 | 39.36 | 61.34 | 34.98 | 57.09 | 50.69 | 44.52 | 53.07
BERT-KM | 36.38 | 5.38 | 16.27 | 11.60 | 1.60 | 13.85 | 37.37 | 14.02 | 33.81
MTP (Ours) | 77.32 | 47.33 | 57.99 | 63.85 | 48.71 | 66.18 | 72.40 | 53.04 | 68.94
MTP-CLNN (Ours) | 81.80 | 55.75 | 65.90 | 78.71 | 67.63 | 81.43 | 79.95 | 66.71 | 79.14

Table 2:  Performance on unsupervised NID. For each dataset, the best results
are marked in bold.

| BANKING | StackOverflow | M-CID
---|---|---|---
KCR | Methods | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI | ACC
25% | BERT-DTC | 56.05 | 20.19 | 32.91 | 33.38 | 16.45 | 30.32 | 36.00 | 13.64 | 27.51
BERT-KCL | 53.85 | 20.07 | 28.79 | 35.47 | 16.80 | 32.88 | 29.35 | 11.58 | 24.76
BERT-MCL | 49.46 | 15.51 | 24.53 | 29.44 | 14.99 | 31.50 | 31.16 | 11.30 | 26.13
CDAC+ | 67.65 | 34.88 | 48.79 | 74.33 | 39.44 | 74.30 | 43.89 | 19.65 | 39.37
DAC | 69.85 | 37.16 | 49.67 | 53.97 | 36.46 | 53.96 | 49.83 | 27.21 | 43.72
MTP (Ours) | 80.00 | 51.86 | 62.75 | 73.75 | 61.06 | 75.98 | 72.40 | 53.04 | 68.94
MTP-DAC (Comb) | 81.48 | 55.64 | 66.12 | 77.22 | 61.42 | 78.60 | 77.79 | 62.88 | 77.02
MTP-CLNN (Ours) | 84.11 | 61.29 | 71.43 | 79.68 | 70.17 | 83.77 | 80.24 | 66.77 | 79.20
50% | BERT-DTC | 69.68 | 35.98 | 48.87 | 53.94 | 36.79 | 51.78 | 51.90 | 28.94 | 44.70
BERT-KCL | 62.86 | 30.16 | 40.81 | 57.63 | 41.90 | 56.58 | 42.48 | 22.83 | 38.11
BERT-MCL | 62.50 | 29.80 | 42.28 | 49.49 | 35.96 | 53.16 | 41.50 | 21.46 | 37.99
CDAC+ | 70.62 | 38.61 | 51.97 | 76.18 | 41.92 | 76.30 | 50.47 | 26.01 | 46.65
DAC | 76.41 | 47.28 | 59.32 | 70.78 | 56.44 | 73.76 | 63.27 | 43.52 | 57.19
MTP (Ours) | 82.92 | 58.46 | 68.29 | 77.11 | 66.45 | 79.28 | 72.40 | 53.04 | 68.94
MTP-DAC (Comb) | 83.43 | 59.78 | 70.42 | 78.91 | 67.37 | 81.27 | 78.17 | 63.41 | 77.68
MTP-CLNN (Ours) | 85.62 | 64.93 | 75.23 | 81.03 | 73.02 | 85.64 | 79.48 | 65.71 | 77.85
75% | BERT-DTC | 74.51 | 44.57 | 57.34 | 67.02 | 55.14 | 71.14 | 60.82 | 38.62 | 55.42
BERT-KCL | 72.18 | 44.29 | 58.70 | 70.38 | 57.98 | 71.50 | 54.22 | 34.60 | 52.15
BERT-MCL | 74.41 | 48.08 | 61.57 | 67.72 | 55.78 | 70.82 | 51.33 | 31.22 | 50.77
CDAC+ | 71.76 | 40.68 | 53.46 | 76.68 | 43.97 | 75.34 | 55.06 | 32.52 | 53.70
DAC | 79.99 | 54.57 | 65.87 | 75.31 | 60.02 | 78.84 | 71.41 | 54.22 | 69.11
MTP (Ours) | 85.17 | 64.37 | 74.20 | 80.70 | 71.68 | 83.74 | 80.95 | 69.27 | 80.92
MTP-DAC (Comb) | 85.78 | 65.28 | 75.43 | 80.89 | 71.17 | 84.20 | 80.94 | 68.27 | 80.89
MTP-CLNN (Ours) | 87.52 | 70.00 | 79.74 | 82.56 | 75.66 | 87.63 | 83.75 | 73.22 | 84.36

Table 3:  Performance on semi-supervised NID with different known class ratio.
The LAR is set to $10\%$. For each dataset, the best results are marked in
bold. Comb denotes the baseline method combined with our proposed MTP.

(a) CDAC+

(b) DAC

(c) MTP (Ours)

(d) MTP-CLNN (Ours)

Figure 3: Visulization of embeddings on StackOverflow. $\text{KCR}=25\%$,
$\text{LAR}=10\%$. Best viewed in color.

(a)

(b)

(c)

Figure 4: Ablation study on the effectiveness of MTP. The LAR is set to 10%.
SUP stands for supervised pre-training on internal labeled data only. The
three columns correspond to results in the three metrics respectively.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: Analysis on the number of nearest neighbors in CLNN for unsupervised
NID. Vertical dashed lines correspond to our empirical estimations of optima.
Horizontal dashed lines represent the results of only training with MTP. When
the number of nearest neighbors is $0$, we simply augment the same instance
twice as in conventional contrastive learning Chen et al. (2020). The three
columns correspond to results in the three metrics respectively.

Experimental Setup. We evaluate our proposed method on both unsupervised and
semi-supervised NID. _Notice that in unsupervised NID, no labeled utterances
from the current domain are provided._ For clarity, we define two variables.
The proportion of known intents is defined as
$|\mathcal{C}_{k}|/(|\mathcal{C}_{k}|+|\mathcal{C}_{u}|)$ and referred to as
“known class ratio (KCR)”, and the proportion of labeled examples for each
known intent is denoted as “labeled ratio (LAR)”. The labeled data are
randomly sampled from the original training set. Notice that, $\text{KCR}=0$
means unsupervised NID, and $\text{KCR}>0$ means semi-supervised NID. In the
following sections, we provide experimental results for both unsupervised NID
and semi-supervised NID with $\text{KCR}=\\{25\%,50\%,75\%\\}$ and
$\text{LAR}=\\{10\%,50\%\\}$.

Evaluation Metrics. We adopt three popular evaluation metrics for clustering:
normalized mutual information (NMI), adjusted rand index (ARI), and accuracy
(ACC).

Baselines and Model Variants. We summarize the baselines compared in our
experiments for both unsupervised and semi-supervised NID. Our implementation
is based on Zhang et al. (2021b).111For fair comparison, the baselines are re-
run with TEXTOIR: https://github.com/thuiar/TEXTOIR, and hence some results
are different from those reported in Lin et al. (2020); Zhang et al. (2021c).

  * •

Unsupervised baselines. (1) GloVe-KM and (2) GloVe-AG are based on GloVe
(Pennington et al., 2014) embeddings and then evaluated with $k$-means
(MacQueen et al., 1967) or agglomerative clustering (Gowda, 1984)
respectively. (3) BERT-KM applies $k$-means on BERT embeddings. (4) SAE-KM
adopts $k$-means on embeddings of stacked auto-encoder. (5) Deep Embedding
Clustering (SAE-DEC) (Xie et al., 2016) and (6) Deep Clustering Network (SAE-
DCN) (Yang et al., 2017) are unsupervised clustering methods based on stacked
auto-encoder.

  * •

Semi-supervised baselines. (1) BERT-KCL (Hsu et al., 2018) and (2) BERT-MCL
(Hsu et al., 2019) employs pairwise similarity task for semi-supervised
clustering. (3) BERT-DTC (Han et al., 2019) extends DEC into semi-supervised
scenario. (4) CDAC+ (Lin et al., 2020) employs a pseudo-labeling process. (5)
Deep Aligned Clustering (DAC) (Zhang et al., 2021c) improves Deep Clustering
(Caron et al., 2018) by aligning clusters between iterations.

  * •

Our model variants include MTP and MTP-CLNN, which correspond to applying
$k$-means on utterance representations learned in stage 1 and stage 2
respectively. Further, we continue to train a DAC model on top of MTP to form
a stronger baseline MTP-DAC for semi-supervised NID.

Implementation. We take pre-trained bert-base-uncased model from Wolf et al.
(2019)222https://github.com/huggingface/transformers as our base model and we
use the [CLS] token as the BERT representation. For MTP, we first train until
convergence on the external dataset, and then when training on
$D^{\text{labeled}}_{\text{known}}$, we use a development set to validate
early-stopping with a patience of $20$ epochs following Zhang et al. (2021c).
For contrastive learning, we project a 768-d BERT embedding to an 128-d vector
with a two-layer MLP and set the temperature as $0.07$. For mining nearest
neighbors, we use the inner product method provided by Johnson et al.
(2017)333https://github.com/facebookresearch/faiss. We set neighborhood size
$K=50$ for BANKING and M-CID, and $K=500$ for StackOverflow, since we
empirically find that the optimal $K$ should be roughly half of the average
size of the training set for each class (see Section 4.4). The neighborhood is
updated every $5$ epochs. For data augmentation, the random token replacement
probability is set to $0.25$. For model optimization, we use the AdamW
provided by Wolf et al. (2019). In stage 1, the learning rate is set to
$5e^{-5}$. In stage 2, the learning rate is set to $1e^{-5}$ for BANKING and
M-CID, and $1e^{-6}$ for StackOverflow. The batch sizes are chosen based on
available GPU memory. All the experiments are conducted on a single RTX-3090
and averaged over $10$ different seeds. More details are provided in the
Appendix.

###  4.2 Result Analysis

Unsupervised NID. We show the results for unsupervised NID in Table 2. First,
comparing the performance of BERT-KM with GloVe-KM and SAE-KM, we observe that
BERT embedding performs worse on NID even though it achieves better
performance on NLP benchmarks such as GLUE, which manifests learning task-
specific knowledge is important for NID. Second, our proposed pre-training
method MTP improves upon baselines by a large margin. Take the NMI score of
BANKING for example, MTP outperforms the strongest baseline SAE-DCN by
$14.38\%$, which demonstrates the effectiveness of exploiting both external
public datasets and unlabeled internal utterances. Furthermore, MTP-CLNN
improves upon MTP by around $5\%$ in NMI, $10\%$ in ARI, and $10\%$ in ACC
across different datasets.

Semi-supervised NID. The results for semi-supervised NID are shown in Table 3.
First, MTP significantly outperforms the strongest baseline DAC in all
settings. For instance, on M-CID, MTP achieves $22.57\%$ improvement over DAC
in NMI. Moreover, MTP is less sensitive to the proportion of labeled classes.
From $\text{KCR}=75\%$ to $\text{KCR}=25\%$ on M-CID, MTP only drops $8.55\%$
in NMI, as opposed to about $21.58\%$ for DAC. The less performance decrease
indicates that our pre-training method is much more label-efficient.
Furthermore, with our proposed contrastive learning, MTP-CLNN consistently
outperforms MTP and the combined baseline MTP-DAC. Take BANKING with
$\text{KCR}=25\%$ for example, MTP-CLNN improves upon MTP by $4.11\%$ in NMI
while surpassing MTP-DAC by $2.63\%$. A similar trend can be observed when
$\text{LAR}=50\%$, and we provide the results in the Appendix.

Visualization. In Fig. 3, we show the t-SNE visualization of clusters with
embeddings learned by two strongest baselines and our methods. It clearly
shows the advantage of our methods, which can produce more compact clusters.
Results on other datasets can be found in the Appendix.

###  4.3 Ablation Study of MTP

To further illustrate the effectiveness of MTP, we conduct two ablation
studies in this section. First, we compare MTP with the pre-training method
employed in Zhang et al. (2021c), where only internal labeled data are
utilized for supervised pre-training (denoted as SUP).444Notice that we make a
simple modification to their pre-training to optimize the entire model rather
than the last few layers for fair comparison. In Fig. 4, we show the results
of both pre-training methods combined with CLNN with different proportions of
known classes. Notice that when $\text{KCR}=0$ there is no pre-training at all
for SUP-CLNN. It can be seen that MTP-CLNN consistently outperforms SUP-CLNN.
Furthermore, the performance gap increases while KCR decreases, and the
largest gap is achieved when $\text{KCR}=0$. This shows the high effectiveness
of our method in data-scarce scenarios.

| BANKING | StackOverflow | M-CID
---|---|---|---
Methods | NMI | ARI | NMI | ARI | NMI | ARI
PUB-CLNN | 75.69 | 44.58 | 42.22 | 24.77 | 73.64 | 56.94
MLM-CLNN | 77.02 | 47.79 | 78.62 | 68.77 | 71.28 | 53.28
MTP-CLNN | 81.80 | 55.75 | 78.71 | 67.63 | 79.95 | 66.71

Table 4:  Ablation study of MTP for unsupervised NID.

Second, we decompose MTP into two parts: supervised pre-training on external
public data (PUB) and self-supervised pre-training on internal unlabeled data
(MLM). We report the results of the two pre-training methods combined with
CLNN as well as MTP in Table 4. We can easily conclude that either PUB or MLM
is indispensable and multi-task pre-training is beneficial.

###  4.4 Analysis of CLNN

Number of Nearest Neighbors. We conduct an ablation study on neighborhood size
$K$ in Fig. 5. We can make two main observations. First, although the
performance of MTP-CLNN varies with different $K$, it still significantly
outperforms MTP (dashed horizontal line) for a wide range of $K$. For example,
MTP-CLNN is still better than MTP when $K=50$ on StackOverflow or $K=200$ on
BANKING. Second, despite the difficulty to search for $K$ with only unlabeled
data, we empirically find an effective estimation method, i.e. to choose $K$
as half of the average size of the training set for each class555We presume
prior knowledge of the number of clusters. There are some off-the-shelf
methods that can be directly applied in the embedding space to determine the
optimal number of clusters (Zhang et al., 2021c).. It can be seen that the
estimated $K\approx 60$ on BANKING and $K\approx 40$ on M-CID (vertical dashed
lines) lie in the optimal regions, which shows the effectiveness of our
empirical estimation method.

Exploration of Data Augmentation. We compare Random Token Replacement (RTR)
used in our experiments with other methods. For instance, dropout is applied
on embeddings to provide data augmentation in Gao et al. (2021), randomly
shuffling the order of input tokens is proven to be effective in Yan et al.
(2021), and EDA (Wei and Zou, 2019) is often applied in text classification.
Furthermore, we compare with a Stop-words Replacement (SWR) variant that only
replaces the stop-words with other random stop-words so it minimally affects
the intents of utterances. The results in Table 5 demonstrate that (1) RTR and
SWR consistently outperform others, which verifies our hypothesis in Section
3.2. (2) Surprisingly, RTR and SWR perform on par with each other. For
simplicity, we only report the results with RTR in the main experiments.

| BANKING | StackOverflow | M-CID
---|---|---|---
Methods | NMI | ARI | NMI | ARI | NMI | ARI
dropout | 79.52 | 50.83 | 75.60 | 57.67 | 79.64 | 66.14
shuffle | 79.02 | 49.72 | 75.70 | 58.95 | 79.68 | 66.09
EDA | 78.29 | 49.02 | 71.50 | 49.80 | 79.73 | 66.39
SWR(Ours) | 82.03 | 56.18 | 78.48 | 67.15 | 79.23 | 65.74
RTR(Ours)* | 81.80 | 55.75 | 78.71 | 67.63 | 79.95 | 66.71

Table 5:  Ablation study on data augmentation for unsupervised NID. * is the
method used in the main results.

##  5 Conclusion

We have provided simple and effective solutions for two fundamental research
questions for new intent discovery (NID): (1) how to learn better utterance
representations to provide proper cues for clustering and (2) how to better
cluster utterances in the representation space. In the first stage, we use a
multi-task pre-training strategy to exploit both external and internal data
for representation learning. In the second stage, we perform contrastive
learning with mined nearest neighbors to exploit self-supervisory signals in
the representation space. Extensive experiments on three intent recognition
benchmarks show that our approach can significantly improve the performance of
NID in both unsupervised and semi-supervised scenarios.

There are two limitations of this work. (1) We have only evaluated on balanced
data. However, in real-world applications, most datasets are highly
imbalanced. (2) The discovered clusters lack interpretability. Our clustering
method can only assign a cluster label to each unlabeled utterance but cannot
generate a valid intent name for each cluster.

##  6 Acknowledgments

We would like to thank the anonymous reviewers for their valuable comments.
This research was supported by the grants of HK ITF UIM/377 and PolyU DaSAIL
project P0030935 funded by RGC.

## References

  * Arora et al. (2020) Abhinav Arora, Akshat Shrivastava, Mrinal Mohit, Lorena Sainz-Maza Lecanda, and Ahmed Aly. 2020.  Cross-lingual transfer learning for intent detection of covid-19 utterances.
  * Bachman et al. (2019) Philip Bachman, R Devon Hjelm, and William Buchwalter. 2019.  Learning representations by maximizing mutual information across views.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Caron et al. (2018) Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45, Online. Association for Computational Linguistics.
  * Chatterjee and Sengupta (2020) Ajay Chatterjee and Shubhashis Sengupta. 2020.  Intent mining from past conversations for conversational agent.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4140–4152, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.
  * Cheung and Li (2012) Jackie Chi Kit Cheung and Xiao Li. 2012.  Sequence clustering and labeling for unsupervised query intent discovery.  In _Proceedings of the fifth ACM international conference on Web search and data mining_ , pages 383–392.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Forman et al. (2015) George Forman, Hila Nachlieli, and Renato Keshet. 2015.  Clustering by intent: A semi-supervised method to discover relevant clusters incrementally.  In _Machine Learning and Knowledge Discovery in Databases_ , pages 20–36, Cham. Springer International Publishing.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Giorgi et al. (2021) John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021.  DeCLUTR: Deep contrastive learning for unsupervised textual representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 879–895, Online. Association for Computational Linguistics.
  * Gowda (1984) K Chidananda Gowda. 1984.  A feature reduction and unsupervised classification algorithm for multispectral data.  _Pattern recognition_ , 17(6):667–676.
  * Gunel et al. (2021) Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021.  Supervised contrastive learning for pre-trained language model fine-tuning.  In _International Conference on Learning Representations_.
  * Hakkani-Tür et al. (2015) Dilek Hakkani-Tür, Yun-Cheng Ju, Geoffrey Zweig, and Gokhan Tur. 2015.  Clustering novel intents in a conversational interaction system with semantic parsing.  In _Sixteenth Annual Conference of the International Speech Communication Association_.
  * Han et al. (2019) Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2019.  Learning to discover novel visual categories via deep transfer clustering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Haponchyk and Moschitti (2021) Iryna Haponchyk and Alessandro Moschitti. 2021.  Supervised neural clustering via latent structured output learning: Application to question intents.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3364–3374, Online. Association for Computational Linguistics.
  * Haponchyk et al. (2018) Iryna Haponchyk, Antonio Uva, Seunghak Yu, Olga Uryupina, and Alessandro Moschitti. 2018.  Supervised clustering of questions into intents for dialog system applications.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2310–2321, Brussels, Belgium. Association for Computational Linguistics.
  * He et al. (2019) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2019.  Momentum contrast for unsupervised visual representation learning.  _arXiv preprint arXiv:1911.05722_.
  * Henderson et al. (2020) Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2020.  ConveRT: Efficient and accurate conversational representations from transformers.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2161–2174, Online. Association for Computational Linguistics.
  * Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020.  A simple language model for task-oriented dialogue.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Hsu et al. (2018) Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. 2018.  Learning to cluster in order to transfer across domains and tasks.  In _International Conference on Learning Representations (ICLR)_.
  * Hsu et al. (2019) Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. 2019.  Multi-class classification without multi-class labels.  In _International Conference on Learning Representations (ICLR)_.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017.  Billion-scale similarity search with gpus.  _arXiv preprint arXiv:1702.08734_.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18661–18673. Curran Associates, Inc.
  * Kim et al. (2021) Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021.  Self-guided contrastive learning for BERT sentence representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2528–2540, Online. Association for Computational Linguistics.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  * Lin et al. (2020) Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020.  Discovering new intents via constrained deep adaptive clustering with cluster refinement.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * MacQueen et al. (1967) James MacQueen et al. 1967.  Some methods for classification and analysis of multivariate observations.  In _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_ , volume 1, pages 281–297. Oakland, CA, USA.
  * Mehri et al. (2020) Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020.  Dialoglue: A natural language understanding benchmark for task-oriented dialogue.  _ArXiv preprint_ , abs/2009.13570.
  * Padmasundari (2018) Srinivas Bangalore Padmasundari. 2018.  Intent discovery through unsupervised semantic text clustering.  _Proc. Interspeech 2018_ , pages 606–610.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Perkins and Yang (2019) Hugh Perkins and Yi Yang. 2019.  Dialog intent induction with deep multi-view clustering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4016–4025, Hong Kong, China. Association for Computational Linguistics.
  * Radford and Narasimhan (2018) Alec Radford and Karthik Narasimhan. 2018.  Improving language understanding by generative pre-training.
  * Rebuffi et al. (2020) Sylvestre-Alvise Rebuffi, Sebastien Ehrhardt, Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2020.  Lsd-c: Linearly separable deep clusters.  _arXiv_.
  * Shi et al. (2018) Chen Shi, Qi Chen, Lei Sha, Sujian Li, Xu Sun, Houfeng Wang, and Lintao Zhang. 2018\.  Auto-dialabel: Labeling dialogue data with unsupervised learning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 684–689, Brussels, Belgium. Association for Computational Linguistics.
  * Van Gansbeke et al. (2020) Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool. 2020.  Scan: Learning to classify images without labels.  In _Proceedings of the European Conference on Computer Vision_.
  * Vedula et al. (2020) Nikhita Vedula, Rahul Gupta, Aman Alok, and Mukund Sridhar. 2020.  Automatic discovery of novel intents & domains from text utterances.
  * Vulić et al. (2021) Ivan Vulić, Pei-Hao Su, Samuel Coope, Daniela Gerz, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkšić, and Tsung-Hsien Wen. 2021.  ConvFiT: Conversational fine-tuning of pretrained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1151–1168, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wei and Zou (2019) Jason Wei and Kai Zou. 2019.  EDA: Easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6382–6388, Hong Kong, China. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.
  * Wu et al. (2020) Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020.  TOD-BERT: Pre-trained natural language understanding for task-oriented dialogue.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 917–929, Online. Association for Computational Linguistics.
  * Xie et al. (2016) Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016.  Unsupervised deep embedding for clustering analysis.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 478–487, New York, New York, USA. PMLR.
  * Xu et al. (2015) Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015.  Short text clustering via convolutional neural networks.  In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_ , pages 62–69, Denver, Colorado. Association for Computational Linguistics.
  * Yan et al. (2021) Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021.  ConSERT: A contrastive framework for self-supervised sentence representation transfer.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5065–5075, Online. Association for Computational Linguistics.
  * Yang et al. (2017) Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. 2017.  Towards k-means-friendly spaces: Simultaneous deep learning and clustering.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3861–3870. PMLR.
  * Zhang et al. (2021a) Dejiao Zhang, Feng Nan, Xiaokai Wei, Shang-Wen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021a.  Supporting clustering with contrastive learning.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5419–5430, Online. Association for Computational Linguistics.
  * Zhang et al. (2021b) Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. 2021b.  TEXTOIR: An integrated and visualized platform for text open intent recognition.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations_ , pages 167–174, Online. Association for Computational Linguistics.
  * Zhang et al. (2021c) Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021c.  Discovering new intents with deep aligned clustering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14365–14373.
  * Zhang et al. (2021d) Haode Zhang, Yuwei Zhang, Li-Ming Zhan, Jiaxin Chen, Guangyuan Shi, Xiao-Ming Wu, and Albert Y. S. Lam. 2021d.  Effectiveness of pre-training for few-shot intent classification.
  * Zhang et al. (2021e) Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021e.  Few-shot intent detection via contrastive pre-training and fine-tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1906–1912, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2020) Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020.  Discriminative nearest neighbor few-shot intent detection by transferring natural language inference.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5064–5082, Online. Association for Computational Linguistics.

##  Appendix A Experimental Details

###  A.1 Datasets

In this section, we provide more details about the datasets. The development
sets are prepared to exclude no unknown intents.

  * •

BANKING (Casanueva et al., 2020) is a fine-grained intent detection dataset in
which 77 intents are collected for banking dialogue system. The dataset is
splitted into 9,003, 1,000 and 3,080 for training, validation, and test sets
respectively.

  * •

StackOverflow (Xu et al., 2015) is a large scale dataset for online
questioning which contains 20 intents with 1,000 examples in each class. We
split the dataset into 18,000 for training, 1,000 for validation, and 1,000
for test.

  * •

M-CID (Arora et al., 2020) is a small scale dataset for cross-lingual Covid-19
queries. We only use the English subset of this dataset which has 16 intents.
We split the dataset into 1,220 for training, 176 for validation, and 349 for
test.

  * •

CLINC150 (Larson et al., 2019) consists of 10 domains across multiple unique
services. We use 8 domains 666The domains “Banking” and “Credit Cards” are
excluded because they are semantically close to the evaluation data. and
remove the out-of-scope data. We only use this dataset during training stage
1.

###  A.2 Implementation

The batch size is set to 64 for stage 1 and 128 for stage 2 in all experiments
to fully utilize the GPU memory. In stage 1, we first train until convergence
on external data and then train with validation on internal data. In stage 2,
we train until convergence without early-stopping.

###  A.3 More Experimental Results

The results on semi-supervised NID when $LAR=50\%$ are shown in Table 6. It
can be seen that our methods still achieve the best performance in this case.
In Fig. 6 and Fig. 7, we show the t-SNE visualization of clusters on BANKING
and M-CID with embeddings learned by two strongest baselines and our methods.
Again, it shows that our methods can produce more compact clusters.

| BANKING | StackOverflow | M-CID
---|---|---|---
KCR | Methods | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI | ACC
25% | BERT-DTC | 66.66 | 32.47 | 45.03 | 36.66 | 20.85 | 35.43 | 40.52 | 17.49 | 32.21
BERT-KCL | 56.88 | 19.40 | 23.28 | 35.26 | 16.18 | 30.86 | 31.81 | 11.13 | 23.18
BERT-MCL | 52.34 | 17.30 | 24.82 | 34.30 | 19.12 | 34.68 | 31.28 | 11.69 | 26.79
CDAC+ | 71.71 | 39.60 | 53.25 | 73.92 | 39.29 | 72.76 | 32.22 | 12.98 | 30.49
DAC | 73.89 | 42.84 | 55.01 | 56.80 | 39.51 | 55.33 | 53.72 | 31.36 | 47.36
MTP (Ours) | 80.94 | 53.44 | 63.68 | 75.30 | 63.27 | 76.97 | 73.77 | 55.37 | 69.00
MTP-DAC (Comb) | 83.05 | 58.36 | 68.17 | 78.15 | 64.64 | 79.25 | 78.36 | 64.33 | 77.36
MTP-CLNN (Ours) | 85.30 | 64.12 | 73.76 | 80.15 | 71.29 | 84.56 | 80.61 | 67.31 | 79.34
50% | BERT-DTC | 76.32 | 48.04 | 60.35 | 55.76 | 39.46 | 55.58 | 57.62 | 35.66 | 50.40
BERT-KCL | 70.60 | 37.47 | 45.37 | 55.82 | 37.29 | 48.32 | 53.88 | 32.02 | 44.76
BERT-MCL | 69.52 | 37.04 | 45.65 | 53.75 | 38.58 | 52.04 | 48.66 | 28.38 | 45.67
CDAC+ | 76.15 | 47.01 | 59.31 | 74.87 | 39.38 | 74.38 | 51.58 | 28.96 | 48.08
DAC | 79.89 | 54.09 | 65.14 | 72.51 | 59.12 | 74.51 | 68.21 | 49.59 | 63.84
MTP (Ours) | 85.03 | 62.97 | 72.63 | 79.58 | 70.49 | 83.51 | 78.53 | 63.93 | 76.50
MTP-DAC (Comb) | 86.78 | 67.23 | 76.48 | 81.36 | 72.58 | 85.18 | 81.42 | 69.36 | 81.98
MTP-CLNN (Ours) | 88.09 | 71.07 | 80.38 | 82.84 | 75.54 | 87.21 | 82.46 | 71.21 | 82.75
75% | BERT-DTC | 81.60 | 58.00 | 69.47 | 69.61 | 57.10 | 72.56 | 69.57 | 51.19 | 66.39
BERT-KCL | 81.23 | 57.94 | 68.33 | 66.85 | 53.02 | 66.14 | 68.46 | 50.21 | 64.67
BERT-MCL | 80.98 | 57.72 | 68.46 | 68.39 | 54.40 | 65.98 | 64.09 | 45.82 | 63.21
CDAC+ | 77.76 | 49.59 | 61.48 | 76.09 | 41.37 | 76.64 | 65.72 | 39.95 | 63.87
DAC | 84.78 | 64.25 | 74.09 | 77.67 | 65.45 | 81.45 | 77.37 | 63.84 | 77.05
MTP (Ours) | 88.69 | 72.65 | 81.24 | 83.27 | 76.19 | 87.12 | 83.53 | 73.45 | 83.87
MTP-DAC (Comb) | 89.52 | 74.57 | 82.97 | 83.97 | 77.24 | 87.91 | 84.89 | 75.84 | 86.59
MTP-CLNN (Ours) | 90.51 | 77.55 | 85.18 | 84.46 | 78.39 | 88.98 | 85.78 | 77.40 | 87.74

Table 6:  Performance on semi-supervised NID with different known class ratio.
The LAR is set to $50\%$. For each dataset, the best results are marked in
bold. Comb denotes the baseline method combined with our proposed MTP.

(a) CDAC+

(b) DAC

(c) MTP (Ours)

(d) MTP-CLNN (Ours)

Figure 6: t-SNE visulization of embeddings on BANKING. $\text{KCR}=25\%$,
$\text{LAR}=10\%$. Best viewed in color.

(a) CDAC+

(b) DAC

(c) MTP (Ours)

(d) MTP-CLNN (Ours)

Figure 7: t-SNE visulization of embeddings on M-CID. $\text{KCR}=25\%$,
$\text{LAR}=10\%$. Best viewed in color.


# A new equation of state for dense hydrogen-helium mixtures
II: taking into account hydrogen-helium interactions

Gilles Chabrier  CRAL, Ecole normale supérieure de Lyon, UMR CNRS 5574, 69364
Lyon Cedex 07, France,
School of Physics, University of Exeter, Exeter, EX4 4QL, UK  Florian Debras
IRAP, Université de Toulouse, UMR CNRS 5277, UPS, Toulouse, France

###### Abstract

In a recent paper (Chabrier et al. 2019), we have derived a new equation of
state (EOS) for dense hydrogen/helium mixtures which covers the temperature-
density domain from solar-type stars to brown dwarfs and gaseous planets. This
EOS is based on the so-called additive volume law and thus does not take into
account the interactions between the hydrogen and helium species. In the
present paper, we go beyond these calculations by taking into account H/He
interactions, derived from quantum molecular dynamics simulations. These
interactions, which eventually lead to H/He phase separation, become important
at low temperature and high density, in the domain of brown dwarfs and giant
planets. The tables of this new EOS are made publicly available.

###### Subject headings:

equation of state — dense plasmas — stars: low-mass stars, brown dwarfs, white
dwarfs — planets and satellites

##  1\. Introduction

In a recent paper, Chabrier et al. (2019, Paper I) have derived a new equation
of state (EOS) for dense hydrogen/helium mixtures which covers the
temperature-density domain from solar-type stars to gaseous planets. These
calculations combine semi-analytic EOS models in the low density, low
temperature molecular/atomic domain and in the high-density, high-temperature
fully ionized domain, respectively, and ab initio quantum molecular dynamics
(QMD) calculations in the intermediate pressure dissociation and ionization
domain. This EOS adequately reproduces all existing experimental results,
namely Hugoniot and isentropic shock experiments for pure H and He. It also
agrees very well with first principle numerical simulations for both the
single elements and the mixture in most of the covered domain. Departure from
the simulations, however, starts occuring below a temperature $T\lesssim
10^{5}$ K in a density domain $0.1\lesssim\rho\lesssim 10\,{\rm g}\,{\rm
cm}^{-3}$, i.e. in the $\sim$Mbar regime (see e.g. Fig. 27 of Paper I). From
the astrophysical point of view, this concerns the domain of so-called
substellar objects: brown dwarfs and giant planets.

This departure reflects the growing importance of the interactions between
hydrogen and helium species, which eventually leads to a phase separation
between these components. It is thus important to include these interactions
in the H/He EOS in order to have a correct treatment of substellar object
thermodynamic properties, thus structure and evolution. The EOS derived in
Paper I is based on the so-called additive volume law (AVL) approximation and
thus does not take into account H/He interactions (see eqns (8)-(11) of Paper
I). It is the aim of this paper to go beyond this limitation by including H/He
interactions, derived from ab initio QMD calculations (Militzer & Hubbard
2013, MH13).

The paper is organized as follows. In §2, we derive an ”effective” EOS for
pure hydrogen based partly on the one derived for H/He by MH13 in the relevant
temperature-pressure domain. In §3, we use this effective hydrogen EOS to
derive a revised H/He EOS for various helium mass fractions and we carry out
extensive comparisons of this new H/He EOS with the MH13 QMD calculations. The
tables are presented in this same section while section 4 is devoted to the
conclusion.

##  2\. An effective Hydrogen equation of state

To derive the effective H EOS based on the QMD calculations by MH13 in the
pressure ionization regime, we proceed as in Debras & Chabrier (2019). The
procedure is similar to the one used in Miguel et al. (2016) except that, in
contrast to these latter, it does take into account the mixing entropy
contribution in the H/He MH13 EOS (eqns.(3) below). As discussed below, such a
contribution is not negligible and affects the resulting EOS. The H/He EOS
table derived by MH13 is based on QMD simulations and provides the internal
energy and pressure for 391 temperature-density points and a typical mass
fraction $Y=0.245$ within the domain $T\in[10^{3},8\times 10^{4}]$ K, $\rho\in
[\sim 0.2,9.0]\,{\rm g}\,{\rm cm}^{-3}$. For 131 of these points, an ab initio
thermodynamic integration technique was performed to provide the Helmholtz
free energy and the entropy (see MH13 for details). Figure 1 shows the
corresponding domain covered by the MH13 EOS in the pure hydrogen diagram.

We first calculate at each $P$-$T$ point calculated by MH13 the corresponding
hydrogen density:

$\displaystyle\frac{1}{\rho_{\mathrm{MH13}}}$ $\displaystyle=$
$\displaystyle\frac{X_{\mathrm{MH13}}}{\rho_{H}}+\frac{Y_{\mathrm{MH13}}}{\rho_{\mathrm{He}}}$
(1) $\displaystyle\Rightarrow\frac{1}{\rho_{\mathrm{H}}}$ $\displaystyle=$
$\displaystyle\frac{1}{X_{\mathrm{MH13}}}\left(\frac{1}{\rho_{\mathrm{MH13}}}-\frac{Y_{\mathrm{MH13}}}{\rho_{\mathrm{He}}}\right),$
(2)

where $\rho_{H}$ is the (sought) mass density for pure hydrogen,
$X_{\mathrm{MH13}}=0.754$, $Y_{\mathrm{MH13}}=0.246$ the mass fractions of
hydrogen and helium in the MH13 simulations ($N_{\mathrm{H}}=220$ and
$N_{\mathrm{He}}=18$ the numbers of H and He particles, respectively),
$\rho_{\mathrm{MH13}}$ the density derived from MH13 by spline procedures;
$\rho_{\mathrm{He}}$ the helium density in the Chabrier et al. (2019) EOS.
This reduced table, which covers only the limited $T$-$P$ domain explored by
MH13, is then combined by spline procedures with the pure H EOS of Paper I in
the remaining $T$-$P$ domain covered by this latter, where the AVL becomes
valid.

Figure 1.— Same figure as Fig. 1 of Chabrier et al. (2019, Paper I) to
illustrate the temperature-density domain of the H EOS. The (blue) dashed
lines in the center of the figure delimitate the domain covered by the
Militzer & Hubbard (2013, MH13) EOS simulations. All other symbols are the
same as in Fig. 1 of Paper I.

Similarly, the pure hydrogen entropy is obtained from

$\displaystyle
S_{H}=\frac{1}{X_{\mathrm{MH13}}}\left(S_{\mathrm{MH13}}-Y_{\mathrm{MH13}}S_{\mathrm{He}}-S_{\mathrm{mix}}\right),$
(3)

with $S_{\mathrm{MH13}}$ the splined specific entropy from MH13,
$S_{\mathrm{He}}$ the helium specific entropy in the Chabrier et al. (2019)
EOS, all at the same ($P$,$T$), and $S_{\mathrm{mix}}$ is the mixing specific
entropy.

As mentioned above and will be shown in §3, $S_{\mathrm{mix}}$ can be of the
order of several percents and thus should not be neglected to derive the
effective pure H EOS. Strictly speaking, the derivation of the effective pure
H EOS is not fully consistent. Indeed, even though the MH13 calculations take
into account the non-ideal H/He mixing entropy, and then so does
$S_{\mathrm{mix}}$ in eqn. (3), eqn. (2) is based on the AVL mixing equations
and thus does not properly take into account the excess volume/density of both
species at constant pressure, $P\Delta(V_{\mathrm{H}}+V_{\mathrm{He}})$. In
our procedure, all the excess volume is somehow included in the pure hydrogen
one. As will be examined in detail in §3.1, in the region of maximum departure
from the AVL, T$\sim 5,000$-10,000 K, P$\sim 10$-100 GPa, i.e. $\rho\sim
0.1$-1$\,{\rm g}\,{\rm cm}^{-3}$, this excess volume remains modest, $\lesssim
4\%$, consistent with the value found for the H/He mixture in detailed
calculations (see e.g. Fig. 17 of Vorberger et al. 2007). An other obvious
limitation of the present calculations is that the effective hydrogen entropy
corresponds to one single value of $Y=Y_{\mathrm{MH13}}=0.246$. The derivation
of fully consistent H/He EOSs, however, would require carrying out ab-initio
QMD simulations for various values of $Y$ over a large enough ($T,\rho$)
domain to smoothly reach the domains where the AVL becomes valid. This
represents quite a heavy numerical task. Given the lack of such calculations,
we will stick to the present derivation, based on a combination of the present
effective hydrogen EOS and our helium EOS, to derive EOS tables for any helium
mass fraction, as required for giant planet interior structure calculations
(see e.g. Debras & Chabrier 2019). We will come back to this point in §3.1.

As mentioned in Paper I, the EOS is calculated initially in a $T$-$\rho$
domain, the thermodynamic variables used in QMD or PIMC calculations (carried
out in the canonical ensemble), and then transformed into a $T$-$P$ one by
bicubic interpolation procedures. It is inevitable that these successive
procedures lead to numerical errors, notably for second derivative variables.
We will examine the impact of these errors in the following section by
comparing our results with the original ab initio calculations. A first
estimate of these errors is displayed in Fig. 1 of Debras & Chabrier (2019).
These authors used at that time a subset of the present EOS to derive Jupiter
interior models that fullfil the JUNO (Iess et al. 2018) and GALILEO (Wong et
al. 2004) constraints. This figure shows the relative error on the density
between the present EOS and the MH13 one, as well as with Miguel et al.
(2016). As seen in this figure, for Jupiter interior conditions, the relative
error on the density, $(\rho-\rho_{\mathrm{MH13}})/\rho_{\mathrm{MH13}}$,
between the present and MH13 calculations for Jupiter interior conditions is
always $<1.0\%$, which is less than the numerical error in the MH13
simulations. In contrast the error for the Miguel et al. calculations reaches
several percents on the density. The error on the entropy is shown to be on
the same amount for the present EOS ($<0.5\%$), whereas it reaches up to
$30\%$ for Miguel et al. (2016).

We stress that this revised pure hydrogen EOS is only intended to be used for
the H/He mixtures to somehow take into account the H-He interactions. It must
not be used as a pure H EOS. For this latter case, one must use the H EOS
derived in Paper I, based in part on various ab-initio simulations for pure H.
For illustrative purposes, Fig. 2 compares the Hugoniot curves for deuterium
calculated with the present H EOS and with the one calculated in Paper I
(Chabrier et al. 2019) with the experimental data of Knudson & Desjarlais
(2017). Even though the differences remain small over most of the domain
probed by the experiments, we see that the agreement with the data in the
region of maximum compression is better with the H EOS calculated in Paper I.
We carried out similar comparisons for the hydrogen and deuterium Hugoniot
experiments of Brygoo et al. (2015), for their various precompressed initial
conditions (see Paper I). We found out that under the conditions probed by
these experiments, the difference between the present and 2019 EOS is even
more modest. This gives confidence in the way we have calculated the present
revised H EOS.

Figure 2.— Shock pressure vs density along the deuterium Hugoniot curve. Solid
triangles: results by Knudson & Desjarlais (2017) for initial temperature and
density $T_{0}=20$ K and $\rho_{0}=0.167\,{\rm g}\,{\rm cm}^{-3}$,
respectively. Empty circles: reanalyzed shock data obtained from various
experiments rescaled to the same initial density (data from Knudson &
Desjarlais (2017)). Solid squares: PIMC calculations of Militzer & Ceperley
(2000). Solid (red) line: present calculations; dashed (black) line: EOS of
Paper I (Chabrier et al. 2019); dotted (blue) line: SCvH EOS (Saumon et al.
1995).

##  3\. The Hydrogen/Helium mixture equation of state

###  3.1. Calculation of the H/He EOS

As mentioned in the conclusion of Paper I, while thermodynamic quantities for
the H/He mixture calculated with the ”additive volume law” (AVL) compare quite
well with ab-initio calculations for isentropes larger than about 12
$k_{B}/e^{-}$ ($\sim 13\,k_{B}/at$ ), i.e. about $9\times 10^{-2}$ MJ kg-1
K-1, the disagreement becomes noticeable below this value. This stems from the
increasing interactions between hydrogen and helium atoms for $T\lesssim{\rm
a\,\,few}\,\,10^{4}$ K and $\rho\gtrsim 0.1\,{\rm g}\,{\rm cm}^{-3}$ (see Fig.
27 of Paper I). Such a domain encompasses essentially all substellar objects,
brown dwarfs and giant planets, and is thus of high astrophysical interest.

The revised H/He EOS and the corresponding thermodynamic quantities are
calculated exactly as in §4 of Paper I. Figures 3-9 display exactly the same
comparisons with the MH13 simulations as in Figures 19-27 of Paper I.

Figures 3 and 4 portray the internal energy par atom and the non-ideal
pressure, $P/\rho$, for several isotherms, for $Y=0.245$, the helium mass
fraction used in MH13. We note the excellent agreement between the present
H/He EOS and the MH13 simulations, whereas the AVL-based EOS (Paper I) departs
noticeably from these latter in the $T$-$\rho$ domain displayed in the
figures. As seen in Fig. 4, for $\rho\,\hbox{\hbox{$>$}\kern-8.00003pt\lower
4.30554pt\hbox{$\sim$}}\,8\,{\rm g}\,{\rm cm}^{-3}$, we recover the fully
ionized plasma model of Chabrier & Potekhin (1998).

Figure 3.— Internal energy per atom vs density for several isotherm
calculations by Militzer & Hubbard (2013, MH13) (as labeled in the figure) for
$Y=0.245$, compared with the present EOS and the one based on the AVL (Paper
I), respectively. For all curves the zero of energy is the same as in MH13.
For sake of clarity, however, curves have been arbitrarily moved upward or
downward by constant shifts. Figure 4.— Same as Fig. 3 for the non-ideal
pressure $P/\rho$. The (blue) dashed lines correspond to the fully ionized
model of Chabrier & Potekhin (1998).

Figures 5 and 6 portray similar comparisons for various isochores. Again we
note the excellent agreement between MH13 and the present calculations and the
departure of the AVL EOS for $T\,\hbox{\hbox{$>$}\kern-8.00003pt\lower
4.30554pt\hbox{$\sim$}}\,10^{4}$ K,
$\rho\,\hbox{\hbox{$>$}\kern-8.00003pt\lower 4.30554pt\hbox{$\sim$}}\,1\,{\rm
g}\,{\rm cm}^{-3}$.

Figure 5.— Internal energy per atom vs temperature for several isochore
calculations by Militzer & Hubbard (2013, MH13) (as labeled in the figure),
compared with the present and AVL results. For all densities, the MH13 values
are the ones given by their fit except for $\rho=8.96\,{\rm g}\,{\rm
cm}^{-3}$, which is out the range of validity of the fit and for which the
empty circles are their simulation data points. Solid lines: present
calculations; blue long-dashed lines and empty circles: MH13; black dotted
lines: AVL (Paper I). For all curves the zero of energy is the same as in
MH13. For the sake of clarity, however, curves have been moved arbitrarily
upward or downward by constant shifts. Figure 6.— Same as Fig. 5 for the non-
ideal pressure $P/\rho$. For the sake of clarity, curves have been shifted
upward arbitrarily by constant shifts. Figure 7.— Entropy vs density for
several isotherm calculations by Militzer & Hubbard (2013) (as labeled in the
figure), compared with the present and AVL results, respectively. Same
labeling as in the previous figures. Figure 8.— Entropy vs temperature for
several isochore calculations by Militzer & Hubbard (2013) (as labeled in the
figure), compared with the present and AVL results, respectively. Same
labeling as in the previous figures. Careful: for sake of clarity, each curve
from $\rho=3.59\,{\rm g}\,{\rm cm}^{-3}$ to $\rho=0.108\,{\rm g}\,{\rm
cm}^{-3}$ has been shifted upward by 1 $k_{b}/$atom w.r.t. to the immediately
higher-density one.

Figures 7 and 8 show the same comparisons for the entropy. Of noticeable
interest is the excellent agreement between the present calculations and the
MH13 simulations for the $T=5000$ K and $T=10,000$ K isotherms, for which
departure from the AVL approximation is the largest over essentially the
entire density range displayed in the figure, highlighting the importance of
H-He interactions in this domain. This brings confidence in our procedure,
even though approximated, to take into account the non-ideal volume and
entropy of mixing, naturally included in the MH13 simulations. The difference
between the present or MH13 calculations and the ones based on the AVL amounts
to $\lesssim 5\%$ of the total entropy in the 5,000-10,000 K, 10-100 GPa (i.e.
$\sim 0.1$-1$\,{\rm g}\,{\rm cm}^{-3}$) $T$-$P$ maximum departure range, of
which more than 3% stems from the ideal entropy contribution (eqn.(11) of
Paper I). We do not show comparisons for the free energy (Figs. 23 and 24 of
Paper I) as the present and AVL calculations are essentially indistinguishable
for this quantity, an obvious consequence of the compensating increased values
of $U$ and $S$ on $F=U-TS$ when taking into account H/He interactions, an
effect already found for the fully ionized mixtures (Chabrier & Ashcroft
1990).

We now carry out similar comparisons along isentropic profiles. This is
illustrated in Figure 9. As mentioned in Paper I and seen in the figure, non-
ideal effects between the H and He species become noticeable for entropies
$S\lesssim 12k_{B}/e^{-}$ ($\lesssim 13\,k_{B}/at$ ), i.e. $\lesssim 9\times
10^{-2}$ MJ kg-1 K-1. Here again, we note the good agreement between the
present calculations and the MH13 simulations. The new calculations yield
cooler and denser profiles for a given pressure than the calculations based on
the AVL approximation, yielding more compact adiabatic structures. Of
noticeable interest is the excellent agreement for the 7$k_{B}/e^{-}$,
characteristic of Jupiter and Saturn interiors. This assesses the validity of
the EOS used in the calculations of Debras & Chabrier (2019). Note that
according to the calculations of Militzer & Hubbard (2013), adiabats for
$S\lesssim 6k_{B}/e^{-}$ pass through the H/He immiscibility region where the
mixed state described by the simulations is no longer thermodynamically
stable.

All these comparisons bring confidence in the present calculations of an EOS
for H/He mixture beyond the AVL approximation, i.e. which includes, even
though in an approximated way, the impact of H-He interactions on the
thermodynamic properties of the mixture. Of course, one must remain cautious
about the accuracy of the present EOS for other helium mass fractions. Only a
comparison with ab-initio simulations for various $Y$ values could verify the
agreement. However, the present test at least assesses the validity of our
H/He EOS for substellar objects of solar or near solar compositions.

Figure 9.— Temperature and pressure profiles for a series of adiabats as
labeled in the figure in $k_{B}/e^{-}$ ($=(k_{B}/{\rm atom})/1.076$ for the
present $Y$ value) for the MH13 mass fraction of helium ($Y=0.245$). Solid
lines: present calculations; blue long-dashed lines: MH13; black dotted lines:
AVL.

###  3.2. Form of the EOS tables

All tables of the new EOS have the same rectangular form as for the ones
presented in Paper I, and are given either with ($T,\rho$) or ($T,P$)
independent variables. Because of the various spline procedures either in the
interpolated regimes between different calculations or in the ($T,P$) to
($T,\rho$) transformations, the EOS, unfortunately, suffers from the same
unphysical numerical oscillations in some domains, notably for the second
derivatives. Note that the present tables have different limits for the
density than the ones of Paper I, namely:

$\displaystyle 2.0$ $\displaystyle\leq$ $\displaystyle\log\,T\leq 8.0,$
$\displaystyle-9.0$ $\displaystyle\leq$ $\displaystyle\log\,P\leq+13.0,$
$\displaystyle-6.0$ $\displaystyle\leq$ $\displaystyle\log\,\rho\leq+6.0,$ (4)

with grid spacings $\Delta\,\log T=0.05$, $\Delta\,\log P=0.05$,
$\Delta\,\log\rho=0.05$, i.e. 121 isotherms, each with 441 values of $P$ or
241 values of $\rho$, and $T$ in K, $P$ in GPa, $\rho$ in $\,{\rm g}\,{\rm
cm}^{-3}$. Indeed, calculations for $\log\,\rho\leq-6.0\,{\rm g}\,{\rm
cm}^{-3}$ were found to yield spurious results. In any case, the concept of
equilibrium thermodynamic quantities for such low densities becomes of dubious
validity. The domains of validity of the EOS remain the same as the ones
defined in Paper I, due to the onset of ion quantum effects or crystallization
(see Figs. 1 and 16 of Paper I and discussions therein).

The tables have been calculated for 3 helium mass fractions, namely
$Y_{\odot}=0.275$, the helium cosmogonic abundance,
$Y_{eq}=0.275+0.017=0.292$, the value used in the Lyon group solar composition
brown dwarf evolutionary calculations (Baraffe et al. 2003, Chabrier et al.
2000, Phillips et al. 2020), and $Y_{eq}=0.28+0.017=0.297$, the value used in
evolutionary calculations of solar composition low-mass stars (Baraffe et al.
2015)111$Y=0.28$ is the helium fraction required to recover the present radius
and luminosity of the Sun in the Baraffe et al. (2015) models. It must be kept
in mind that the exact abundance of the Sun is not known.. The ’equivalent’
helium fraction, $Y_{eq}=Y_{\odot}+Z_{\odot}$, with $Z_{\odot}=0.017$ (Asplund
et al. 2009), is the simplest (even though crude) way to take into account the
(small) contribution of heavy elements to the EOS.

A word of caution: In the stellar range, non ideal H/He contributions are
basically inconsequential. Therefore, in this domain, users should use the AVL
EOS tables (Chabrier et al. 2019) in order to avoid further numerical issues
due to spline procedures. The present tables should be used essentially in the
brown dwarf and giant planet domain, where H/He interactions start to play a
role (see Chabrier et al. 2021, in preparation).

##  4\. Conclusion

In this paper, we have derived a new equation of state for hydrogen and helium
mixtures which incorporate the results of the simulations performed by
Militzer & Hubbard (2013) for $Y=0.245$. In order to do so we first calculate
an effective EOS for pure hydrogen and combine this latter with our pure
helium EOS (Paper I) to derive the one for the mixture. This allows us to
somehow take into account the impact of the H-He interactions upon the
thermodynamic quantities of the mixture. Although the procedure is not
entirely satisfactory, due to the lack of a proper determination of the non-
ideal volume and entropy of mixing from the simulations, comparisons between
our calculations and the ones by MH13 for different thermodynamic quantities
for various isotherms, isochores and isentropes in the region of pressure
dissociation/ionization show an excellent agreement. Although being not a
genuine proof of validity of our procedure for any mixture composition, this
at least assesses its validity for solar-like H/He compositions. The
comparison between MH13 simulations and the present calculations for cool
isentropes, in particular, for which H/He interactions become dominant,
yielding eventually a demixing process, show an excellent agreement. The
impact of the H/He interactions on these isentropic (T,P) structures is found
to be substantial, yielding cooler and denser profiles, as already found in
MH13. This assesses the validity of the present H/He EOS in the domain of
substellar objects, brown dwarfs and giant planets, to derive more correct
structure profiles and cooling sequences. This will be examined in more
details in a forthcoming paper (Chabrier et al., 2021, in preparation).

An obvious weakness of the present EOS calculations is that they rely entirely
on one single set of ab-initio simulations, carried out for one single helium
composition. A point of concern is that various ab initio (quantum molecular
dynamics or path integral) calculations still show differences (see discussion
in Paper I). As pointed out in the conclusion of Paper I (see also Ramakrishna
et al. 2020), the different exchange-correlation functionals used in the
calculations for liquid hydrogen, for instance, yield pressures that can
differ by as much as $\,\hbox{\hbox{$>$}\kern-8.00003pt\lower
4.30554pt\hbox{$\sim$}}\,10\%$ in the domain of pressure dissociation and
ionisation. Whether these differences will have a strong or small impact on
the present EOS calculation and its astrophysical applications remains to be
determined. More ab-initio simulations, possibly with different numerical
tools, over a larger range of helium mass fractions, are strongly needed to
clarify this issue and definitely validate or not the present EOS calculations
for different H/He mixture compositions. This is crucial notably to determine
the internal structure and composition of our Solar System planets.

Calculations for other values of $Y$ can be calculated as described by
eqns.(8)-(11) of Paper I, using the present revised H EOS and the He EOS of
Paper I (requests should be sent to G. Chabrier). However, we stress again
that the present revised H EOS should be used only for this purpose and must
not be used as a pure hydrogen EOS, notably when comparing with ab initio
simulations or high pressure experiments for pure hydrogen or deuterium.

The present revised H/He EOS tables for $Y=0.275$, $Y=0.292$, and $Y=0.297$,
as well as the ones for pure H and pure He (Chabrier et al., 2019) are
available in a .tar.gz package with this article. The present data, along with
the 2019 dataset, can also be obtained at http://perso.ens-
lyon.fr/gilles.chabrier/DirEOS

This work has been partly supported by the Programme National de Planétologie
(PNP).

REFERENCES

Asplund, M., et al., 2009, ARA&A, 47, 481
Baraffe, I., Chabrier, G., Barman, T., Allard, F., & Hauschild, P., 2003, A&A,
402, 701
Baraffe, I., Homeier, D., Allard, F., & Chabrier, G., 2015, A&A, 577, 42
Brygoo, S. et al., 2015, J. Appl. Phys., 118, 195901
Chabrier, G., & Ashcroft, N., 1990, Phys. Rev. A, 42, 2284
Chabrier, G., & Potekhin, A., 1998, Phys. Rev. A, 58, 4941
Chabrier, G., Baraffe, I., Allard, F., & Hauschild, P., 2000, Astrophys. J. ,
542, 464
Chabrier, G., Mazevet, S. & Soubiran, F., 2019, Astrophys. J. , 872, 51
Debras, F. & Chabrier, G., 2019, Astrophys. J. , 872, 100
Iess, L. et al., 2018, Nature, 555, 220
Knudson, M., & Desjarlais, M., 2017, Phys. Rev. Lett., 118, 035501
Miguel, Y., Guillot, T. & Fayon, L., 2016, A&A, 596, 114
Militzer, B., & Hubbard, W., 2013, Astrophys. J. , 774, 148
Phillips, M., Trembin, P., Baraffe, I. et al., 2020, A&A, 637, 38
Ramakrishna, K., Dornheim, T. & Vorberger, J., 2020, Phys. Rev. B, 101, 195129
Saumon, D., Chabrier,G., & van Horn, H., 1995, Astrophys. J., Suppl. Ser. ,
99, 713
Vorberger, J. et al., 2007, Phys. Rev. B, 75, 4206
Wong, M.H. et al., 2004, Icarus, 171, 153