# Fast 2D Bicephalous Convolutional Autoencoder for Compressing 3D Time
Projection Chamber Data

Yi Huang<EMAIL_ADDRESS>Brookhaven National LabUSA ,  Yihui Ren
<EMAIL_ADDRESS>Brookhaven National LabUSA ,  Shinjae Yoo<EMAIL_ADDRESS>Brookhaven National LabUSA and  Jin Huang<EMAIL_ADDRESS>Brookhaven National
LabUSA

(2023)

###### Abstract.

High-energy large-scale particle colliders produce data at high speed in the
order of 1 terabytes per second in nuclear physics and petabytes per second in
high energy physics. Developing real-time data compression algorithms to
reduce such data at high throughput to fit permanent storage has drawn
increasing attention. Specifically, at the newly constructed sPHENIX
experiment at the Relativistic Heavy Ion Collider (RHIC), a time projection
chamber is used as the main tracking detector, which records particle
trajectories in a volume of three-dimensional (3D) cylinder. The resulting
data are usually very sparse with occupancy around $10.8\%$. Such sparsity
presents a challenge to conventional learning-free lossy compression
algorithms, such as SZ, ZFP, and MGARD. The 3D convolutional neural network
(CNN)-based approach, Bicephalous Convolutional Autoencoder (BCAE),
outperforms traditional methods both in compression rate and reconstruction
accuracy. BCAE can also utilize the computation power of graphical processing
units suitable for deployment in a modern heterogeneous high-performance
computing environment. This work introduces two BCAE variants: BCAE++ and
BCAE-2D. BCAE++ achieves a $15\%$ better compression ratio and a $77\%$ better
reconstruction accuracy measured in mean absolute error compared with BCAE.
BCAE-2D treats the radial direction as the channel dimension of an image,
resulting in a $3\times$ speedup in compression throughput. In addition, we
demonstrate an unbalanced autoencoder with a larger decoder can improve
reconstruction accuracy without significantly sacrificing throughput. Lastly,
we observe both the BCAE++ and BCAE-2D can benefit more from using half-
precision mode in throughput ($76-79\%$ increase) without loss in
reconstruction accuracy. The source code and links to data and pretrained
models can be found at https://github.com/BNL-DAQ-LDRD/NeuralCompression_v2.

deep learning, autoencoder, high-throughput inference, data compression,
sparse data, high energy and nuclear physics

††journalyear: 2023††copyright: rightsretained††conference: Workshops of The
International Conference on High Performance Computing, Network, Storage, and
Analysis; November 12–17, 2023; Denver, CO, USA††booktitle: Workshops of The
International Conference on High Performance Computing, Network, Storage, and
Analysis (SC-W 2023), November 12–17, 2023, Denver, CO, USA††doi:
10.1145/3624062.3625127††isbn: 979-8-4007-0785-8/23/11††ccs: Applied
computing††ccs: Applied computing Physical sciences and engineering††ccs:
Applied computing Physics

##  1\. Introduction

High-energy particle accelerators, such as the Large Hadron Collider (LHC)
(Evans and Bryant, [n. d.]) and the Relativistic Heavy Ion Collider (RHIC)
(sPHENIX, 2019b), play a critical role in advancing knowledge about the
fundamental building blocks of the universe. Particle accelerators work by
accelerating charged particles close to the speed of light and colliding them,
where they interact and produce new subatomic particles. Particle detectors
are built around the collision point to detect these particle products, which
encode the information about interactions at the collision. Basically, a
tracking detector acts as a camera capturing three-dimensional (3D) particle
trajectories. If there is a particle passing through it, each pixel or voxel
will register an analog-to-digital (ADC) number above a zero-suppression
threshold. This 3D “camera” can work with a large number of channels (millions
to billions) at a high frame rate (10 kHz to GHz), producing a large amount of
ADC data.

Figure 1. Detector model.

Specifically, the recently constructed sPHENIX experiment at RHIC (sPHENIX,
2019b) consists of layers of tracking and calorimeter detectors aiming to
study the microscopic nature of strongly interacting matter, ranging from
nucleons to the strongly coupled quark-gluon plasma. Most sPHENIX data come
from its main tracking detector, a Time Projection Chamber (TPC) (shown in
Figure 1). sPHENIX TPC digitizes 42M-voxels 3D pictures of the collision
continuously at 77 kHz. Traditionally, to reduce and store the data in time, a
filtering system, called level-1 trigger, has been used. The trigger
determines which data are more valuable, leading to a small subset of data
being selected and stored for later analysis. Instead of a level-1 trigger,
developing a high-throughput real-time compression algorithm to reduce and
store _all_ collision signals has become increasingly important for future
collider experiments with streaming data acquisition (DAQ) that aims to record
all collisions (Bernauer et al., 2023; Abdul Khalek et al., 2022).

There is abundant literature in the lossy compression community, but few
existing methods have been optimized or designed for sparse 3D TPC data. For
example, the effectiveness of the error-bounded SZ (Di and Cappello, [n. d.];
Tao et al., [n. d.]; Liu et al., [n. d.]) compression algorithm method has
been demonstrated in climate science and cosmology data. The fixed-rate
compression method (ZFP) (Lindstrom, [n. d.]) has been motivated by
hydrodynamics simulations, while the MultiGrid Adaptive Reduction of Data
(MGARD) (Ainsworth et al., [n. d.]; Liang et al., [n. d.]) method has been
developed for compressing turbulent channel flow and climate simulation. We
hypothesize that most data challenges impacting the high-performance computing
(HPC) community stem from distributed high-fidelity simulation in climate
science, fluid dynamics, cosmology, and molecular dynamics. Therefore, by
introducing this unique challenge from the particle accelerator community, we
seek to ignite research interest in the scientific data reduction community.
Although all these compression algorithms have demonstrated reasonable
performance with 3D TPC data, a specially designed neural network-based model,
Bicepheoulous Convolutional Auto-Encoder (BCAE) (Huang et al., 2021), can
outperform them in both compression rate and reconstruction accuracy. However,
as an initial proof-of-concept, BCAE has some drawbacks, including suboptimal
compression throughput.

This work proposes an improved version of BCAE, called BCAE++, which improves
the compression ratio from $27$ to $31$ and decreases the mean absolute error
(MAE) from $.198$ to $.112$. MAE is an indicator of reconstruction
accuracy—the lower, the better. We also introduce a two-dimensional (2D)
variant of BCAE, BCAE-2D, by replacing 3D CNN layers with a 2D one and
treating the radial dimension of the TPC data as a channel dimension. This
change has resulted in $3\times$ speedup in throughput. Because only the
encoder portion of the auto-encoder neural network architecture will be used
in real time and the decoder (decompression) can be used offline, this work
also explores if expanding the number of decoder parameters can improve
reconstruction accuracy. Lastly, we demonstrate a post-training “trick” using
the half-precision representation of a network. It can enhance the throughput
by over $70\%$ without losing reconstruction accuracy.

##  2\. TPC Data and BCAE Compression Methods

###  2.1. TPC Data Preparation

Figure 1 shows that the sPHENIX TPC is a perfect testbed for developing a
high-throughput real-time compression algorithm. It is located between the
inner silicon vertex tracker and the electromagnetic calorimeter. Along the
radial dimension, the TPC is composed of $48$ cylindrical layers of small
sensors, which are grouped into three layer groups: inner, middle, and outer.
Each layer group has $16$ consecutive layers. In the digitized data and for
each TPC layer, the voxels are presented as a rectangular grid with rows along
the $z$ (or horizontal) direction and columns along the azimuthal direction.
Within one layer group, all layers have the same number of rows and columns.
This allows us to represent the ADC values from one layer group as a 3D array.
This study focuses on the outer layer group, where the array of ADC values has
shape $(16,2304,498)$ in the radial, azimuthal, and horizontal orders.

Figure 2. Example of a TPC Wedge. The $z$ axis is in the horizontal direction.
The $x$ and $y$ axes are on the plane spanned by the radial and azimuthal
directions.

To match the subdivision of the TPC data assembly module in the readout chain,
the voxel data are divided into $24$ equal-size non-overlapping sections: $12$
along the azimuthal direction ($30$ degrees per section) and $2$ along the
horizontal direction (divided by the transverse plane passing the collision
point). We call one such section a _TPC wedge_ (Figure 2). The array of ADC
values from each TPC wedge in the outer layer has shape $(16,192,249)$, listed
in radial, azimuthal, and horizontal directions, respectively. All ADC data
from the same wedge will be transmitted to the same group of front-end
electronics, after which a real-time lossy compression algorithm could be
deployed. Therefore, TPC wedges are used as the direct input to the deep
neural network compression algorithms.

Here, we use the simulation data of $1310$ events for central
$\sqrt{s_{NN}}=200$ GeV Au$+$Au collisions with 170 kHz pile-up collisions.
The data were generated with the HIJING event generator (Wang and Gyulassy,
1991) and Geant4 Monte Carlo detector simulation package (Allison et al.,
2016) integrated with the sPHENIX software framework (sPHENIX, 2019a). The
simulated TPC readout (ADC values) from these events are represented in a
$10$-bit unsigned integer $\in[0,1023]$. To reduce unnecessary data
transmission between detector pixels and front-end electronics, a zero-
suppression algorithm has been applied. All ADC values below $64$ are
suppressed to zero as most of them are noise. This zero-compression makes the
TPC data sparse at about $10\%$ occupancy (non-zero values).

We divide the $1310$ total events into $1048$ events for training and $262$
for testing. Each event contains $24$ outer-layer wedges. Thus, the training
partition contains $25152$ TPC outer-layer wedges, while the testing portion
has $6288$ wedges. The compression algorithm aims to compress each wedge
independently.

Figure 3. Log ADC distribution.

Finally, as trajectory locations must be interpolated from neighboring sensors
using the ADC values, it is important to preserve the relative ADC ratio
between the sensors. Hence, for this study, we trained autoencoders to
reconstruct the _log ADC values_ ($\log_{2}(\text{ADC}+1)$) instead of the raw
ADC values. A log ADC value is a float number in $[0.,10.]$. Because of the
zero-suppression at $64$, all nonzero log ADC values exceed $6$. The ground
truth distribution of log ADC values is plotted in Figure 3.

###  2.2. Bicephalous Convolutional Auto-Encoder (BCAE)

Figure 4. Bicephalous convolutional autoencoder (BCAE).

An autoencoder (Hinton and Salakhutdinov, [n. d.]) is composed of one encoder
and one decoder. The output from the encoder is called a _code_. Autoencoders
are commonly used for data compression. The compression ratio is defined as
the ratio between the size of an input and its code—the smaller the code, the
higher the compression ratio. The decoder takes in a code and produces a
reconstruction of the input. The distance between the original input and the
reconstruction is used as the loss function to train the autoencoder network.
Although this approach may work well enough in cases where the input
distribution is more regular (resembling a Gaussian distribution), it may
struggle with a distribution such as those of zero-suppressed log ADC values
(Alanazi et al., 2020; Hashemi et al., 2019).

As evident in Figure 3, the log ADC value is bi-modal and has a sharp edge at
$6.0$. For this irregular distribution, we need renovated autoencoder
structures. The BCAE (Huang et al., 2021) was proposed as a potential
solution. As illustrated in Figure 4, in addition to the reconstruction
decoder $D_{\textrm{reg}}$, BCAE also has a segmentation decoder
$D_{\textrm{seg}}$ for voxel-wise bi-class classification. The segmentation
decoder determines whether a voxel is zero (class $0$) or nonzero (class $1$).
The output produced by $D_{\textrm{seg}}$ is assessed using the focal loss
$\mathcal{L}_{\textrm{seg}}$, a specialized loss function designed to address
imbalanced datasets (Lin et al., 2017). The output generated by the regression
decoder $D_{\textrm{reg}}$ is combined with that from $D_{\textrm{seg}}$ to
form the reconstruction of the input, which then is evaluated using a
regression loss $\mathcal{L}_{\textrm{seg}}$. This study uses MAE for the
regression loss.

Specifically, if we denote the total number of voxels by $M$ and let
$\hat{l}_{x}$ be the output of $D_{\textrm{seg}}$ for voxel $x$, the focal
loss is defined to be

(1)
$\mathcal{L}_{\textrm{seg}}\left(\left\\{\left.\hat{l}_{x}\right|x\right\\};\gamma\right)=\frac{1}{M}\sum_{x}-l_{x}\log_{2}\left(\hat{l}_{x}\right)\left(1-\hat{l}_{x}\right)^{\gamma}\\\
-\left(1-l_{x}\right)\log_{2}\left(1-\hat{l}_{x}\right)\left(\hat{l}_{x}\right)^{\gamma}.$

Here, $l_{x}=1$ if the voxel is positive and $0$ otherwise, and $\gamma$ is
the focusing parameter. We employ the focal loss because, on average, only
$10.8\%$ of ADC values are nonzero. In this study, we set the focusing
parameter $\gamma$ to be $2$. Given a classification threshold $h$ and let
$\hat{v}_{x}$ be the prediction for voxel $x$ produced by $D_{\textrm{reg}}$,
the masked prediction $\tilde{v}_{x}$ is defined as
$\tilde{v}_{x}=\hat{v}_{x}\mathbf{1}_{\hat{l}_{x}>h}$, where $\mathbf{1}$ is
the characteristic function. Hence, the regression loss
$\mathcal{L}_{\textrm{reg}}$ is defined as

(2)
$\mathcal{L}_{\textrm{reg}}\left(\left\\{\left.\tilde{v}_{x},v_{x}\right|x\right\\};h\right)=\frac{1}{M}\sum_{x}\left|\tilde{v}_{x}-v_{x}\right|.$

Finally, to manage the gap between $0$ and $6$, we adopt another technique
proposed by (Huang et al., 2021) called regression output transformation. We
apply an output activation function $T(x)=6.+3.\exp(x)$ to the output from the
regression decoder. Note that by applying $T$, _all_ regression output values
are above $6.$, and the zero values in the reconstruction will result from the
masking by segmentation output (refer to the definition of $\tilde{v}_{x}$).

###  2.3. BCAE++ and BCAE-HT

Two modifications are made to the original BCAE (Huang et al., 2021). First,
we pad the horizontal direction from length $249$ to length $256$ with zero.
This makes halving the dimension more straightforward and enables using
convolution/deconvolution with kernel size $4$, padding $1$, and stride $2$
uniformly throughout the encoder and decoder construction. This change
streamlines the neural network architecture search in a programmatic way. In
addition, this modification reduces the code dimension from $(8,17,13,16)$ to
$(8,16,12,16)$ and improves the BCAE compression ratio from $27.041$ to
$31.125$. Zero-padding in the horizontal direction is clipped during the
evaluation, so reconstruction accuracy metrics are not inflated. Second, we
remove all the normalization layers in BCAE as they do not affect
reconstruction performance significantly in a sufficiently long training.
However, we can speed up training and inference without them. Based on these
modifications, we constructed BCAE++ with a similar number of parameters to
the original BCAE but with better performance (Table 1) and a larger
compression ratio.

We also introduce a high-throughput (HT) variation, BCAE-HT. The difference
between BCAE++ and BCAE-HT is in the number of features (output channels) in
the four residual blocks of their respective encoders. For BCAE++, the numbers
of features are $8,16,32,32$ (same as BCAE), while those for BCAE-HT are
$2,4,4,8$. The change reduces the encoder size of BCAE-HT to $5\%$ of those
for BCAE and BCAE++ and yields a $76\%$ improvement in throughput.

Input: number of blocks $m$, number of downsampling layers $d$

Output: A PyTorch module

1

2Initialize network $N$ to be an empty module list;

3 Append $N$ with
$L_{\textbf{in}}=\texttt{Conv2D}\left(i=16,o=32,k=7,p=3\right)$

4for  _$i\leftarrow 1$ to $m$_ do

5 if  _$i\leq d$_ then

6 Append $N$ with $\texttt{AvgPool2D}\left(k=2,s=2\right)$;

7

8 Append $N$ with two residual block
$\texttt{Res}\left(i=32,o=32,k=3,p=1\right)$;

9Append $N$ with $L_{\textbf{out}}=\texttt{Conv2D}\left(i=32,o=16,k=1\right)$;

10

11return _$N$_ ;

Algorithm 1 BCAE_encoder_2D

###  2.4. BCAE-2D

Due to the thinness of the TPC wedge along the radial dimension—$16$ layers
versus $192$ in the azimuthal and $249$ in the horizontal dimensions—it may be
more reasonable to treat the layer dimension as channels of an “image.”
Moreover, while the layers all have different radii, the number of columns
(along the azimuthal direction) in one layer group remains the same. This
means the distance between two adjacent voxels along the azimuthal direction
is farther apart in an outer layer than an inner one. This breaks the
inductive bias of translation invariance of a 3D convolution along the radial
direction, making 2D convolution an even more appropriate choice for a TPC
wedge.

We detail the construction of the 2D encoder and decoder in Algorithm 1 and
Algorithm 2, respectively. The algorithms use $i$ and $o$ to denote the number
of input and output channels, $k$ to show kernel size, $p$ to indicate padding
(default is $0$), and $s$ to signify stride (default is $1$).

Input: number of blocks $n$, number of upsampling layers $d$, output
activation function $A$

Output: A PyTorch module

1

2# NOTE: a decoder must have the same number of upsampling steps as the
downsampling steps in its corresponding encoder

3Initialize network $N$ to be an empty module list;

4

5for  _$i\leftarrow 1$ to $n$_ do

6 if  _$i\leq d$_ then

7 Append $N$ with $\texttt{Upsample}\left(\texttt{scale\\_factor}=2\right)$;

8

9 Append $N$ with two residual block
$\texttt{Res}\left(i=32,o=32,k=3,p=1\right)$;

10Append $N$ with
$L_{\textbf{out}}=\texttt{Conv2D}\left(i=32,o=16,k=1\right)$;

11 Append $N$ with output activation function $A$;

12 return _$N$_ ;

Algorithm 2 BCAE_decoder_2D Figure 5. Reconstruction performance comparison.

A BCAE-2D encoder $E$ is constructed using BCAE_encoder_2D with two
parameters: the number of encoder blocks $m$ and number of downsampling layers
$d$. The segmentation decoder $D_{\textrm{seg}}$ is constructed by
BCAE_decoder_2D with Sigmoid as the output activation function $A$, while the
regression decoder $D_{\textrm{reg}}$ is constructed with the identity
function as $A$. The number of upsampling layers $d$ in the decoders must be
equal to the number of downsampling layers in their corresponding encoder. For
simplicity in this study, we construct the two decoders in a BCAE-2D model
with the same number of blocks ($n$). Hence, a BCAE-2D model can be denoted by
BCAE-2D$(m,n,d)$, where $m$ is the number of encoder blocks in Algorithm 1,
$n$ is the number of decoder blocks in Algorithm 2, and $d$ is the number of
downsampling/upsampling layers.

We keep $d$ at $3$ so the compression ratio of a BCAE-2D model is the same as
the 3D variants. A BCAE-2D model with $d=3$ produces code with shape
$(32,24,32)$. In Section 3.5, we conduct a grid search on the numbers of
encoder and decoder blocks. After balancing reconstruction accuracy and
throughput, we choose BCAE-2D$(m=4,n=8,d=3)$ to represent the BCAE-2D models.
Hence, in the sequel, we use BCAE-2D$(m=4,n=8,d=3)$ as the default BCAE-2D.

###  2.5. Training Procedure

We implement all BCAEs with PyTorch 2.0. The training is conducted on the
$25152$ outer-layer TPC wedges in the training partition of the datasets,
while $6288$ TPC wedges are reserved for testing. We set the classification
threshold $h$ in Equation (2) to be $.5$ for both training and testing. All
BCAE models are trained with a batch size of $4$, and we train all BCAE++ and
BCAE-HT for $1000$ epochs. The initial learning rate is set at $10^{-3}$ and
remains constant for the first $100$ epochs. In the remaining epochs, we
decrease the learning rate by $5\%$ every $20$ epochs. All BCAE-2D models are
trained for $500$ epochs. We set the initial learning rate at $10^{-3}$ and
keep it constant for the first $50$ epochs. In the remaining epochs, we
decrease the learning rate by $5\%$ every $10$ epochs. For all BCAE models, we
use the AdamW (Loshchilov and Hutter, 2017) optimizer with
$\left(\beta_{1},\beta_{2}\right)=\left(0.9,0.999\right)$ and weight decay
$0.01$.

To improve the classification performance, we balance the contribution of the
segmentation and regression loss dynamically as follows: assume the
segmentation and regression losses at epoch $t$ are $\rho^{t}_{\textrm{s}}$
and $\rho^{t}_{\textrm{r}}$, respectively. Denote the coefficient of the
segmentation loss at epoch $t$ by $c_{t}$. Then, the coefficient for
$\mathcal{L}_{\textrm{seg}}$ for epoch $t+1$ is given by

$c_{t+1}=\frac{.5c_{t}+\left(\rho^{t}_{\textrm{r}}/\rho^{t}_{\textrm{s}}\right)}{1.5}.$

We set $c_{0}$ to be $2000$.

Table 1. Performance, encoder model size, and throughput comparison. The encoder size is measured in the number of trainable parameters. The reconstruction accuracy metrics and throughput are all measured with half-precision mode. The best performance with respect to each metric is underlined. model | MAE $\downarrow$ | PSNR $\uparrow$ | precision $\uparrow$ | recall $\uparrow$ | encoder size $\downarrow$ | throughput $\uparrow$
---|---|---|---|---|---|---
BCAE-2D | $0.152$ | $11.726$ | $0.906$ | $0.907$ |  $169.0$k | $\sim 6.9$k
BCAE++ | $0.112$ | $14.325$ | $0.934$ | $0.936$ |  $226.2$k |  $\sim 2.6$k
BCAE-HT | $0.138$ | $12.376$ | $0.916$ | $0.915$ | $9.8$k |  $\sim 4.6$k
BCAE(Huang et al., 2021) | $0.198$ | $9.923$ | $0.878$ | $0.861$ |  $201.7$k | $\sim 2.4k$

##  3\. Result

###  3.1. Compression Ratio

The compression ratio is computed by the ratio between the input and the code.
Both input and code are treated as $16$-bit float. As mentioned in Section 2.3
and 2.4, the shape of the code produced by the BCAE-2D is $(32,24,32)$, and
those for BCAE++ and BCAE-HT are both $(8,16,12,16)$. Because the TPC wedge
has shape $(192,249,16)$, the compression ratio is $31.125$ for all newly
introduced BCAE variants. This is greater than the compression ratio of
$27.041$ in the original BCAE (Huang et al., 2021).

###  3.2. Comparing Encoder Model Size and Throughput

The encoder model size is measured in the number of trainable parameters.
Encoder throughput is measured in the number of TPC wedges processed per
second. The input and output are allocated in GPU memory. Therefore, the file
system input/output (IO) and host-to-device data transfer are not considered.
All throughput experiments are conducted on a single NVIDIA RTX A6000 GPU with
driver version 535. On the software side, we used PyTorch 2.0 compiled with
CUDA 12.2.

As shown in Table 1, BCAE++ has the largest encoder size of $226$k parameters,
followed by the original BCAE with $202$k. The BCAE-2D has $169$k but
significantly higher throughput. Reducing the number of BCAE++ parameters
results in the BCAE-HT model size of $9.8$k. Although better than BCAE++
($2.6$k), the BCAE-HT’s throughput ($4.6$k) is not as high as BCAE-2D
($6.9$k).

Figure 6. Panel A-C: Throughput in half- and full-precision modes on a single
NVIDIA RTX A6000 GPU. Panel D: Diagnosing the lack of speedup by changing from
full-precision to half-precision in the BCAE-HT model. This is due to small
kernel sizes and the lack of Tensor Core activities. Panel E: Throughput in
half-precision of BCAE-2D with $m=3,4,5,6,7$ encoder blocks and $3$
downsampling layers. The encoder size is measured in the number of parameters.

###  3.3. Reconstruction Accuracy Comparison

We evaluated performance of the BCAEs using four metrics: MAE, peak signal-to-
noise ratio (PSNR), precision, and recall. Here, precision and recall are
defined as follows:

$\textbf{precision}=\frac{\sum_{x}\mathbf{1}_{\hat{l}_{x}>h}\mathbf{1}_{x>6}}{\sum_{x}\mathbf{1}_{\hat{l}_{x}>h}};\quad\textbf{recall}=\frac{\sum_{x}\mathbf{1}_{\hat{l}_{x}>h}\mathbf{1}_{x>6}}{\sum_{x}\mathbf{1}_{x>6}}.$

We compare BCAE-2D and BCAE++ performance in two computation modes shown in
Table 2. Given that compressing in half-precision yields negligible
performance degradation while significantly boosting throughput, it is the
most likely computation model for future deployment. Hence, all BCAEs
performance reported in Table 1 are obtained with the half-precision
computation mode. BCAE++ achieves the best scores in all reconstruction
measurements.

Figure 5 compares the reconstruction performance of BCAE-2D, BCAE++, and BCAE-
HT on one test TPC wedge. The noticeably different plots (second row) indicate
the reconstruction produced by BCAE++ is the most accurate.

###  3.4. Investigating Half-precision Speedup

Throughput is tested in two computation modes: full-precision and half-
precision. In full-precision mode, the encoder weights and input are all set
to $32$-bit floats. In half-precision mode, we manually cast the encoder
weights and input to $16$-bit floats. In Figure 6AB, for BCAE-2D and BCAE++,
half-precision affords more than $70\%$ improvement in throughput. In full-
precision mode, BCAE-HT and BCAE-2D have a similar throughput of $4000$ frames
per second. However, BCAE-HT’s speedup is much less (Figure 6C). This is due
to the extremely small model size ($9.8$k) of BCAE-HT after reducing the 3D
convolution channel sizes from BCAE++’s $(8,16,32,32)$ to $(2,4,4,8)$. As
shown in Figure 6D, Tensor Core units are not used by those time-consuming
convolution computations.

Table 2. Reconstruction accuracy in full- and half-precision computation mode. model | mode | MAE | precision | recall
---|---|---|---|---
BCAE-2D | full | $0.151937$ | $0.905469$ | $0.906916$
half | $0.151965$ | $0.905326$ | $0.90705$
BCAE++ | full | $0.112347$ | $0.933817$ | $0.935779$
half | $0.112342$ | $0.933852$ | $0.935741$
BCAE-HT | full | $0.138443$ | $0.915891$ | $0.914562$
half | $0.138441$ | $0.915780$ | $0.914701$

Figure 7. Reconstruction accuracy of BCAE-2D models with varying encoder and
decoder depths.

###  3.5. Investigating Auto-encoder Design

Here, we study how the depth (the number of blocks) of a BCAE-2D model’s
encoder and decoders influence its reconstruction accuracy. For this purpose,
we conduct a grid search on the number of encoder blocks ($m$ in Algorithm 1),
ranging from $3$ to $7$, and number of decoder blocks ($n$ in Algorithm 2),
from $3$ to $11$. Figure 7 illustrates the reconstruction accuracy of BCAE-2D
models in MAE, precision, and recall. While the performance benefits
significantly from deepening the decoders, the influence of encoder depth is
relatively ambiguous. The benefit of a deep encoder is more obvious only when
it is paired with decoders that are significantly deeper. We also calculate
the compression throughput in half-precision mode and demonstrate the result
in Figure 6E. After balancing the reconstruction accuracy and compression
throughput, we choose the BCAE-2D model with $m=4$ encoder blocks and $n=8$
decoder blocks to represent the BCAE-2D models.

##  4\. Conclusion and Discussion

The TPC tracking detectors examined in this work represent a modern particle
accelerator detector that produces a large volume of data at an extreme rate
(1 TB/s to 1 PT/s). Unlike common simulation data from hydrodynamics, climate
science, and cosmology, TPC data are zero-compressed and sparse, presenting a
unique challenge to real-time data compression algorithms. We present two BCAE
variants: BCAE++ and BCAE-2D. Compared to the BCAE, BCAE++ improves the
compression rate by $15\%$ and reconstruction accuracy by $77\%$ measured in
MAE. The BCAE-2D model treats the radial dimension of TPC data as a channel
dimension and achieves a 3x speedup during inference compared to BCAE++.

Based on this initial effort, there are several research directions worthy of
future pursuit. For example, to further optimize the neural network throughput
performance, we want to incorporate network pruning, quantization, and sparse
CNN techniques. We also seek to extend our throughput comparison to include
results of GPU-accelerated conventional lossy compression methods, such as
MGARD-GPU and cuSZ (Tian et al., [n. d.]). Finally, we anticipate this work
can attract additional research interest in particle detector data compression
by the scientific data reduction community.

###### Acknowledgements.

This work was supported in part by Brookhaven National Laboratory under
Laboratory Directed Research & Development No. 23-048 and the Office of
Nuclear Physics within the U.S. DOE Office of Science under Contract No.
DESC0012704.

## References

  * (1)
  * Abdul Khalek et al. (2022) R. Abdul Khalek et al. 2022\.  Science Requirements and Detector Concepts for the Electron-Ion Collider: EIC Yellow Report.  _Nucl. Phys. A_ 1026 (2022), 122447.  https://doi.org/10.1016/j.nuclphysa.2022.122447 arXiv:2103.05419 [physics.ins-det]
  * Ainsworth et al. ([n. d.]) Mark Ainsworth, Ozan Tugluk, Ben Whitney, and Scott Klasky. [n. d.].  Multilevel Techniques for Compression and Reduction of Scientific Data—The Multivariate Case.  41, 2 ([n. d.]), A1278–A1303.  https://doi.org/10.1137/18M1166651 Publisher: Society for Industrial and Applied Mathematics.
  * Alanazi et al. (2020) Yasir Alanazi, Nobuo Sato, Tianbo Liu, W Melnitchouk, Michelle P Kuchera, Evan Pritchard, Michael Robertson, Ryan Strauss, Luisa Velasco, and Yaohang Li. 2020\.  Simulation of electron-proton scattering events by a Feature-Augmented and Transformed Generative Adversarial Network (FAT-GAN).  _arXiv preprint arXiv:2001.11103_ (2020).
  * Allison et al. (2016) J. Allison et al. 2016\.  Recent developments in Geant4.  _Nucl. Instrum. Meth. A_ 835 (2016), 186–225.  https://doi.org/10.1016/j.nima.2016.06.125
  * Bernauer et al. (2023) J. C. Bernauer et al. 2023\.  Scientific computing plan for the ECCE detector at the Electron Ion Collider.  _Nucl. Instrum. Meth. A_ 1047 (2023), 167859.  https://doi.org/10.1016/j.nima.2022.167859 arXiv:2205.08607 [physics.ins-det]
  * Di and Cappello ([n. d.]) Sheng Di and Franck Cappello. [n. d.].  Fast Error-Bounded Lossy HPC Data Compression with SZ. In _2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_ (2016-05). 730–739.  https://doi.org/10.1109/IPDPS.2016.11 ISSN: 1530-2075.
  * Evans and Bryant ([n. d.]) Lyndon Evans and Philip Bryant. [n. d.].  LHC Machine.  3, 8 ([n. d.]), S08001.  https://doi.org/10.1088/1748-0221/3/08/S08001
  * Hashemi et al. (2019) Bobak Hashemi, Nick Amin, Kaustuv Datta, Dominick Olivito, and Maurizio Pierini. 2019.  LHC analysis-specific datasets with Generative Adversarial Networks.  _arXiv e-prints_ (2019), arXiv–1901.
  * Hinton and Salakhutdinov ([n. d.]) G. E. Hinton and R. R. Salakhutdinov. [n. d.].  Reducing the Dimensionality of Data with Neural Networks.  313, 5786 ([n. d.]), 504–507.  https://doi.org/10.1126/science.1127647 Publisher: American Association for the Advancement of Science Section: Report.
  * Huang et al. (2021) Yi Huang, Yihui Ren, Shinjae Yoo, and Jin Huang. 2021\.  Efficient Data Compression for 3D Sparse TPC via Bicephalous Convolutional Autoencoder. In _2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA)_. 1094–1099.  https://doi.org/10.1109/ICMLA52953.2021.00179
  * Liang et al. ([n. d.]) Xin Liang, Ben Whitney, Jieyang Chen, Lipeng Wan, Qing Liu, Dingwen Tao, James Kress, David Pugmire, Matthew Wolf, Norbert Podhorszki, and Scott Klasky. [n. d.].  MGARD+: Optimizing Multilevel Methods for Error-Bounded Scientific Data Reduction.  71, 7 ([n. d.]), 1522–1536.  https://doi.org/10.1109/TC.2021.3092201 Conference Name: IEEE Transactions on Computers.
  * Lin et al. (2017) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017.  Focal loss for dense object detection. In _Proceedings of the IEEE international conference on computer vision_. 2980–2988.
  * Lindstrom ([n. d.]) Peter Lindstrom. [n. d.].  Fixed-Rate Compressed Floating-Point Arrays.  20, 12 ([n. d.]), 2674–2683.  https://doi.org/10.1109/TVCG.2014.2346458 Conference Name: IEEE Transactions on Visualization and Computer Graphics.
  * Liu et al. ([n. d.]) Yuanjian Liu, Sheng Di, Kai Zhao, Sian Jin, Cheng Wang, Kyle Chard, Dingwen Tao, Ian Foster, and Franck Cappello. [n. d.].  Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints.  33, 12 ([n. d.]), 4440–4457.  https://doi.org/10.1109/TPDS.2022.3194695 Conference Name: IEEE Transactions on Parallel and Distributed Systems.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * sPHENIX (2019a) sPHENIX. 2019a.  sPHENIX Software repositories https://github.com/sPHENIX-Collaboration.  (2019).  https://github.com/sPHENIX-Collaboration
  * sPHENIX (2019b) sPHENIX. 2019b.  Technical Design Report: sPHENIX experiment at RHIC.  (2019).  https://indico.bnl.gov/event/5905/
  * Tao et al. ([n. d.]) Dingwen Tao, Sheng Di, Zizhong Chen, and Franck Cappello. [n. d.].  Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization. In _2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS)_ (2017-05). 1129–1139.  https://doi.org/10.1109/IPDPS.2017.115 ISSN: 1530-2075.
  * Tian et al. ([n. d.]) Jiannan Tian, Sheng Di, Kai Zhao, Cody Rivera, Megan Hickman Fulp, Robert Underwood, Sian Jin, Xin Liang, Jon Calhoun, Dingwen Tao, and Franck Cappello. [n. d.].  cuSZ: An Efficient GPU-Based Error-Bounded Lossy Compression Framework for Scientific Data. In _Proceedings of the ACM International Conference on Parallel Architectures and Compilation Techniques_ (New York, NY, USA, 2020-09-30) _(PACT ’20)_. Association for Computing Machinery, 3–15.  https://doi.org/10.1145/3410463.3414624
  * Wang and Gyulassy (1991) Xin-Nian Wang and Miklos Gyulassy. 1991.  HIJING: A Monte Carlo model for multiple jet production in p p, p A and A A collisions.  _Phys. Rev. D_ 44 (1991), 3501–3516.  https://doi.org/10.1103/PhysRevD.44.3501


# Effective Open Intent Classification with K-center Contrastive Learning and
Adjustable Decision Boundary

Xiaokang Liu1, Jianquan Li2, Jingjing Mu2, Min Yang3, Ruifeng Xu4, Benyou
Wang5,6 Min Yang is corresponding author.

###### Abstract

Open intent classification, which aims to correctly classify the known intents
into their corresponding classes while identifying the new unknown (open)
intents, is an essential but challenging task in dialogue systems. In this
paper, we introduce novel K-center contrastive learning and adjustable
decision boundary learning (CLAB) to improve the effectiveness of open intent
classification. First, we pre-train a feature encoder on the labeled training
instances, which transfers knowledge from known intents to unknown intents.
Specifically, we devise a K-center contrastive learning algorithm to learn
discriminative and balanced intent features, improving the generalization of
the model for recognizing open intents. Second, we devise an adjustable
decision boundary learning method with expanding and shrinking (ADBES) to
determine the suitable decision conditions. Concretely, we learn a decision
boundary for each known intent class, which consists of a decision center and
the radius of the decision boundary. We then expand the radius of the decision
boundary to accommodate more in-class instances if the out-of-class instances
are far from the decision boundary; otherwise, we shrink the radius of the
decision boundary. Extensive experiments on three benchmark datasets clearly
demonstrate the effectiveness of our method for open intent classification.
For reproducibility, we submit the code at: https://github.com/lxk00/CLAP

## Introduction

Intent classification is an essential task in dialogue systems by categorizing
input sequences into pre-defined intent classes. It is usually formulated as a
supervised classification problem. Recently, superior classification
performance has been achieved by deep neural networks (Bendale and Boult 2016;
Hendrycks and Gimpel 2016), where sufficient labeled training data is provided
during model training. However, these methods make a closed world assumption
which is impractical for real-world applications, and cannot deal with the
previously unseen intent classes (Ryu et al. 2017; Zheng, Chen, and Huang
2020).

Open intent classification, which aims to correctly classify the known intents
into their corresponding classes while identifying the unknown intents as the
“open intent”, improves customer satisfaction by reducing false-positive
errors, making the intent classification more challenging. Early studies
employ SVM to reject unseen class examples (Tax and Duin 2004; Rifkin and
Klautau 2004; Jain, Scheirer, and Boult 2014) based on feature engineering
which is labor-intensive. Recently, there are several works being proposed to
detect open intents by learning a decision boundary of each known intent in
the similarity space (Bendale and Boult 2016; Hendrycks and Gimpel 2016; Shu,
Xu, and Liu 2017). Among them, ADB (Zhang, Xu, and Lin 2021) is a
representative open intent classification method, which learns the adaptive
spherical decision boundary for each known class with the aid of a pre-trained
text encoder.

Despite the noticeable progress of previous works, there are still several
challenges for open intent classification, which are not addressed well in
prior studies. First, the superior performance of standard supervised learning
is highly dependent on the quality and balance of the training labels. The
noisy and unbalanced training data may significantly degrade the
generalization performance of deep models on out-of-distribution intents.
Although KCL (Kang et al. 2020) combined the strengths of supervised learning
and contrastive learning to learn representations that were both
discriminative and balanced, it merely shortens the distance between the
anchor sample and the other positive samples, but ignores the distances among
the chosen positive samples. To address this limitation, we learn more robust
representations by incorporating instance-level semantic discriminativeness
into representation learning via contrastive learning. Concretely, we propose
a K-center contrastive learning (KCCL) method to learn more robust and
balanced text representations. Different from the general supervised
contrastive loss, our KCCL method encourages each positive instance to move in
the direction of the other $K$ positive instances so that each positive
instance moves towards the center of the positive samples, resulting in better
clustering performance and speeding up the convergence. In this way, we can
better gather the semantic features with the same intent, resulting in better
representation space for learning the intent decision boundary.

Second, ADB (Zhang, Xu, and Lin 2021) is a representative method to learn the
adaptive decision boundary for open intent classification, which learns the
radius of a specific class boundary by solely considering the instances
belonging to the given class. In particular, the ADB method only takes known
intents to adjust the boundary and disregards negative samples, which can lead
to performance deterioration on out-of-distribution intents. For example, when
plenty of unknown intents fall inside the decision boundary, the ADB method
cannot make correct adjustments (i.e., shrink boundary) accordingly. Since the
goal of open intent classification is to distinguish the known intents from
the unknown ones, the decision boundary learning should consider both known
and unknown intents. To solve the above problem and learn a more accurate
decision boundary of each intent class, we devise an expanding and shrinking
approach to further adjust the decision boundary by considering out-of-class
(negative) instances for determining the intent decision boundary.
Specifically, we can expand the radius of the decision boundary to accommodate
more in-class instances if out-of-class instances are far from the decision
boundary; otherwise, we shrink the radius of the decision boundary.

Our contributions are summarized as follows:

  * •

We propose a K-center contrastive learning method to learn robust text
representations and make a clear margin of different intents by gathering the
semantic features with the same intent.

  * •

We propose an adjustable spherical decision boundary learning method to
determine accurate and suitable decision conditions of each intent class by
expanding or shrinking the radius of the decision boundary.

  * •

We conduct extensive experiments on three benchmark datasets. Experimental
results show that our method achieves better performance than the state-of-
the-art methods for open intent classification.

## Related Work

### Open Intent Classification

Open intent classification, which aims to correctly classify the known intents
into their corresponding classes while identifying the new unknown intents,
has attracted noticeable attention due to its broad applications in dialogue
systems and question answering. Early approaches leverage machine learining
methods (e.g., SVM) to reject unseen classes for open world recognition
(Schölkopf et al. 2001; Tax and Duin 2004; Rifkin and Klautau 2004; Jain,
Scheirer, and Boult 2014). However, these methods relied heavily on feature
engineering which is labor intensive.

With the advance of deep learning, deep neural networks have dominated the
literature of open intent classification, which can learn the deep semantic
features of input texts automatically. Brychcín and Král (2017) calibrated the
confidence of the softmax outputs to compute the calibrated confidence score
and leveraged the score to obtain the decision boundary for unknown intent
detection. Yu et al. (2017) leveraged adversarial learning to produce positive
and negative samples from known samples for training the open intent
classifier. Ryu et al. (2018) exploited the generative adversarial networks
(Goodfellow et al. 2014) to generate out-of-distribution (OOD) samples with
the generator and learned to reject produced OOD samples with the
discriminator. However, these methods need a confidence score, which is
difficult to be pre-defined in advance, to determine the likelihood of an
utterance being out-of-scope.

Subsequently, Lin and Xu (2019) proposed the DeepUNK method, which utilized a
margin-based method to train the classifier and detected the unknown intent
with local outlier factor. Zhang, Xu, and Lin (2021) introduced a post-
processing method to learn a spherical decision boundary for each known intent
class with the aid of a pre-trained intent encoder. Prem et al. (2021)
presented a post-processing method, which utilized multi-objective
optimization to tune a deep neural network based intent classifier and make
the classifier capable of detecting unknown intents. Zhan et al. (2021)
constructed a set of pseudo outliers by using inliner features via self-
supervision in the training stage. The pseudo outliers were used to train a
discriminative classifier that can be directly applied to and generalize well
on the test task. However, these approaches do not take full advantage of the
representation learning for learning better discriminative and balanced
features.

### Contrastive Learning

Self-supervised learning (SSL) has attracted much attention since it can avoid
manually annotating large-scale datasets and uses the learned representation
well for downstream tasks (Hendrycks et al. 2019; Misra and van der Maaten
2020). Contrastive learning is a popular self-supervised representation
learning method (Le-Khac, Healy, and Smeaton 2020). The key idea behind
contrastive learning is to narrow the distance between the anchor sample and
the positive samples and increase the distance between the anchor sample and
the negative samples. SimCSE (Gao, Yao, and Chen 2021) is a representative
contrastive learning method, which learned superior sentence representations
from either unlabeled or labeled data. KCL (Kang et al. 2020) effectively
combined the strengths of the supervised method and the contrastive learning
method to learn representations that were both discriminative and balanced.
CERT (Fang et al. 2020) created augmentation samples of sentences using back-
translation, and then fine-tuned a pretrained language representation model by
predicting whether two augmentation samples are from the same original
sentence or not.

Figure 1: An overview of our method. In the first stage (left), we leverage a
novel K-center contrastive learning method to improve the text representation
learning. In the second stage (right), we design the adjustable decision
boundary learning with expanding and shrinking to determine the suitable
decision conditions for the known intents.

## Our Methodology

##### Problem Definition

Suppose we have a training data set $\mathcal{S}=\\{(x_{i},y_{i})\\}_{i}^{N}$
of $N$ input instances, where each input sequence
$x_{i}=\\{w_{1},\ldots,w_{n}\\}$ has a corresponding output intent label
$y_{i}\in\\{1,\ldots,C\\}$ and $n$ is the sequence length. Here, $C$ denotes
the number of different intent classes in the training dataset $\mathcal{S}$.
The goal of open intent classification is to correctly classify the $K$ known
intents into their corresponding classes while identifying the unknown intents
as the ($C+1$)$th$ intent type (also called “open intent”).

##### Model Overview

As illustrated in Figure 1, our open intent classification method can be
divided into two training stages. In the first stage, we employ K-center
contrastive learning method to learn the sentence representations. In the
second stage, we design a novel adaptive boundary learning method to
comprehensively learn the in-distribution and out-of-distribution data, so
that the decision boundary can be clearly determined.

### Initial Representation Learning

Following previous work (Zhang, Xu, and Lin 2021), we employ the pre-trained
BERT (Devlin et al. 2019) as sentence encoder to learn the feature
representations of input sentences. In particular, for each input sentence
$x_{i}$, we obtain the hidden representations of tokens
$[CLS,\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{n},SEP]\in\mathbb{R}^{(n+2)\times
H}$ from the last layer of BERT, where $\mathbf{v}_{i}$ is the hidden
representation of the $i$-th token and $H=768$ denotes the dimension size of
token embeddings. $CLS$ and $SEP$ represent the beginning and ending symbols.
We use mean-pooling over all tokens representations to obtain the sentence
representations $\mathbf{o}_{i}$ for input sequence $x_{i}$:

$\mathbf{o}_{i}={\mathrm{m}ean\textit{-}pooling}([CLS,\mathbf{v}_{1},\ldots,\mathbf{v}_{n},SEP])$
(1)

To further enhance the feature representation learning, we use the ReLU
activation function following the linear mapping to get the feature
representation $\mathbf{h}_{i}\in\mathbb{R}^{H}$, and then obtain
$\mathbf{z}_{i}$ by normalizing $\mathbf{h}_{i}$:

$\displaystyle\mathbf{h}_{i}={\rm
ReLU}(W_{1}\mathbf{o}_{i}+\mathbf{b}_{1}),~{}~{}\mathbf{z}_{i}=\frac{\mathbf{h}_{i}}{\left\|\mathbf{h}_{i}\right\|_{2}}$
(2)

where $W_{1}$ and $\mathbf{b}_{1}$ are learnable parameters.
$\left\|\cdot\right\|_{2}$ are Euclidean normalization over the input vector.

### K-center Contrastive Representation Learning

The general supervised contrastive learning (van den Oord, Li, and Vinyals
2018) aims to learn an embedding space where the instances from the same
category are pulled closely and the instances from different category are
pushed apart. Specifically, we minimize a contrastive loss function
$\mathcal{L}_{\rm CL}$ to optimize the encoder network:

$\displaystyle\mathcal{L}_{\rm
CL}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{{e}^{\mathbf{z}_{i}\cdot\mathbf{z}_{i}^{+}/\tau}}{{e}^{\mathbf{z}_{i}\cdot\mathbf{z}_{i}^{+}/\tau}+\sum_{\mathbf{z}_{i}^{-}\in
Z_{i}^{-}}{e}^{\mathbf{z}_{i}\cdot\mathbf{z}_{i}^{-}/\tau}}$ (3)

where $\mathbf{z}_{i}^{+}$ indicates one positive sample randomly chosen from
the training set which has the same intent label with $\mathbf{z}_{i}$.
$\boldsymbol{Z}_{i}^{-}$ indicates the set of randomly chosen negative
training samples which have different intent labels with $\mathbf{z}_{i}$.
$\tau$ is a temperature value.

The above supervised contrastive loss defined in Eq. (3) learns more robust
representations by incorporating instance-level semantic discriminativeness
into the representation learning. However, conventional supervised contrastive
learning uses all the samples from the same class to construct the positive
pairs, which could result in the dominance of instance-rich classes in the
representation learning, as revealed in (Kang et al. 2020).

##### K-center Contrastive Learning

To alleviate the problem of dominating the instance-rich classes, we propose a
K-center contrastive learning (KCCL) method to learn more balanced feature
representations, inspired by (Kang et al. 2020). Different from (Kang et al.
2020) that compares $K$ positive instances and the negative instances, we also
compare each instance pair within the set of $K$ positive instances. To be
specific, for each training instance $\mathbf{z}_{i}$ with intent label
$y_{i}$, we randomly select $K$ samples with intent class $y_{i}$ to form the
positive sample set $\boldsymbol{Z}^{+}_{i}$. We also randomly select $M$
training samples from other intent classes as the negative sample set
$\boldsymbol{Z}^{-}_{i}$. The K-center contrastive loss function
$\mathcal{L}_{\rm KCCL}$ can be formalized as follows:

$\displaystyle\mathcal{L}_{\rm KCCL}=-\frac{1}{N\cdot
K\cdot(K+1)}\sum_{i=1}^{N}\sum_{m=1}^{K+1}\sum_{n=1,n\neq m}^{K+1}$ (4)
$\displaystyle\log\frac{e^{\mathbf{z}_{i,m}^{+}\cdot\mathbf{z}_{i,n}^{+}/\tau}}{e^{\mathbf{z}_{i,m}^{+}\cdot\mathbf{z}_{i,n}^{+}/\tau}+\sum_{\boldsymbol{z}_{i}^{-}\in\boldsymbol{Z}_{i}^{-}}{(e^{\mathbf{z}_{i,m}^{+}\cdot\boldsymbol{z}_{i}^{-}/\tau}+e^{\mathbf{z}_{i,n}^{+}\cdot\mathbf{z}_{i}^{-}/\tau}})}$

where $\mathbf{z}_{i,m}^{+}$ and $\mathbf{z}_{i,n}^{+}$ are two positive
samples in $\boldsymbol{Z}^{+}_{i}$. Different from the previous contrastive
loss defined in Eq. (3), our KCCL method additionally shortens the distances
between positive samples. In this way, we can better gather the semantic
features with the same intent (van den Oord, Li, and Vinyals 2018; Kang et al.
2020), resulting in better representation space for learning the intent
decision boundary.

In addition to supervised contrastive loss, we also leverage supervised cross-
entropy (CE) loss $\mathcal{L}_{\rm CE}$ to learn semantic discriminative
features:

$\mathcal{L}_{\rm CE}=-\frac{1}{N}\sum_{i=1}^{N}\mathbf{y}_{i}\log{\rm
softmax}(W_{2}\mathbf{z}_{i}+\mathbf{b}_{2})$ (5)

where $\mathbf{y}_{i}$ is the one-hot vector for intent label $y_{i}$. $W_{2}$
and $\mathbf{b}_{2}$ are learnable parameters.

Overall, the loss function $\mathcal{L}_{S_{1}}$ of the first stage training
is obtained as a weighted sum between $\mathcal{L}_{\rm KCCL}$ and
$\mathcal{L}_{\rm CE}$:

$\mathcal{L}_{S_{1}}=\lambda\cdot\mathcal{L}_{\rm
KCCL}+(1-\lambda)\cdot\mathcal{L}_{\rm CE}$ (6)

where $\lambda$ is a pre-defined hyperparameter for controlling the impact of
the two loss functions.

### Adaptive Decision Boundary Learning via Expanding and Shrinking

In this section, we propose a novel inflating and shrinking approach to learn
adaptive decision boundaries for open intent classification.

#### Decision Boundary Learning

After learning the representations of input sentences via the KCCL method, we
learn a decision boundary for each known intent class $k$, which consists of a
decision center $\mathbf{c}_{k}$ and the radius of the decision boundary
$\mathrm{\Delta}_{k}$. Formally, suppose there is a dataset $\mathcal{S}_{k}$
containing all the training instances with intent class $k$. The decision
center $\mathbf{c}_{k}$ is a fixed vector and the radius of the decision
boundary $\mathrm{\Delta}_{k}$ is a learnable parameter. The decision center
$\mathbf{c}_{k}$ of class $k$ can be acquired by averaging the representations
of instances in $\mathcal{S}_{k}$:

$\mathbf{c}_{k}=\frac{1}{|\mathcal{S}_{k}|}\sum_{\mathbf{z}_{i}\in\mathcal{S}_{k}}\mathbf{z}_{i}$
(7)

where $|\mathcal{S}_{k}|$ denotes the number of instances in
$\mathcal{S}_{k}$. For each known instance $\mathbf{z}_{i}$ with the latent
class $k$, the radius $\mathrm{\Delta}_{k}$ of the decision boundary should
satisfy the following constraints:

$\left\|\mathbf{z}_{i}-\mathbf{c}_{k}\right\|_{2}\leq\Delta_{k}$ (8)

However, it is non-trivial to optimize the radius $\mathrm{\Delta}_{k}$ of the
decision boundary for open intent classification. As revealed in (Zhang, Xu,
and Lin 2021), the radius should be large enough to surround known intent
samples as much as possible so as to reduce the empirical risk, while we
should also decrease the open space risk by shrinking the radius to avoid To
balance the trade-off between the open space risk and the empirical risk, the
adaptive decision boundary (ADB) method (Zhang, Xu, and Lin 2021) is proposed
to optimize each learnable boundary radius as follows:

$\displaystyle\mathcal{L}_{\rm
ADB}=\frac{1}{N}\sum_{i=1}^{N}\gamma_{i}\cdot(\left\|\mathbf{z}_{i}-\mathbf{c}_{y_{i}}\right\|_{2}-\Delta_{y_{i}})+$
(9)
$\displaystyle(1-\gamma_{i})\cdot(\Delta_{y_{i}}-\left\|\mathbf{z}_{i}-\mathbf{c}_{y_{i}}\right\|_{2})$

where $\mathbf{z}_{i}$ is the representation of input sequence $x_{i}$
obtained in the first stage. $y_{i}$ is the label of $\mathbf{z}_{i}$.
$\gamma$ is a variable to control the impact of the two constrains, which is
defined as:

$\gamma_{i}=\begin{cases}1,&\text{ if
}\left\|\mathbf{z}_{i}-\mathbf{c}_{y_{i}}\right\|_{2}>\Delta_{y_{i}},\\\
0,&\text{ if
}\left\|\mathbf{z}_{i}-\mathbf{c}_{y_{i}}\right\|_{2}\leq\Delta_{y_{i}}.\end{cases}$
(10)

The parameters can be updated via stochastic gradient descent (SGD). As shown
in Eq. (9), the ADB method learns the radius of a specific class boundary by
solely considering the instances belonging to the given class.

#### Expanding and Shrinking

To learn a more accurate and suitable decision boundary of each intent class,
we devise an expanding and shrinking approach to further adjust the decision
boundary by considering out-of-class (negative) instances for determining the
intent decision boundary. Specifically, we can expand the radius of the
decision boundary to accommodate more in-class instances if out-of-class
instances are far from the decision boundary; otherwise, we shrink the radius
of the decision boundary. According to our statistics, the distances from the
unknown intents to the decision center conforms to a skewed normal
distribution. We adjust the decision boundary radius according to both tails
of the distribution. If there are large volumes of samples that fall in the
right tail, then we can expand the decision boundary, and If there are large
volumes of samples that fall in the left tail, then we should shrink the
decision boundary.

Formally, we introduce an expansion parameter $e$ and a shrink parameter $s$
for better expanding and shrinking the decision boundary. For each training
instance $\mathbf{z}_{i}$, if the distance between the out-of-class (negative)
sample $\mathbf{z}_{i}^{-}$ and the decision center $\mathbf{c}_{y_{i}}$ is
less than $\mathrm{\Delta}_{y_{i}}+s$, then we should reduce the value of the
radius $\mathrm{\Delta}_{y_{i}}$. On the contrary, if the distance between the
negative sample $\mathbf{z}_{i}^{-}$ and the decision center
$\mathbf{c}_{y_{i}}$ is greater than $\mathrm{\Delta}_{y_{i}}+e$, then we
increase the value of the radius $\mathrm{\Delta}_{y_{i}}$. We define a novel
loss function $\mathcal{L}_{\rm ADBES}$ for adaptive decision boundary
learning with expanding and shrinking operations (ADBES) as follows:

$\displaystyle\mathcal{L}_{\rm ADBES}=\mathcal{L}_{\rm
ADB}+\frac{1}{N}\sum_{i=1}^{N}\bigg{\\{}\eta\cdot\alpha_{i}\cdot\Big{[}\left\|\mathbf{z}_{i}^{-}-\mathbf{c}_{y_{i}}\right\|_{2}-$
(11)
$\displaystyle~{}~{}~{}~{}(\Delta_{y_{i}}+e)\Big{]}+\eta\cdot\beta_{i}\cdot\Big{[}(\Delta_{y_{i}}+s)-\left\|\mathbf{z}_{i}^{-}-\mathbf{c}_{y_{i}}\right\|_{2}\Big{]}\bigg{\\}}$

where $\eta$ is a pre-defined hyperparameter for controlling the impact of the
expanding and shrinking approach. $\mathbf{z}_{i}^{-}$ is a randomly chosen
negative sample which has different intent label with $\mathbf{z}_{i}$.
$\alpha$ and $\beta$ determine the expanding or shrinking operations over the
decision boundaries, which are defined as follows:

$\displaystyle\alpha_{i}=\begin{cases}1,&\left\|\mathbf{z}_{i}^{-}-\mathbf{c}_{y_{i}}\right\|_{2}>\Delta_{y_{i}}+e,\\\
0,&\left\|\mathbf{z}_{i}^{-}-\mathbf{c}_{y_{i}}\right\|_{2}\leq\Delta_{y_{i}}+e.\end{cases}$
(12)
$\displaystyle\beta_{i}=\begin{cases}1,&\left\|\mathbf{z}_{i}^{-}-\mathbf{c}_{y_{i}}\right\|_{2}<\Delta_{y_{i}}+s,\\\
0,&\left\|\mathbf{z}_{i}^{-}-\mathbf{c}_{y_{i}}\right\|_{2}\geq\Delta_{y_{i}}+s.\end{cases}$
(13)

With the loss function $\mathcal{L}_{\rm ADBES}$, we can learn more accurate
and suitable decision boundaries, which not only effectively surround most of
the known intent instances but also identify the open intent instances.

### Open Intent Classification during Inference

During the inference (testing) phase, the input samples may come from both
known and unknown intent classes. Given each test sample $x_{j}$, we obtain
its feature representation $\mathbf{z}_{j}$ via the K-center contrastive
learning. With the learned radius of the decision boundaries, the intent class
$\hat{y}_{j}$ can be predicted by thresholding the Euclidean distance between
$\mathbf{z}_{j}$ and the decision center $\mathbf{c}_{k}$ (i.e.,
$d(\mathbf{z}_{j},\mathbf{c}_{k})$) w.r.t. the corresponding radius
$\Delta_{k}$. That is, the model will mark the target instance as “Unknown” if
it is outside any of the learned decision boundaries. Formally, we predict the
open intent as follows:

$\hat{y}_{j}=\begin{cases}{\rm Unknown},{\rm
if}~{}d(\mathbf{z}_{j},\mathbf{c}_{k})>\Delta_{k},k\in\mathcal{Y};\\\ {\rm
arg}~{}{\rm min}_{k\in\mathcal{Y}}d(\mathbf{z}_{j},\mathbf{c}_{k}),~{}{\rm
otherwise}.\end{cases}$ (14)

where $\mathcal{Y}=\\{1,\ldots,K\\}$ represents the label space of the known
intents.

## Experimental Setup

### Datasets

We conduct extensive experiments on three publicly available benchmark
datasets. BANKING (Casanueva et al. 2020) is a fine-grained dataset in the
banking domain, which contains 77 intents and 13,083 customer service queries.
OOS (Larson et al. 2019) is a dataset for intent classification and out-of-
scope prediction. It consists of 150 intents, 22,500 in-domain queries and
1,200 out-of-domain queries. StackOverflow (Xu et al. 2015) is a dataset
released originally in Kaggle.com. It contains 3,370,528 technical question
titles. We use the processed dataset which has 20 different classes and 1,000
samples for each class. We provide the detailed statistics of the datasets in
Table 1.

### Evaluation Metrics

Following previous work (Zhang, Xu, and Lin 2021), we adopt the widely used
accuracy and macro F1-score as the evaluation metrics for open intent
classification. We treat all unknown classes as an open class, and calculate
the overall accuracy (denoted as Acc) and macro F1-score (denoted as F1) over
all intent classes. In addition, to better evaluate the capability of the
classifiers for identifying known and unknown intents, we also compute the
macro F1-score over known intents (denoted as Known) and unknown intents
(denoted as Unknown), respectively.

### Baseline Methods

We compare our method with the five state-of-the-art open intent
classification methods, including `MSP` (Hendrycks and Gimpel 2016) that
adopts a softmax prediction for out-of-distribution detection, `DOC` (Shu, Xu,
and Liu 2017) that uses deep learning to build a multi-class classifier and
leverages Gaussian fitting to tight the decision boundary, `OpenMax` (Bendale
and Boult 2016) that applies the concept of meta-recognition to the activation
patterns in the penultimate layer to reduce the risk of open space, `DeepUnk`
(Lin and Xu 2019) that replaces softmax loss with margin loss to learn
discriminative deep features by forcing the network to maximize inter-class
variance and minimize intra-class variance, `ADB` (Zhang, Xu, and Lin 2021)
that presents a new post-processing method for open intention classification
by learning the spherical decision boundary for each known class.

### Implementation Details

Following the same settings as in (Zhang, Xu, and Lin 2021), we keep some
classes in the training set as unknown and integrate them back during testing.
In particular, we vary the number of known classes in the training set with
the proporation of 25%, 50%, and 75% classes, and adopt all intent classes for
testing. It is noteworthy that the unknown classes are only utilized during
testing. To conduct a fair evaluation and ensure the stability of our model,
we run our model ten times and report the average results over ten runs of
experiments.

In the first training stage, we build our model on top of the pre-trained BERT
model (base-uncased) with 12-layer transformer and adopt most of its default
hyperparameter settings. We freeze all the parameters of BERT except the last
transformer layer to speed up the training process and avoid over-fitting. The
number of positive samples in KCCL are in 1 to 10 and the number of negative
samples $M$ is set to be 1. $\lambda$ is set to be 0.25. In the second
training stage, we freeze BERT model and train the decision boundary only. The
batch size is set to be 32, $e$ in range 0.5 to 1.2, $s$ from 0 to 0.5, $\eta$
from 0 to 1. We utilize Adam to optimize the model with a learning rate of
2e-5.

Dataset | Class | Train/Valid/Test | Length (max/mean)
---|---|---|---
BANKING | 77 | 9003 / 1000 / 3080 | 79 / 11.91
OOS | 150 | 15000 / 3000 / 5700 | 28 / 8.31
StackOverflow | 20 | 12000 / 2000 / 6000 | 41 / 9.18

Table 1: Statistics of experimental datasets.

|  | BANKING | OOS | StackOverflow
---|---|---|---|---
| Methods | ALL | Unknown | Known | ALL | Unknown | Known | ALL | Unknown | Known
| MSP | 50.09 | 41.43 | 50.55 | 47.62 | 50.88 | 47.53 | 37.85 | 13.03 | 42.82
| DOC | 58.03 | 61.42 | 57.85 | 66.37 | 81.98 | 65.96 | 47.73 | 41.25 | 49.02
| OpenMax | 54.14 | 51.32 | 54.28 | 61.99 | 75.76 | 61.62 | 45.98 | 36.41 | 47.89
25% | DeepUnk | 61.36 | 70.44 | 60.88 | 71.16 | 87.33 | 70.73 | 52.05 | 49.29 | 52.60
| ADB | 71.62 | 84.56 | 70.94 | 77.19 | 91.84 | 76.80 | 80.83 | 90.88 | 78.82
| CLAB | 75.87 | 88.73 | 75.20 | 81.26 | 94.49 | 80.91 | 86.03 | 95.12 | 84.21
| MSP | 71.18 | 41.19 | 71.97 | 70.41 | 57.62 | 70.58 | 63.01 | 23.99 | 66.91
| DOC | 73.12 | 55.14 | 73.59 | 78.26 | 79.00 | 78.25 | 62.84 | 25.44 | 66.58
| OpenMax | 74.24 | 54.33 | 74.76 | 80.56 | 81.89 | 80.54 | 68.18 | 45.00 | 70.49
50% | DeepUnk | 77.53 | 69.53 | 77.74 | 82.16 | 85.85 | 82.11 | 68.01 | 43.01 | 70.51
| ADB | 80.90 | 78.44 | 80.96 | 85.05 | 88.65 | 85.00 | 85.83 | 87.34 | 85.68
| CLAB | 83.08 | 81.82 | 83.36 | 87.03 | 90.63 | 86.98 | 87.68 | 89.30 | 87.52
| MSP | 83.60 | 39.23 | 84.36 | 82.38 | 59.08 | 82.59 | 77.95 | 33.96 | 80.88
| DOC | 83.34 | 50.60 | 83.91 | 83.59 | 72.87 | 83.69 | 75.06 | 16.76 | 78.95
| OpenMax | 84.07 | 50.85 | 84.64 | 73.16 | 76.35 | 73.13 | 79.78 | 44.87 | 82.11
75% | DeepUnk | 84.31 | 58.54 | 84.75 | 86.23 | 81.15 | 86.27 | 78.28 | 37.59 | 81.00
| ADB | 85.96 | 66.47 | 86.29 | 88.53 | 83.92 | 88.58 | 85.99 | 73.86 | 86.80
| CLAB | 88.12 | 70.74 | 88.42 | 90.53 | 86.91 | 90.57 | 88.11 | 77.59 | 88.81

Table 2: The results of open intent classification by varying the proportions
(25%, 50% and 75%) of known classes. Here, “ALL”, “Unknown” and “Known” denote
the macro F1 score over all intent classes, unknown (open) intent class and
known classes, respectively.

## Experimental Results

### Main Results

The results are shown in the Table 2. We report the macro F1 scores of all
compared methods over all intent classes (denoted as “ALL”), unknown (open)
intent class (denoted as “Unknown”) and known classes (denoted as “Known”),
respectively From the results, we can observe that our method substantially
and consistently outperforms the state-of-the-art baseline methods by a
noticeable margin, which demonstrates the effectiveness of our method.
Specifically, compared with the most competitive ADB method, our method
improves macro F1 scores by 4.25%/2.18%/2.16% on BANKING, by 4.07%/1.98%/2.00%
on OOS , and by 5.20%/1.85%/2.12% on StackOverflow with the proportions of
25%/50%/75% known classes, respectively.

Dataset | Method | ALL | Unknown | Known
---|---|---|---|---
BANKING | ADB | 71.62 | 84.56 | 70.94
CLAB | 75.87 | 88.73 | 75.20
w/o ADBES | 74.11 | 86.17 | 73.47
w/o KCCL | 73.10 | 86.28 | 72.41
OOS | ADB | 77.19 | 91.84 | 76.80
CLAB | 81.26 | 94.49 | 80.91
w/o ADBES | 79.25 | 92.85 | 78.89
w/o KCCL | 78.86 | 92.61 | 78.50
StackOverflow | ADB | 80.83 | 90.88 | 78.82
CLAB | 86.03 | 95.12 | 84.21
w/o ADBES | 84.41 | 94.15 | 82.47
w/o KCCL | 83.35 | 92.88 | 81.44

Table 3: Ablation results of CLAB on three datasets over with 25% known
classes

In addition, we observe that our method not only obtains significant
improvements on unknown classes, but also substantially improves the
performances on known classes compared with the baselines. This may be because
our method can learn accurate and suitable decision boundaries for intent
classes.

### Ablation Study

To verify the effectiveness of KCCL and ADBES methods for open intent
classification, we perform ablation tests on the three datasets with 25% known
classes. In particular, we remove the KCCL (denoted as w/o KCCL) from our
proposed CLAB method by replacing the first stage loss function
$\mathcal{L}_{S_{1}}$ with the cross-entropy loss $\mathcal{L}_{\rm CE}$. In
addition, we remove the ADBES (denoted as w/o ADBES) from CLAB by replacing
the decision boundary learning method with ADB (Zhang, Xu, and Lin 2021).

The ablation test results are summarized in Table 3. The performance of CLAB
drops sharply when discarding KCCL. This is because KCCL enables CLAB to learn
more robust and balanced text representations. In addition, ADBES also makes a
great contribution to CLAB. This is reasonable since ADBES plays a critical
role in learning adjustable decision boundaries for the known intents. Not
surprisingly, both KCCL and ADBES techniques contribute a noticeable
improvement to our method, since the performance of ADB (without both KCCL and
ADBES) is much worse than that of CLAB. This is within our expectation since
the KCCL can learn better feature representations and ADBES can learn better
adjustable decision boundaries of known intents.

Percent | Method | ALL | Unknown | Known
---|---|---|---|---
| ADBES | 73.10 | 86.28 | 72.41
25% | KCL+ADBES | 74.16 | 86.31 | 73.52
| KCCL+ADBES | 75.87 | 88.73 | 75.20
| ADBES | 81.46 | 81.86 | 81.45
50% | KCL+ADBES | 81.95 | 81.12 | 82.19
| KCCL+ADBES | 83.08 | 81.82 | 83.36
| ADBES | 86.32 | 69.02 | 86.59
75% | KCL+ADBES | 87.12 | 70.14 | 87.40
| KCCL+ADBES | 88.12 | 70.74 | 88.42

Table 4: Performance comparison between KCCL and KCL on the BANKING dataset.

### Analysis of K-center Contrastive Learning

##### Performance Comparison between KCCL and KCL

To investigate the advantage of KCCL over the previously published K-positive
contrastive learning (KCL) (Kang et al. 2020), we conduct experiments by
varying the proportions (25%, 50%, and 75%) of known classes on the BANKING
dataset. For a fair comparison, for the KCL, we also use the ADBES method to
learn the decision boundary. We run each experiment ten times and report the
average results in Table 4. From the results, we can observe that our KCCL
method performs significantly better than KCL, especially with 25% known
intents. This verifies the effectiveness of KCCL in learning better relations
among the positive samples.

In addition, we also report the learning curves corresponding to the average
cosine similarity between intra-class and inter-class samples on the BANKING
dataset. As illustrated in Figure 2, our model can achieve better upper bounds
and faster convergence speed than the compared methods. Specifically, the
average similarity between intra-class samples increases sharply until 400
iterations, while the average similarity between inter-class samples gets
converged until 400 iterations. This verifies the effectiveness of our KCCL
for learning more discriminative representations than the compared methods.

##### Effect of Positive Samples in KCCL

To verify the effect of the number of positive samples ($K$), we conduct
experiments by varying the value of $K$ on three datasets with the proportion
25% of known intents. We report the F1 results in Table 5. We can observe from
the results that different values of $K$ have slightly fluctuations on the
final results. In addition, we also notice that the optimal $K$ value for
different datasets is different. The model with a smaller number of positive
samples can achieve the best effect on the three corpora. Generally, our
method is not very sensitive to the number of positive samples (i.e., $K$).

$K$ | BANKING | OOS | StackOverflow
---|---|---|---
1 | 72.58 | 80.18 | 86.25
2 | 73.31 | 80.27 | 86.39
3 | 73.21 | 80.86 | 86.37
4 | 74.83 | 80.58 | 86.61
5 | 75.01 | 80.49 | 86.26
6 | 74.55 | 80.64 | 85.99
7 | 72.99 | 80.75 | 85.31
8 | 74.32 | 80.74 | 84.31
9 | 73.01 | 80.34 | 84.60
10 | 72.97 | 80.74 | 85.18
Table 5: The macro F1 score of our model by varying the value of $K$ on three
datasets with the proportion 25% of known intents. Here, $K$ denotes the
number of positive examples used in KCCL.

### Analysis of Adjustable Decision Boundary

To further investigate the tightness and coverage of decision boundary learned
by our ADBES method and the ADB method, we use different ratios of $\Delta$ as
boundaries during testing. The change of the decision boundary is limited to a
small interval, as previous research has found that a large decision boundary
change will inevitably lead to a significant drop in the F1-score (Zhang, Xu,
and Lin 2021). In particular, for both ADB and ADBES methods, we incorporate
KCCL with the fine-tuned BERT models to learn intent features of known
intents, and then the ADBES and ADB methods are applied to learn the spherical
decision boundary for each intent class. The experimental results are shown in
Figure 3. We observe that ADBES achieves the best performance with the learned
$\Delta$ among all tested decision boundaries, verifying the effectiveness of
the decision boundaries learned by ADBES. However, the best result is achieved
by ADB when using $0.95\Delta$, which means that the decision boundaries
learned by ADB are overrelaxed.

Figure 2: Similarity comparison of different loss functions on BANKING with
25% known classes. 10 random sentences are sampled from the test set of each
class. Figure (a) denotes intra-class cosine similarities. Figure (b) denotes
inter-class cosine similarities. Figure 3: Influence of the decision boundary
during inference. The blue and orange dash lines denotes the results learned
by ADBES and ADB respectively.

### Distance Distribution Visualization

Figure 4: Distance distribution visualization. Figures (a), (b) and (c)
represent the distributions and decision boundaries learned by our method with
proportions (25%, 50%, and 75%) of known classes on the StackOverflow dataset.
Figures (d), (e) and (f) represent the distributions learned by the ADB
method. The circles are divided into 5 sectors representing 5 classes. The
points close to the center of each circle are known class samples, while the
points that have different colors with the known classes represent open class
samples. The center of the circle represents the decision center of the
corresponding class. The graphs are drawn according to the Euclidean distance
from the sample points to the decision centers. We reduce the value of the
distance exceeding 1 to 0.99. We use the red arc to represent the decision
boundary.

We visualize the distance distribution and decision boundary of our CLAB
method and ADB (without KCCL training). The experiments are conducted on the
test set with proportions (25%, 50%, and 75%) of known classes on the
StackOverflow dataset. For a fair comparison, we normalize the sentence
representations produced by the original ADB method. For each dataset, we
randomly selected five categories for demonstration.

As shown in Figure 4, we have two observations. First, the distance
distribution overlap between open intent samples and known intent samples
learned by the ADB baseline is less than our CLAB method, especially with 25%
and 50% known classes. Second, the decision boundaries learned by our CLAB
method are more accurate and effective for open intent classification,
especially when the proportion of known classes is 25%. The ability of our
method to identify the open intents is significantly stronger than that of the
ADB method.

## Conclusion

In this paper, we proposed a novel two-stage method CLAB to improve the
effectiveness of open intent classification. First, we devised a K-center
contrastive learning algorithm to learn discriminative and balanced intent
features, improving the generalization of the model for recognizing open
intents. Second, we introduced an adjustable decision boundary learning method
with expanding and shrinking to determine the suitable decision conditions. In
particular, we expanded the radius of the decision boundary to accommodate
more in-class instances if the out-of-class instances were far from the
decision boundary; otherwise, we shrunk the radius of the decision boundary.
Extensive experiments on three benchmark datasets showed the effectiveness of
CLAB.

## Acknowledgements

This work was supported by the National Key Research and Development Program
of China (No. 2022YFF0902100), the National Natural Science Foundation of
China (61906185, 62006062, 62176076), Shenzhen Science and Technology
Innovation Program (Grant No. KQTD20190929172835662), Shenzhen Basic Research
Foundation (No. JCYJ20210324115614039 and No. JCYJ20200109113441941), Chinese
Key-Area Research and Development Program of Guangdong Province
(2020B0101350001), and Guangdong Provincial Key Laboratory of Big Data
Computing, the Chinese University of Hong Kong, Shenzhen.

## References

  * Bendale and Boult (2016) Bendale, A.; and Boult, T. E. 2016.  Towards open set deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1563–1572.
  * Brychcín and Král (2017) Brychcín, T.; and Král, P. 2017.  Unsupervised dialogue act induction using gaussian mixtures.  _EACL_.
  * Casanueva et al. (2020) Casanueva, I.; Temčinas, T.; Gerz, D.; Henderson, M.; and Vulić, I. 2020\.  Efficient intent detection with dual sentence encoders.  _arXiv preprint arXiv:2003.04807_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _ArXiv_ , abs/1810.04805.
  * Fang et al. (2020) Fang, H.; Wang, S.; Zhou, M.; Ding, J.; and Xie, P. 2020.  Cert: Contrastive self-supervised learning for language understanding.  _arXiv preprint arXiv:2005.12766_.
  * Gao, Yao, and Chen (2021) Gao, T.; Yao, X.; and Chen, D. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  _arXiv preprint arXiv:2104.08821_.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27.
  * Hendrycks and Gimpel (2016) Hendrycks, D.; and Gimpel, K. 2016.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_.
  * Hendrycks et al. (2019) Hendrycks, D.; Mazeika, M.; Kadavath, S.; and Song, D. X. 2019.  Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty.  In _NeurIPS_.
  * Jain, Scheirer, and Boult (2014) Jain, L. P.; Scheirer, W. J.; and Boult, T. E. 2014.  Multi-class Open Set Recognition Using Probability of Inclusion.  In _European Conference on Computer Vision_.
  * Kang et al. (2020) Kang, B.; Li, Y.; Xie, S.; Yuan, Z.; and Feng, J. 2020.  Exploring balanced feature spaces for representation learning.  In _International Conference on Learning Representations_.
  * Larson et al. (2019) Larson, S.; Mahendran, A.; Peper, J. J.; Clarke, C.; Lee, A.; Hill, P.; Kummerfeld, J. K.; Leach, K.; Laurenzano, M. A.; Tang, L.; et al. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  _arXiv preprint arXiv:1909.02027_.
  * Le-Khac, Healy, and Smeaton (2020) Le-Khac, P. H.; Healy, G.; and Smeaton, A. F. 2020.  Contrastive representation learning: A framework and review.  _IEEE Access_ , 8: 193907–193934.
  * Lin and Xu (2019) Lin, T.-E.; and Xu, H. 2019.  Deep unknown intent detection with margin loss.  _arXiv preprint arXiv:1906.00434_.
  * Misra and van der Maaten (2020) Misra, I.; and van der Maaten, L. 2020.  Self-Supervised Learning of Pretext-Invariant Representations.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 6706–6716.
  * Prem et al. (2021) Prem, P.; Ahmad, Z.; Ekbal, A.; Sengupta, S.; Jain, S.; and Rammani, R. 2021.  Unknown Intent Detection using Multi-Objective Optimization on Deep Learning Classifiers.  In _The 35th Pacific Asia Conference on Language, Information and Computation_ , 387–395.
  * Rifkin and Klautau (2004) Rifkin, R. M.; and Klautau, A. 2004.  In Defense of One-Vs-All Classification.  _Journal of Machine Learning Research_ , 5: 101–141.
  * Ryu et al. (2017) Ryu, S.; Kim, S.; Choi, J.; Yu, H.; and Lee, G. G. 2017.  Neural sentence embedding using only in-domain sentences for out-of-domain sentence detection in dialog systems.  _Pattern Recognition Letters_ , 88(Mar.1): 26–32.
  * Ryu et al. (2018) Ryu, S.; Koo, S.; Yu, H.; and Lee, G. G. 2018.  Out-of-domain detection based on generative adversarial network.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 714–718.
  * Schölkopf et al. (2001) Schölkopf, B.; Platt, J. C.; Shawe-Taylor, J. C.; Smola, A. J.; and Williamson, R. C. 2001.  Estimating the Support of a High-Dimensional Distribution.  _Neural Computation_ , 13(7): 1443–1471.
  * Shu, Xu, and Liu (2017) Shu, L.; Xu, H.; and Liu, B. 2017.  Doc: Deep open classification of text documents.  _arXiv preprint arXiv:1709.08716_.
  * Tax and Duin (2004) Tax, D. M. J.; and Duin, R. P. W. 2004.  Support Vector Data Description.  _Machine Learning_.
  * van den Oord, Li, and Vinyals (2018) van den Oord, A.; Li, Y.; and Vinyals, O. 2018.  Representation Learning with Contrastive Predictive Coding.  _ArXiv_ , abs/1807.03748.
  * Xu et al. (2015) Xu, J.; Wang, P.; Tian, G.; Xu, B.; Zhao, J.; Wang, F.; and Hao, H. 2015.  Short text clustering via convolutional neural networks.  In _Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing_ , 62–69.
  * Yu et al. (2017) Yu, Y.; Qu, W.-Y.; Li, N.; and Guo, Z. 2017.  Open-category classification by adversarial sample generation.  _IJCAI_.
  * Zhan et al. (2021) Zhan, L.-M.; Liang, H.; Liu, B.; Fan, L.; Wu, X.-M.; and Lam, A. 2021.  Out-of-Scope Intent Detection with Self-Supervision and Discriminative Training.  _arXiv preprint arXiv:2106.08616_.
  * Zhang, Xu, and Lin (2021) Zhang, H.; Xu, H.; and Lin, T.-E. 2021.  Deep open intent classification with adaptive decision boundary.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 14374–14382.
  * Zheng, Chen, and Huang (2020) Zheng, Y.; Chen, G.; and Huang, M. 2020.  Out-of-Domain Detection for Natural Language Understanding in Dialog Systems.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , PP(99): 1–1.