11institutetext: ISTI-CNR, Pisa, Italy 22institutetext: University of Pisa,
Pisa, Italy 33institutetext: CNIT, Florence, Italy 44institutetext: Mercatorum
University, Rome, Italy
44email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Adversarial Magnification to Deceive Deepfake Detection through Super
Resolution

Anonymous Submission  Davide Alessandro Coccomini 1122 0000-0002-0755-6154
Roberto Caldelli 3344 0000-0003-3471-1196 Giuseppe Amato 11
0000-0003-0171-4315 Fabrizio Falchi 11 0000-0001-6258-5313 Claudio Gennaro 11
0000-0002-3715-149X

###### Abstract

Deepfake technology is rapidly advancing, posing significant challenges to the
detection of manipulated media content. Parallel to that, some adversarial
attack techniques have been developed to fool the deepfake detectors and make
deepfakes even more difficult to be detected. This paper explores the
application of super resolution techniques as a possible adversarial attack in
deepfake detection. Through our experiments, we demonstrate that minimal
changes made by these methods in the visual appearance of images can have a
profound impact on the performance of deepfake detection systems. We propose a
novel attack using super resolution as a quick, black-box and effective method
to camouflage fake images and/or generate false alarms on pristine images. Our
results indicate that the usage of super resolution can significantly impair
the accuracy of deepfake detectors, thereby highlighting the vulnerability of
such systems to adversarial attacks. The code to reproduce our experiments is
available at: https://github.com/davide-coccomini/Adversarial-Magnification-
to-Deceive-Deepfake-Detection-through-Super-Resolution

###### Keywords:

Deepfake Detection, Super Resolution, Adversarial Attacks

##  1 Introduction

Manipulating content to spread misinformation and damage the reputation of
people has never been easier than nowadays. We are witnessing the unstoppable
evolution of those known as Deepfakes. These are counterfeit media contents
which often show people saying or doing things they never actually said or
did, distorting reality. Distinguishing pristine contents from manipulated
ones is extremely difficult. For this reason, various deepfake detectors have
been developed. These are, however, subject to various issues such as the need
to be up-to-date to keep up with the latest deepfake generation methods or the
ability to handle real-world situations. It is precisely in real-world
contexts that deepfake detection systems could be faced with targeted attacks
made to deceive them. Known as adversarial attacks, these are techniques that
introduce noise or adversarial patches, specifically crafted to deceive the
detector. Although they can also be very effective, these techniques may
require deep knowledge of the deepfake detector they are trying to fool. In
this paper, we attempt to exploit a Super Resolution (SR) technique, to
camouflage deepfake images in a quick and black-box manner (in the sense that
the attack is model-agnostic). Our approach allows us to cause a significant
increase in the False Negative Rate (fake samples classified as pristine) of
up to $18\%$. We also have shown how the usage of SR on pristine images can
cause a drastic increase in false alarms of up to $14\%$, highlighting the
inadequacy of some deepfake detectors, which will probably arise as these
techniques continue to proliferate.

##  2 Related Works

###  2.1 Deepfake Generation and Detection

The generation of deepfakes involves the use of techniques that manipulate
human faces to achieve realistic alterations in appearance or identity. Two
primary approaches are commonly employed: Variational AutoEncoders (VAEs) and
Generative Adversarial Networks (GANs). VAE-based methods utilize encoder-
decoder pairs to decompose and recompose distinct faces. On the other hand,
GAN-based methods use a discriminator to distinguish real and fake images,
paired with a generator that creates fake faces to fool the discriminator.
Notable Deepfake generation methods include Face2Face[21] and FaceSwap[3]. As
deepfakes becomes more credible, there is a growing demand for systems capable
of detecting them. To address this problem various deepfake detectors have
been developed. Some methods are capable of analyzing deepfake videos by
considering also the temporal information[10, 6, 24, 7] but most approaches
focus on frame-based classification, evaluating each video frame
individually[9, 2] and being available to manage also simply still deepfake
images. Also, competitions such as [12] and [13] have been organized to
stimulate the resolution of this task. The problem of deepfakes has also been
extended to the detection of synthetic images in general such in [8, 11, 4]
increasing the variety of fake contents.

###  2.2 Adversarial Attacks

Adversarial attacks, such as noise addition and adversarial patches, exploit
vulnerabilities in deepfake detectors to deceive them. Adversarial noise
introduces subtle perturbations, while adversarial patches overlap patterns to
trigger misclassification. The authors of [15] propose a framework called
FakeRetouch, which aims to reduce artifacts in deepfake images without
sacrificing image quality. By adding noise and using deep image filtering,
they achieve high fidelity to the original deepfake images reducing the
accuracy of deepfake detectors. In [14] the authors propose a statistical
consistency attack (StatAttack) against deepfake detectors by minimizing the
statistical differences between natural and deepfake images through the
addition of statistical-sensitive degradations.

###  2.3 Super Resolution

Super Resolution (SR) is a technique which aims to reconstruct a high-
resolution version of a low-resolution image by utilizing information from
multiple input images[5] or by using prior knowledge about the relationship
between high-resolution and low-resolution image pairs[23, 16]. One of the
main SR techniques is the one proposed in [18] where an Enhanced Deep Super
Resolution network (EDSR) is presented; it introduces some improvements to the
ResNet architecture for SR previously proposed in [16]. They remove batch
normalization layers to increase flexibility and reduce memory usage. They
also propose the use of residual scaling layers to stabilize the training
procedure. The model constructed with these modifications, and pre-trained
with a lower upscaling factor was able to achieve good results in terms of
convergence speed and final performance.

##  3 The proposed attack

Figure 1: SR attack pipeline: pre-processing and attack phases. The face size
is reduced by a factor $K$, then restored to its original resolution using a
SR algorithm and pasted back onto the source frame.

The proposed attack consists of exploiting SR techniques to modify a deepfake
image and camouflage it in the eyes of a deepfake detector. The scope of the
attack is then to mislead the deepfake detector and make the false negative
rate increase. The SR process, in an attempt to improve the resolution of an
image, could smooth the artifacts introduced by some deepfake generation
techniques, thus undermining the learning performed by the deepfake detection
model. Figure 1 shows the proposed framework for implementing the SR attack.
Specifically, for each of the frames of a video (or for each image if the
attack is applied to a still image) to be analyzed, a pretrained face detector
(e.g., MTCNN[22]) is applied. This step has been added to the pipeline for two
main reasons. The first motivation is related to the scope of the attack
itself since an attacker wants to manipulate a minimal part of the image in
order to avoid adding artifacts when not needed. Applying the SR on the whole
frame may add artifacts on the background finishing to have the inverse
effect. The second reason behind the usage of a face detector is the common
practice of both deepfake detectors and generators to focus only on the face
and so it is very likely that the deepfake detector against which the attack
is applied will only focus on the face and that the artifacts to be removed
are concentrated on the face. The face extracted from the network has a
specific resolution which is dependent on factors such as video resolution,
distance from the camera, etc. Since the goal of SR is to raise the resolution
of the image by a factor $K\in\mathbb{N}$, the image is firstly down-scaled by
a factor $1/K$ and then given as input to an SR model (e.g. EDSR[18]) to be SR
up-scaled by a factor K. The face image resulting from this process has the
same size as the original detected one and so can be again put inside the
source image from which it has been detected. So, to apply this method there
is no need to know anything about the deepfake detector that will be used for
the final detection, then the proposed method can be effectively considered a
black-box attack and can be applied against any deepfake detector and on
images manipulated with any deepfake generation method. Furthermore, this
attack can also be carried out on deepfake content already generated and does
not need to be integrated into the deepfake creation procedure.

##  4 Experiments

###  4.1 Dataset

Since we want to evaluate our attack on a variety of deepfake generation
methods, we chose the well-known FaceForensics++ (FF++)[19] dataset for our
experiments. The dataset consists of both pristine and manipulated videos
created using various deepfake generation methods, namely Deepfakes[1],
Face2Face[21], FaceShifter[17], FaceSwap[3], and NeuralTextures[20]. However,
as this dataset consists of videos and the proposed attack exploits single-
image SR, ten frames were randomly extracted for each of them on which face
detection was then carried out. A training set and a test set were created for
each deepfake generation method in FF++. Each training set consists of 14400
images, half of which were manipulated with one of the available methods. Each
test set consists of 2800 images, half of which are manipulated again with the
proposed attack. A total of five training and test sets are therefore
available and all of them are perfectly balanced between the two classes
(pristine and fake). To choose which videos should be used for training or
test set we used the split made available in [19].

###  4.2 Experimental Setup

To investigate the impact of the application of SR on the performance of
deepfake detectors, we selected three architectures, namely Resnet50, Swin-
Small and XceptionNet, and trained them on faces extracted from FF++ to
perform a binary classification by pristine/fake image. For each training, the
model only sees pristine images and fake ones manipulated with one of the
available FF++ methods (SR is not applied). All models are pretrained on
ImageNet and were fine-tuned with a learning rate of 0.01 for 30 epochs on an
Nvidia Tesla T4. The test is carried out considering two different setups, in
the first the models are tested by applying the SR-attack on both fake and
pristine images. In the second, the pristine images are un-attacked and only
the fake ones are passed through the SR process. The face is always extracted
from each frame using a pretrained MTCNN[22]. The scale factor used in our
experiments is $K=2$ and so the extracted face is resized of a factor $1/K$
and then up-scaled through EDSR[18] restoring the original resolution. After
this process, the face can be re-pasted to the frame exploiting the
coordinates extracted during face detection.

##  5 Results

###  5.1 Impact of Super Resolution on Deepfake Detection

Table 1: Evaluation on FF++ test set (half pristine and half fake). The SR
column indicates if the SR adversarial technique has been applied to the
images. Both pristine and fake images are attacked with SR.

Model | Forgery Method | SR |  FNR $\downarrow$ |  FPR $\downarrow$ |  Recall $\uparrow$ |  Precision $\uparrow$ |  AUC $\uparrow$ |  Accuracy $\uparrow$
---|---|---|---|---|---|---|---|---
Resnet50 | Deepfakes | $\times$ | 5.5 | 3.2 | 94.5 | 96.7 | 99.2 | 95.6
$\checkmark$ | 6.9 | 10.1 | 93.1 | 90.2 | 97.7 | 91.5
Face2Face | $\times$ | 5.0 | 3.2 | 95.0 | 96.7 | 98.9 | 95.9
$\checkmark$ | 14.4 | 4.7 | 85.6 | 94.8 | 97.0 | 90.5
FaceSwap | $\times$ | 6.4 | 1.9 | 93.6 | 98.0 | 99.1 | 95.9
$\checkmark$ | 21.1 | 2.6 | 78.9 | 96.8 | 96.0 | 88.1
FaceShifter | $\times$ | 6.1 | 3.4 | 93.9 | 96.5 | 98.8 | 95.3
$\checkmark$ | 24.8 | 3.3 | 75.2 | 95.8 | 96.8 | 86.0
NeuralTextures | $\times$ | 14.1 | 8.1 | 85.9 | 91.3 | 95.4 | 88.9
$\checkmark$ | 14.4 | 16.9 | 85.6 | 83.5 | 92.1 | 84.4
Swin | Deepfakes | $\times$ | 5.9 | 3.6 | 94.1 | 96.3 | 99.1 | 95.3
$\checkmark$ | 6.1 | 12.4 | 93.9 | 88.4 | 97.4 | 90.7
Face2Face | $\times$ | 6.3 | 3.3 | 93.7 | 96.6 | 98.9 | 95.2
$\checkmark$ | 24.4 | 1.7 | 75.6 | 97.8 | 96.1 | 87.0
FaceSwap | $\times$ | 4.9 | 4.6 | 95.1 | 95.3 | 98.6 | 95.2
$\checkmark$ | 21.9 | 5.3 | 78.1 | 93.7 | 93.9 | 86.4
FaceShifter | $\times$ | 7.2 | 4.1 | 92.8 | 95.8 | 98.7 | 94.4
$\checkmark$ | 18.9 | 3.1 | 81.1 | 96.3 | 97.4 | 89.0
NeuralTextures | $\times$ | 12.9 | 12.8 | 87.1 | 87.2 | 94.9 | 87.1
$\checkmark$ | 13.2 | 23.9 | 86.8 | 78.4 | 90.4 | 81.5
XceptionNet | Deepfakes | $\times$ | 5.3 | 2.6 | 94.7 | 97.4 | 99.3 | 96.1
$\checkmark$ | 5.6 | 12.4 | 94.4 | 88.4 | 97.9 | 91.0
Face2Face | $\times$ | 9.6 | 3.3 | 90.4 | 96.5 | 98.4 | 93.6
$\checkmark$ | 18.3 | 5.3 | 81.7 | 93.9 | 95.7 | 88.2
FaceSwap | $\times$ | 5.1 | 3.2 | 94.9 | 96.7 | 98.8 | 95.8
$\checkmark$ | 15.8 | 4.9 | 84.2 | 94.5 | 96.6 | 89.6
FaceShifter | $\times$ | 7.1 | 3.9 | 92.9 | 96.0 | 98.8 | 94.5
$\checkmark$ | 15.6 | 4.3 | 84.4 | 95.2 | 97.4 | 90.0
NeuralTextures | $\times$ | 13.1 | 7.2 | 86.9 | 92.3 | 95.9 | 89.8
$\checkmark$ | 9.8 | 21.6 | 90.2 | 80.7 | 92.7 | 84.3

Table 1 shows the results obtained from the deep learning models considered to
perform the deepfake detection task on the FF++ test set, with and without the
usage of SR on both fake and pristine images. Observing the accuracy results
on all the methods considered, the application of SR leads to a relevant drop
in performance in all cases, confirming the hypothesis that the SR process can
generate confusion in the deepfake detectors, thereby leading them to make
errors. More in detail, looking at the False Negative Rate (FNR) and False
Positive Rate (FPR) all the models seem to have a peak when the SR attack is
applied. When the deepfake generation method used on the image is Deepfakes or
NeuralTextures, the impact on the FNR is less evident but the same detector
that results in more robust on the fake images, fails on the pristine images
attacked with SR and we see a huge increase in the FPR. The situation is
exactly the opposite for the methods Face2Face, FaceSwap and FaceShifter on
which the models seem to be more sensible on the fake images attacked with SR
and so have an important increase on FNR while a slight swing in FPR is
registered. Increasing the FNR is the main interest for an attacker as it can
be useful to be able to camouflage fake images against an automatic system
that may be trying to filter them out. Vice versa, the increase in the FPR in
some cases, highlights a serious problem in deepfake detection systems that,
if SR became more widespread (e.g. on social media to improve the final visual
quality), would end up confusing legitimate images for deepfakes and also open
the door for an attacker to deliberately raise false alarms in the system.
That the use of SR pushes all Deepfake Detection models into error is also
shown in Figure 2 where it can be seen that in all cases, the AUCs obtained by
the models on SR images are drastically lower (dashed lines) than their
counterpart tested on images on which the SR process has not been applied
(solid lines).

Figure 2: ROC curves on FF++ dataset for the three considered models.

(a) Resnet50

(b) Swin

(c) XceptionNet

Table 2: Evaluation on FF++ test set (half pristine and half fake). The SR
column indicates if the SR adversarial technique has been applied to the
images. Only the fake images are attacked with SR.

Model | Forgery Method | SR |  FNR $\downarrow$ |  FPR $\downarrow$ |  Recall $\uparrow$ |  Precision $\uparrow$ |  AUC $\uparrow$ |  Accuracy $\uparrow$
---|---|---|---|---|---|---|---|---
Resnet50 | Deepfakes | $\times$ | 5.5 | 3.2 | 94.5 | 96.7 | 99.2 | 95.6
$\checkmark$ | 6.9 | 93.1 | 96.7 | 98.9 | 95.0
Face2Face | $\times$ | 5.0 | 3.2 | 95.0 | 96.7 | 98.9 | 95.9
$\checkmark$ | 14.4 | 85.6 | 96.4 | 97.6 | 91.2
FaceSwap | $\times$ | 6.4 | 1.9 | 93.6 | 98.0 | 99.1 | 95.9
$\checkmark$ | 21.1 | 78.9 | 97.6 | 95.6 | 88.5
FaceShifter | $\times$ | 6.1 | 3.4 | 93.9 | 96.5 | 98.8 | 95.3
$\checkmark$ | 24.8 | 75.2 | 95.7 | 95.2 | 85.9
NeuralTextures | $\times$ | 14.1 | 8.1 | 85.9 | 91.3 | 95.4 | 88.9
$\checkmark$ | 14.4 | 85.6 | 91.3 | 94.7 | 88.7
Swin | Deepfakes | $\times$ | 5.9 | 3.6 | 94.1 | 96.3 | 99.1 | 95.3
$\checkmark$ | 6.1 | 93.9 | 96.3 | 98.7 | 95.1
Face2Face | $\times$ | 6.3 | 3.3 | 93.7 | 96.6 | 98.9 | 95.2
$\checkmark$ | 24.4 | 75.6 | 95.8 | 96.2 | 86.2
FaceSwap | $\times$ | 4.9 | 4.6 | 95.1 | 95.3 | 98.6 | 95.2
$\checkmark$ | 21.9 | 78.1 | 94.4 | 94.1 | 86.7
FaceShifter | $\times$ | 7.2 | 4.1 | 92.8 | 95.8 | 98.7 | 94.4
$\checkmark$ | 18.9 | 81.1 | 95.2 | 96.6 | 88.5
NeuralTextures | $\times$ | 12.9 | 12.8 | 87.1 | 87.2 | 94.9 | 87.1
$\checkmark$ | 13.2 | 86.8 | 87.2 | 93.9 | 87.0
XceptionNet | Deepfakes | $\times$ | 5.3 | 2.6 | 94.7 | 97.4 | 99.3 | 96.1
$\checkmark$ | 5.6 | 94.4 | 97.3 | 99.2 | 95.9
Face2Face | $\times$ | 9.6 | 3.3 | 90.4 | 96.5 | 98.4 | 93.6
$\checkmark$ | 18.3 | 81.7 | 96.1 | 96.9 | 89.2
FaceSwap | $\times$ | 5.1 | 3.2 | 94.9 | 96.7 | 98.8 | 95.8
$\checkmark$ | 15.8 | 84.2 | 96.3 | 97.1 | 90.5
FaceShifter | $\times$ | 7.1 | 3.9 | 92.9 | 96.0 | 98.8 | 94.5
$\checkmark$ | 15.6 | 84.4 | 95.6 | 97.1 | 90.2
NeuralTextures | $\times$ | 13.1 | 7.2 | 86.9 | 92.3 | 95.9 | 89.8
$\checkmark$ | 9.8 | 90.2 | 92.6 | 96.5 | 91.5

To evaluate deepfake detectors in a realistic context, an alternative test set
was considered in which pristine images are not subjected to the SR process.
In fact, an attacker has much more interest in generating false negatives than
false positives, so as to go undetected by automated systems. As can be seen
from the experiments reported in Table 2 in this setup the accuracy decreases,
though more slightly, in almost all the cases with some deepfake generation
methods on which the detectors are more robust to the attack. More in detail,
the Face2Face, FaceSwap and FaceShifter images enhanced with the SR attack,
are very difficult to detect, probably because the artifacts which the
detector has learnt to recognize during the training process, are hidden by
the SR process and this is translated in an higher FNR and a lower Recall
value. In all the cases, the FPR is not affected by the usage of the SR attack
since the pristine images are not attacked in this setup.

###  5.2 Visual Impact Analysis

When performing an SR attack on a fake image, it is important that it remains
as indistinguishable to human eyes as possible so as to preserve its meaning
but also to make it less suspicious to users. To assess the impact of our
attack on the image appearance, we compared the similarity of each image pair
(non-SR, SR) through two commonly used quality metrics, _Structural Similarity
Index (SSIM)_ and _Peak Signal-to-Noise Ratio (PSNR)_. The SSIM is calculated
as
$\text{SSIM}(x,y)=\frac{{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{xy}+C_{2})}}{{(\mu_{x}^{2}+\mu_{y}^{2}+C_{1})(\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2})}}$,
where $x$ and $y$ are the two compared images, $\mu_{x}$ and $\mu_{y}$ are the
average values of $x$ and $y$ respectively, $\sigma_{x}$ and $\sigma_{y}$ are
the standard deviations, $\sigma_{xy}$ is the covariance between x and y and
$C_{1}$ and $C_{2}$ are two constants used for stability. To calculate the
PSNR we used the formula
$\text{PSNR}(x,y)=10\cdot\log_{10}(\frac{{\text{MAX}^{2}}}{{\text{MSE}}(x,y)})$,
where $x$ and $y$ are the two compared images, MAX is the maximum possible
pixel value of the images and $\text{MSE}(x,y)$ is the Mean Squared Error
between the images.

The values obtained from each image pair were used to calculate the mean to
see the similarity between images attacked with and without SR for each
category.

Table 3: Evaluation of similarity between non-SR images and SR ones for each forgery method and for the pristine case. Forgery Method |  SSIM $\uparrow$ | PSNR $\uparrow$ (dB)
---|---|---
_Pristine_ | 0.968 | 39.8
Deepfakes | 0.970 | 40.3
Face2Face | 0.968 | 39.8
FaceShifter | 0.973 | 40.9
FaceSwap | 0.967 | 39.7
NeuralTextures | 0.972 | 40.5

As can be seen from Table 3 the similarity between the SR images and the non-
SR ones is very high, with SSIM values around $0.97$ and PSNR around $40dB$
meaning a strong similarity and minimal changes brought by the SR process. We
also checked if exists a correlation between the SSIM value and the variation
in the error of the classifiers. In other words, we explored if a lower SSIM
value is related to a higher number of misclassifications during the
detection. From our experiments, in all the methods the correlation is lower
than $\pm 0.1$ meaning that the variation in detectors’ performances is more
related to the type of changes done to the image and not to the quantity of
these.

###  5.3 Qualitative Evaluation

To better understand the effect of the SR Attack on images, we visually
analyzed some examples of deepfakes (e.g. Face2Face and FaceSwap) correctly
detected by a Resnet50-based detector before the application of the attack but
misclassified after it.

Figure 3: Examples of fake images that are correctly detected by a
Resnet50-based deepfake detector but not detected when the SR attack is
applied.

These methods tend to introduce rather specific artifacts that, as visible in
Figure 3, are then smoothed by the SR. This makes the work of the deepfake
detector more difficult, as it has learnt to recognize such anomalies. As can
be seen from the figure also the visual difference is minimal, as already
stated by the analysis conducted in Section 5.2, but it is enough to make some
artifacts around the mouth (FaceSwap) or on the nose (Face2Face) to disappear.

##  6 Conclusions

In this work, we examined the impact of applying SR on deepfake images in the
context of deepfake detection. According to our experiments, the use of these
techniques has a huge impact on the performance of deepfake detectors, causing
the FNR to be drastically raised depending on the deepfake generation
technique used and the artifacts introduced by it into the image. Also, a
tendency was observed for deepfake detectors trained on specific deepfake
generation methods to mistake pristine SR images for fake images when the SR
attack is applied, causing the FPR to rise dramatically. In conclusion, SR
attack can become an effective black-box attack in deepfake detection. In
future work, we will explore the impact of detected face resolution on the
attack performance, explore more SR techniques and also see if using SR as a
data augmentation during the training process could be effective to make
detectors robust to this attack.

## Acknowledgments

This work was partially supported by the project SERICS (PE00000014) under the
NRRP MUR program funded by the EU - NGEU and by the H2020 project AI4Media (GA
n. 951911).

## References

  * [1] Deepfakes, https://github.com/deepfakes/faceswap
  * [2] Dfdc 1st place solution, "https://github.com/selimsef/dfdc_deepfake_challenge"
  * [3] Faceswap, https://github.com/MarekKowalski/FaceSwap/
  * [4] Amoroso, R., Morelli, D., Cornia, M., Baraldi, L., Bimbo, A.D., Cucchiara, R.: Parents and children: Distinguishing multimodal deepfakes from natural images (2023)
  * [5] Arefin, M.R., Michalski, V., St-Charles, P.L., Kalaitzis, A., Kim, S., Kahou, S.E., Bengio, Y.: Multi-image super-resolution for remote sensing using deep recurrent networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2020)
  * [6] Baxevanakis, S., Kordopatis-Zilos, G., Galopoulos, P., Apostolidis, L., Levacher, K., Schlicht, I.B., Teyssou, D., Kompatsiaris, I., Papadopoulos, S.: The mever deepfake detection service: Lessons learnt from developing and deploying in the wild. In: International Workshop on Multimedia AI against Disinformation (2022)
  * [7] Caldelli, R., Galteri, L., Amerini, I., Del Bimbo, A.: Optical flow based cnn for detection of unlearnt deepfake manipulations. Pattern Recognition Letters (2021). https://doi.org/https://doi.org/10.1016/j.patrec.2021.03.005, https://www.sciencedirect.com/science/article/pii/S0167865521000842
  * [8] Coccomini, D.A., Esuli, A., Falchi, F., Gennaro, C., Amato, G.: Detecting images generated by diffusers (2023). https://doi.org/10.48550/ARXIV.2303.05275
  * [9] Coccomini, D.A., Messina, N., Gennaro, C., Falchi, F.: Combining efficientnet and vision transformers for video deepfake detection. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds.) Image Analysis and Processing – ICIAP 2022. Springer International Publishing (2022)
  * [10] Coccomini, D.A., Zilos, G.K., Amato, G., Caldelli, R., Falchi, F., Papadopoulos, S., Gennaro, C.: Mintime: Multi-identity size-invariant video deepfake detection (2022). https://doi.org/10.48550/ARXIV.2211.10996
  * [11] Dogoulis, P., Kordopatis-Zilos, G., Kompatsiaris, I., Papadopoulos, S.: Improving synthetically generated image detection in cross-concept settings. ACM (2023). https://doi.org/10.1145/3592572.3592846
  * [12] Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., Ferrer, C.C.: The deepfake detection challenge (dfdc) dataset. arXiv preprint (2020)
  * [13] Guarnera, L., Giudice, O., Guarnera, F., Ortis, A., Puglisi, G., Paratore, A., Bui, L.M.Q., Fontani, M., Coccomini, D.A., Caldelli, R., Falchi, F., Gennaro, C., Messina, N., Amato, G., Perelli, G., Concas, S., Cuccu, C., Orrù, G., Marcialis, G.L., Battiato, S.: The face deepfake detection challenge. Journal of Imaging (2022). https://doi.org/10.3390/jimaging8100263
  * [14] Hou, Y., Guo, Q., Huang, Y., Xie, X., Ma, L., Zhao, J.: Evading deepfake detectors via adversarial statistical consistency. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [15] Huang, Y., Juefei-Xu, F., Guo, Q., Xie, X., Ma, L., Miao, W., Liu, Y., Pu, G.: Fakeretouch: Evading deepfakes detection via the guidance of deliberate noise. arXiv preprint (2020)
  * [16] Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2017)
  * [17] Li, L., Bao, J., Yang, H., Chen, D., Wen, F.: Advancing high fidelity identity swapping for forgery detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
  * [18] Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017)
  * [19] Rossler, A., Cozzolino, D., Verdoliva, L., Riess, C., Thies, J., Niessner, M.: Faceforensics++: Learning to detect manipulated facial images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)
  * [20] Thies, J., Zollhöfer, M., Nießner, M.: Deferred neural rendering: Image synthesis using neural textures (2019). https://doi.org/10.1145/3306346.3323035
  * [21] Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: Real-time face capture and reenactment of rgb videos. Commun. ACM (2018). https://doi.org/10.1145/3292039
  * [22] Xiang, J., Zhu, G.: Joint face detection and facial expression recognition with mtcnn. In: 2017 4th International Conference on Information Science and Control Engineering (ICISCE) (2017). https://doi.org/10.1109/ICISCE.2017.95
  * [23] Yang, W., Zhang, X., Tian, Y., Wang, W., Xue, J.H., Liao, Q.: Deep learning for single image super-resolution: A brief review. IEEE Transactions on Multimedia (2019). https://doi.org/10.1109/TMM.2019.2919431
  * [24] Zheng, Y., Bao, J., Chen, D., Zeng, M., Wen, F.: Exploring temporal coherence for more general video face forgery detection. In: ICCV (2021). https://doi.org/10.1109/ICCV48922.2021.01477


# Conformal Intent Classification and Clarification for Fast and Accurate
Intent Recognition

Floris den Hengst
Vrije Universiteit Amsterdam
<EMAIL_ADDRESS>
&Ralf Wolter
ING Group NV
<EMAIL_ADDRESS>
Patrick Altmeyer
TU Delft
<EMAIL_ADDRESS>
&Arda Kaygan
ING Group NV
<EMAIL_ADDRESS>
###### Abstract

We present Conformal Intent Classification and Clarification (CICC), a
framework for fast and accurate intent classification for task-oriented
dialogue systems. The framework turns heuristic uncertainty scores of any
intent classifier into a clarification question that is guaranteed to contain
the true intent at a pre-defined confidence level. By disambiguating between a
small number of likely intents, the user query can be resolved quickly and
accurately. Additionally, we propose to augment the framework for out-of-scope
detection. In a comparative evaluation using seven intent recognition datasets
we find that CICC generates small clarification questions and is capable of
out-of-scope detection. CICC can help practitioners and researchers
substantially in improving the user experience of dialogue agents with
specific clarification questions.

Conformal Intent Classification and Clarification for Fast and Accurate Intent
Recognition

Floris den Hengst Vrije Universiteit Amsterdam<EMAIL_ADDRESS>Ralf Wolter
ING Group NV<EMAIL_ADDRESS>

Patrick Altmeyer TU Delft<EMAIL_ADDRESS>Arda Kaygan ING Group NV
<EMAIL_ADDRESS>

##  1 Introduction

Intent classification (IC) is a crucial step in the selection of actions and
responses in task-oriented dialogue systems. To offer the best possible
experience with such systems, IC should accurately map user inputs to a
predefined set of intents. A widely known challenge of language in general,
and IC specifically, is that user utterances may be incomplete, erroneous, and
contain linguistic ambiguities.

Although IC is inherently challenging, a key strength of the conversational
setting is that disambiguation or _clarification_ questions (CQs) can be posed
(Purver et al., 2003; Alfieri et al., 2022). Posing the right CQ at the right
time results in a faster resolution of the user query, a more natural
conversation, and higher user satisfaction (van Zeelt et al., 2020; Keyvan and
Huang, 2022; Siro et al., 2022). CQs have been considered in the context of
information retrieval Zamani et al. (2020) but have received little attention
in the context of task-oriented dialogue.

Deciding when to ask a CQ and how to pose it are challenging tasks (DeVault
and Stone, 2007; Keyvan and Huang, 2022). First, it is not clear when the
system can safely proceed under the assumption that the true intent was
correctly identified. Second, it is not clear when the model is too uncertain
to formulate a CQ (Cavalin et al., 2020). Finally, it is unclear what the
exact information content of the clarification question should be.

We present Conformal Intent Classification and Clarification (CICC), a
framework for deciding when to ask a CQ, what its information content should
be, and how to formulate it. The framework uses conformal prediction to turn a
models’ predictive uncertainty into prediction sets that contain the true
intent at a predefined confidence level (Shafer and Vovk, 2008; Angelopoulos
et al., 2023). The approach is agnostic to the intent classifier, does not
require re-training of this model, guarantees that the true intent is in the
CQ, allows for rejecting the input as too ambiguous if the model is too
uncertain, has interpretable hyperparameters, generates clarification
questions that are small and is amenable to the problem of detecting out-of-
scope inputs.

In a comparative evaluations with seven data sets and three IC models, we find
that CICC outperforms heuristic approaches to predictive uncertainty
quantification in all cases. The benefits of CICC are most prominent for
ambiguous inputs, which arise naturally in real-world dialogue settings
(Zamani et al., 2020; Larson et al., 2019).

##  2 Related Work

We discuss related work on ambiguity and uncertainty detection within IC and
CP with language models.

#### Clarification Questions

Various works acknowledge the problem of handling uncertainty in intent
classification and to address it with CQs. Dhole (2020) proposes a rule-based
approach for asking discriminative CQs. The approach is limited to CQs with
two intents, lacks a theoretical foundation, and provides no intuitive way of
balancing coverage with CQ size. Keyvan and Huang (2022) survey ambiguous
queries in the context of conversational search and list sources of ambiguity.
They mention that clarification questions should be short, specific, and based
on system uncertainty. We propose a principled approach to asking short and
specific questions based on uncertainty of any underlying intent classifier
for the purposes of task-oriented dialogue.

Alfieri et al. (2022) propose an approach for asking a CQ containing a fixed
top-$k$ most likely intents with intent-specific uncertainty thresholds. This
approach does not come with any theoretical guarantees and its hyperparameters
need to be tuned on an additional data set whereas our approach comes with
guarantees on coverage of the true intent and with intuitively interpretable
hyperparameters that can be tuned on the same calibration set. We do not
compare directly to this method but include top-$k$ selection in our
benchmark.

CQs have been studied in other domains, including information retrieval Zamani
et al. (2020), product description improvement Zhang and Zhu (2021), and open
question-answering Kuhn et al. (2023). In contrast to the task-specific domain
investigated in this work, these domains leave more room for asking generic
questions for clarification and do not easily allow for incorporating model
uncertainty. Furthermore, the proposed methods require ad hoc tuning of scores
based on heuristic metrics of model uncertainty, and do not provide ways to
directly balance model uncertainty with CQ size.

#### Uncertainty and out-of-scope detection

The out-of-scope detection task introduced by Larson et al. (2019) is a
different task from the task of handling model uncertainty and ambiguous
inputs (Cavalin et al., 2020; Yilmaz and Toraman, 2020; Zhan et al., 2021;
Zhou et al., 2021). However, predictive uncertainty is often used in
addressing the out-of-scope detection task. Although the tasks of handling
ambiguous input and detecting out-of-scope input are different, we briefly
discuss approaches that leverage model uncertainty for out-of-scope detection
here.

Various out-of-scope detection approaches train an intent classifier and tune
a decision boundary based on a measure of the classifier’s confidence Shu et
al. (2017); Lin and Xu (2019); Yan et al. (2020); Hendrycks et al. (2020).
Samples for which the predictive uncertainty of the model lies on one side of
the boundary are classified as out-of-scope. These approaches use the models’
heuristic uncertainty to decide whether an input is out-of-sample whereas we
first turn the models’ heuristic uncertainty into a prediction with
statistical guarantees and then use this prediction to decide when and how to
formulate a clarification question. We additionally propose an adaptation of
the CICC framework for out-of-scope detection.

#### Conformal Prediction on NLP tasks

Conformal Prediction has been used in several NLP tasks, including sentiment
classification by Maltoudoglou et al. (2020), named-entity recognition by
Fisch et al. (2022) and paraphrase detection by Giovannotti and Gammerman
(2021). However, the application to intent classification, task-oriented
dialogue and the combination with CQs presented here is novel to our
knowledge.

##  3 Methodology

We address the problem of asking CQs in task oriented dialogue systems in the
following way. We take a user utterance and a pre-trained intent classifier,
and then return an appropriate response based on the predictive uncertainty of
the model. Algorithm 1 lists these steps, and an example input is presented in
Figure 1. In this section we describe and detail the components of CICC. We
start by providing a background on conformal prediction.

Figure 1: The conformal intent classification and clarification interaction
loop.

###  3.1 Conformal Prediction

Conformal Prediction is a framework for creating statistically rigorous
prediction sets from a heuristic measure of predictive uncertainty of a
classifier (Shafer and Vovk, 2008; Angelopoulos et al., 2023). We here focus
on split conformal prediction as it does not require any retraining of the
underlying model, and refer to it simply as conformal prediction from here on
out.

For a classification task with classes $\mathcal{Y}:\\{1,\dots,K\\}$, a test
input $X_{t}\in\mathcal{X}$ with label $Y_{t}\in\mathcal{Y}$, and a user-
defined error level $\alpha\in\left[0,1\right)$, CP returns a set
$\mathcal{C}(X_{t})\subseteq\mathcal{Y}$ for which the following holds Vovk et
al. (1999) even when using a finite amount of samples:

$\mathbb{P}\left(Y_{t}\in\mathcal{C}(X_{t})\right)\geq 1-\alpha$ (1)

If e.g. $\alpha=0.01$ the set $\mathcal{C}(X_{t})$ is therefore _guaranteed_
to contain the true $Y_{t}$ in 99% of test inputs.

Conformal prediction uses a heuristic measure of uncertainty of a pretrained
model and a modestly sized calibration set to generate prediction sets.
Formally, we assume a held-out calibration set $D:\\{(X_{i},Y_{i})\\}$ of size
$n$, a pre-trained classifier $\hat{f}:\mathcal{X}\to\mathbb{R}^{K}$, and a
nonconformity function $s:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}$ that
returns heuristic uncertainty scores where larger values express higher
uncertainty. An example of a nonconformity function for a neural network
classifier is one minus the softmax outputs of the true class:

$s(X_{i}):=1-\hat{f}(X_{i})_{Y_{i}}.$ (2)

This score is high when the softmax score of the true class is low, i.e., when
the model is badly wrong.

The nonconformity function $s$ is evaluated on $D$ to generate a set of
nonconformity scores $\mathcal{S}=\\{s(X_{i},Y_{i})\\}$. Next, the quantile
$\hat{q}$ of the empirical distribution of $\mathcal{S}$ is determined so that
the desired coverage ratio $(1-\alpha)$ is achieved. This can be done by
choosing $\hat{q}=\lceil(n+1)(1-\alpha)\rceil/n$111this is essentially the
$\hat{q}$ quantile with a minor adjustment where $\lceil\cdot\rceil$ denotes
the ceiling function. Then, for a given test input $X_{t}$, all classes
$y\in\mathcal{Y}$ with high enough confidence are included in a prediction set
$\mathcal{C}(X_{t})$ :

$\mathcal{C}(X_{t}):=\\{y:s(X_{t},y)\leq\hat{q}\\}.$ (3)

This simple procedure guarantees that (1) holds i.e. that the true $Y_{t}$ is
in the set at the specified confidence $1-\alpha$. Note the lack of retraining
or ensembling of classifiers, that the procedure requires little compute and
that $D$ can be relatively small as long as it contains a fair number of
examples for all classes and is exchangeable222distributed identically but not
necessarily independently with the test data Papadopoulos et al. (2002).

There are various implementations of conformal prediction with different
nonconformity functions and performance characteristics. The most simple
approach is known as _marginal_ conformal prediction and it uses the
nonconformity function in (2). Marginal conformal prediction owes its names
from adhering to the guarantee (1) marginalized over $\mathcal{X}$ and
$\mathcal{Y}$, i.e. it satisfies the coverage requirement (1) on average,
rather than e.g. for a particular input $X_{t}$. Marginal CP can be
implemented following the steps described previously: (i) compute
nonconformity scores $S$ using (2), (ii) obtain $\hat{q}$ as described
previously, and (iii) construct a prediction set using (3) at test time. A
benefit of this approach is that it generates prediction sets with the
smallest possible prediction set size on average. A limitation is that its
prediction set sizes may not reflect hardness of the input Sadinle et al.
(2019).

Alternatively, one can ensure conditional adherence to (1) with so-called
conditional or adaptive conformal predictors. A benefit of conditional
approaches is that higher model uncertainty results in larger prediction sets.
However, a downside is that these sets are expected to be larger on average
than those obtained with a marginal approach.Romano et al. (2020) introduce a
conditional CP approach that consists of broadly the same steps as marginal CP
but with a different nonconformity function $s$ and a different prediction set
construction. First we define a permutation $\pi(X)$ of $\\{1\dots K\\}$ that
sorts $\hat{f}(X)$ in descending order. Conditional CP can defined as: (i) sum
all predictor outputs $\hat{f}(X_{i})_{k}$ for all $\\{k\in
K|\hat{f}(X_{i})_{k}\geq\hat{f}(X_{i})_{Y_{i}}\\}$, (ii) obtain $\hat{q}$ as
before, and (iii) include all for a test input $X_{t}$:

$\mathcal{C}(X_{t}):=\\{\pi_{1}(X_{t}),\dots,\pi_{k}(x)\\},$ (4)

where

$k=\text{sup}\left\\{k^{\prime}:\sum_{j=1}^{k^{\prime}}\hat{f}(X_{t})_{\pi_{j}(X_{t})}<\hat{q}\right\\}+1.$
(5)

Angelopoulos et al. (2021) introduce an approach with a term to regularize the
prediction set size: their approach is therefore known as Regularized Adaptive
Prediction Sets (RAPS). It effectively adds an increasing penalty to the
ranked model outputs in the first step of conditional CP in order to promote
smaller prediction sets where possible. Since the second and third step are
similar to conditional CP, its prediction sets still adhere to the coverage
guarantee (1).

In general, a suitable conformal prediction technique strikes the right
balance between three desiderata: (i) adhering to the coverage requirement in
(1), (ii) producing small prediction sets and (iii) adaptivity. Whereas the
former two can be measured easily, metrics for adaptivity require some more
care. Angelopoulos et al. (2021) introduce a general-purpose metric for
adaptivity. It is based on the coverage and referred to as the size-stratified
classification (SSC) score:

$\text{SSC}=\min_{b\in\\{1,\dots,K\\}}\frac{1}{|\mathcal{I}_{b}|}\sum_{i\in\mathcal{I}_{b}}\mathbbm{1}\left\\{Y_{i}\in\mathcal{C}\left(X_{i}\right)\right\\}$
(6)

for a classification task defined as above and
$\mathcal{I}_{b}\subset\\{1,\dots,n\\}$ the set of inputs with prediction set
size $b$, i.e.
$\mathcal{I}_{b}:=\left\\{X_{i},|\mathcal{C}(X_{i})|=b\right\\}$.

Within CICC, conformal prediction is applied to a pre-trained intent
classifier to create a set of intents that contains the true user intent at a
predefined confidence for any user utterance. The sets are then used in making
a decision on when to ask a clarification question and how to formulate it. We
continue to discuss when and how such questions are asked based on Algorithm 1
in the following section.

Algorithm 1 CICC algorithm

Input: utterance $X$, classifier $\hat{f}$, chat/voice-bot $c$, calibration
set $D$, generative LM $g$

Parameters: error rate $\alpha$, threshold $th$, ambiguity response $a$

Output: response $R$

1: set $\leftarrow$ conformal prediction$\left(\hat{f}(X),D,\alpha\right)$

2: if $|$set$|==1$ then

3: $R\leftarrow c($set.get()$)$. {bot response}

4: else if $|$set$|>th$ then

5: $R\leftarrow a$. {input too ambiguous}

6: else

7: $R\leftarrow g($set$,X)$ {clarification question}

8: end if

###  3.2 When to Ask a Clarification Question

For a user utterance $X$, a pre-trained intent classifier $\hat{f}$ and a
nonconformity function $s$, we generate a prediction set that covers the true
user intent with confidence $1-\alpha$ (Algorithm 1, ln 1). If the set
contains a single intent, the model is confident that the true intent has been
detected and the dialogue can be handled as usual (ln 2-3).

If the set contains many intents, that is, more than a user-specified
threshold $th\in\mathbb{N}_{>0}$, then there is no reasonable ground for
formulating a clarification question. Instead, a generic request to rephrase
the question can be asked (ln 4-5), or a hand-over to a human operator could
be implemented here. In the remaining case, i.e. if the prediction set is of
reasonable size, a CQ is asked (ln 6-7).

CICC comes with two parameters to control when a CQ should be asked. Both have
clear semantics and can be interpreted intuitively. The first is the threshold
$th$ that controls when the input is too ambiguous to ask a CQ (Algorithm 1 ln
4-5). This parameter is set by the chatbot owner on the basis of best
practices in, and knowledge of chat- and voicebot interaction patterns. In
general, this number should remain small to reduce the cognitive load on
users. We advise to set this value no higher than seven Miller (1956); Plass
et al. (2010).

The second parameter is the error rate $\alpha$. It controls the trade-off
between the prediction set size and how certain we want to be that the
prediction set covers the true intent. As $\alpha\to 0$, our confidence that
the true intent is included in the set grows, but so does the size of the
prediction set. Because conformal prediction is not compute intensive,
$\alpha$ can be set empirically. Thus, CICC provides a means of selecting
between _achievable_ trade-offs between prediction set sizes and error rates.
We continue to discuss how specific CQs are formulated in CICC.

###  3.3 Generating a Clarification Question

When a CQ is in order (ln 6-7 in Alg. 1), it needs to be formulated. We
propose to generate a CQ based on the original input $X$ and the prediction
set, as it is guaranteed to contain the true intent at a typically high level
of confidence. Because the alternatives in the CQ are the most likely intents
according to the model, and because the number of alternatives in the CQ
corresponds to the models’ uncertainty, asking a CQ provides a natural way of
communicating model uncertainty to the user while quickly determining the true
user intent.

CICC makes no assumptions about the approach for generating a CQ. Anything
from hardcoded questions, templating, or a generative LM can be used. However,
we recognize that the number of possible questions is large: it consists of
the powerset of all $n$ intents up to size $th$ excluding sets of size one and
zero. Therefore, we opt to use a generative LM in our solution.

We prompt the LM to formulate a clarification question by giving it some
examples of clarification questions for a set of example intents to
disambiguate between. We additionally provide the original utterance $X$ to
enable the formulation of CQ relative to the original utterance. See Appendix
A for details.

###  3.4 Out-of-scope Detection

Ambiguity is a part of natural language which could lead to model uncertainty.
Specific reasons for uncertainty in intent recognition are inputs that are
very short and long, imprecise and incomplete inputs, etc. However, a
particularly interesting type of uncertainty stems from inputs that represent
intent classes that are not known at training time Zhan et al. (2021). These
inputs are referred to as out-of-scope (OOS) and detecting these inputs can be
seen as a binary classification task for which data sets with known OOS
samples have been developed.

CICC rejects inputs about which the model is too uncertain (Algorithm 1, ln 5)
and this naturally fits with the OOS detection task as follows: we can view a
rejection of an input as a classification of that input as OOS. Therefore,
although handling ambiguity in the model gracefully and detection OOS inputs
are separate challenges, vanilla CICC implements a form of OOS detection.

Additionally, the CICC framework can be leveraged for OOS detection if OOS
samples are known at calibration time. Specifically, we can optimize
parameters $\alpha$ and $th$ to maximize predictive performance expressed by
some suitable metric such as the F1-score on the calibration set. OOS samples
can be obtained from other intent recognition data sets in other domains. This
practice is described in detail by e.g. Zhan et al. (2021) under the name of
open-domain outliers. We refer to versions of CICC which have been optimized
for F1-score in this way as CICC-OOS.

##  4 Experimental Setup

This section lists the experiments performed to comparatively evaluate CICC
across seven data sets and on three IC models333
https://github.com/florisdenhengst/cicc.

| samples | intents
---|---|---
ACID Acharya and Fung (2020) | 22172 | 175
ATIS Hemphill et al. (1990) | 5871 | 26
B77 Casanueva et al. (2020) | 13083 | 77
B77-OOS | 16337 | 78
C150-IS Larson et al. (2019) | 18025 | 150
C150-OOS Larson et al. (2019) | 19025 | 151
HWU64 Liu et al. (2021) | 25716 | 64
IND |  $\sim$20k | 61
MTOD (eng) Schuster et al. (2019) | 43323 | 12
Table 1: Characteristics of datasets used

#### Data

We evaluate CICC on six public intent recognition data sets in English and an
additional real-life industry data set (IND) from the banking domain in the
Dutch language. Table 1 shows the data sets and their main characteristics.
All data sets were split into train-calibration-test splits of proportions
0.6-0.2-0.2 with stratified sampling, except for the ATIS data set in which
stratified sampling is impossible due to the presence of intents with a single
sample. Random sampling was used for this data set instead. We use an in-scope
version (C150-IS) of the ‘unbalanced’ data set by Larson et al. (2019) in
which all out-of-scope samples have been removed.

For evaluation on out-of-scope (OOS) detection, we use two datasets: a version
of C150 with all OOS samples divided over the calibration and test splits, and
no OOS samples in the train split (C150-OOS), and a version of B77 with so-
called open-domain outliers in which samples from the ATIS dataset make up
half of the samples in the calibration and test splits to represent OOS inputs
(B77-OOS) Zhan et al. (2021).

#### Models

We employ fine-tuned BERT by Devlin et al. (2019) for all public data sets and
a custom model similar to BERT for the IND data set Alfieri et al. (2022). We
base the nonconformity scores on the softmax output in these settings. In
order to test performance on a commercial offering, we additionally evaluate
using DialogflowCX (DFCX) on the B77 data
set.444https://cloud.google.com/dialogflow/cx/docs This commercial offering
outputs heuristic certainty scores in the range $\left[0,100\right]$ for the
top five most certain recognized intents. These outputs were normalized to sum
to $1$, all other scores were set to $0$ to determine the nonconformity
scores.

#### Baselines

In practice CQs can be formulated using heuristics Alfieri et al. (2022). We
compare CICC to the following baselines using the models’ heuristic
uncertainty scores:

  * B1

select all intents with score $>1-\alpha$, select the top $k=5$ if this
selection is empty.

  * B2

select all intents with a score $>1-\alpha$.

  * B3

select the top $k=5$ intents.

#### Metrics

We evaluate the approaches on a set of metrics that together accurately convey
the added benefit of asking a confirmation question. We use the _size_ of the
prediction set $\mathcal{C}(X_{i})$ and how often the input is rejected as too
ambiguous for the model (Algorithm 1, ln 5). For a test set of size $n$:

$\text{Amb}:=\frac{1}{n}\sum_{i=0}^{n}\begin{cases}1&\text{if}~{}|\mathcal{C}(X_{i})|\geq
th\\\ 0&\text{otherwise}.\end{cases}$ (7)

First, we report how often the true intent is detected for the $m\leq n$
inputs that are not rejected (Algorithm 1, lns 3 and 5). This metric is known
as coverage (cov) and can be seen as a generalisation of accuracy for set-
valued predictions:

$\text{Cov}:=\frac{1}{m}\sum_{i=0}^{m}\mathbbm{1}_{\mathcal{C}(X_{i})}\left(Y_{i}\right).$
(8)

Second, we report the average size of the clarification questions for accepted
inputs (Algorithm 1, ln 7). This metric can be seen as an analogue to
precision for set-valued predictions:

$|\text{CQ}|=\frac{1}{m}\sum_{i=0}^{m}|\mathcal{C}(X_{i})|.$ (9)

Finally, we report the relative number of times the prediction set is of size
one

$\text{Single}:=\frac{1}{m}\sum_{i=0}^{m}\begin{cases}1&\text{if}~{}|\mathcal{C}(X_{i})|=1,\\\
0&\text{otherwise,}\end{cases}$ (10)

in which case the dialogue can continue as usual (Algorithm 1, ln 3). We
additionally report the SSC as defined above in (6).

For out-of-scope detection we report the standard metrics F1-score and AUROC.

#### Parameters

We varied $\alpha$ and found the best settings empirically on the calibration
set. We report our key results for the best $\alpha$ and additionally
investigate the effect of varying $\alpha$.

We set the threshold $th$ at seven to avoid excessive cognitive load for users
for all experiments, except when using DFCX in which case we set $th$ to four
Miller (1956); Plass et al. (2010). The reason for this is that DFCX currently
only outputs non-zero scores for the top five intents. Hence, the set contains
all intents that have a non-zero confidence score with this setting.

We include the following conformal prediction approaches and select an
approach that produces the best empirical results in terms of coverage and CQ
size: marginal, conditional (also known as adaptive) Romano et al. (2020) and
RAPS Angelopoulos et al. (2021). Marginal conformal prediction was selected in
all experiments, details can be found in Figure 2.

##  5 Results

Setting | $1-\alpha$ | $th$ |  | Cov$\uparrow$ | Single$\uparrow$ | $|\text{CQ}|\downarrow$ | Amb
---|---|---|---|---|---|---|---
ACID | .98 | 7 | CICC | .98 | .87 | 3.01 | .03
|  |  | B1 | .98 | .88 | 5 | 0
|  |  | B2 | .95 | 1 | $-$ | 0
|  |  | B3 | .99 | 0 | 5 | 0
ATIS | .99 | 7 | CICC | .99 | .98 | 2.54 | 0
|  |  | B1 | .99 | .73 | 5 | 0
|  |  | B2 | .98 | 1.00 | - | 0
|  |  | B3 | 1.00 | 0 | 5 | 0
B77/BERT | .97 | 7 | CICC | .98 | .73 | 2.84 | .04
|  |  | B1 | .97 | .84 | 5 | 0
|  |  | B2 | .93 | 1 | $-$ | 0
|  |  | B3 | .98 | 0 | 5 | 0
B77/DFCX | .90 | 4 | CICC | .91 | .66 | 2.63 | .02
|  |  | B1 | .95 | .71 | 5 | .27
|  |  | B2 | .90 | .98 | 2.26 | 0
|  |  | B3 | .97 | 0 | 5 | 1
C150-ID | .99 | 7 | CICC | .99 | .97 | 2.66 | 0
|  |  | B1 | .99 | .82 | 5 | 0
|  |  | B2 | .98 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
HWU64 | .95 | 7 | CICC | .95 | .82 | 2.81 | .01
|  |  | B1 | .97 | .70 | 5 | 0
|  |  | B2 | .90 | 1 | $-$ | 0
|  |  | B3 | .98 | 0 | 5 | 0
IND | .90 | 7 | CICC | .91 | .25 | 3.46 | .11
|  |  | B1 | .88 | .42 | 5 | 0
|  |  | B2 | .70 | 1 | $-$ | 0
|  |  | B3 | .91 | 0 | 5 | 0
MTOD | .99 | 7 | CICC | .99 | 1 | $-$ | 0
|  |  | B1 | 1 | .98 | 5 | 0
|  |  | B2 | .99 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
Table 2: Test set results where underline indicates meeting coverage
requirement. Bold denotes best when meeting this requirement, omitted for last
column due to missing ground truth for ambiguous.

Table 2 lists the main results. The first column shows the coverage, i.e. the
percentage of test samples in which the ground truth is captured in the
prediction set. We see that only CICC and B3 adhere to the requirement of
coverage $\geq 1-\alpha$ in all settings. The second column shows the fraction
of test samples for which a single intent is detected. We see that CICC
outperforms the baselines that meet the coverage requirement in five out of
seven data sets.

The third column lists the average size of the CQ. We see that CICC yields the
smallest CQs and that the number of inputs that is deemed too ambiguous is
relatively small for CICC. The last column denotes the relative number of
inputs that is rejected as too ambiguous. CICC rejects a relatively low number
of inputs. Upon inspection, many of these inputs could be classified as
different intents based on the textual information alone (see Appendix B). For
the B77/DFCX setting, we see that B1 predicts a single output frequently, at
the cost of rejecting inputs as too ambiguous. This contrasts with CICC, which
rejects inputs much less frequently and instead asks a small CQ.

Dataset | Algorithm | 1-$\alpha$ | $th$ | F1$\uparrow$ | AUROC$\uparrow$
---|---|---|---|---|---
C150-OOS | CICC | .990 | 7 | .07 | .88
| CICC-OOS | .995 | 6 | .91 | .97
B77-OOS | CICC | .970 | 7 | .76 | .92
| CICC-OOS | .994 | 6 | .90 | .97
Table 3: Results for the OOS detection task.

Figure 2: Test set results for varying error rate $\alpha$.

We continue by looking at the results for OOS detection in Table 3. We find
that vanilla CICC does not perform well on the OOS detection in comparison to
the specialized CICC-OOS variant. The specialized CICC-OOS favours a
relatively low $\alpha$ as this simultaneously forces the approach toward
large prediction sets for OOS samples and small prediction sets for in-sample
inputs. At the same time, using the CICC-OOS settings for parameters $\alpha$
and $th$ in the regular CICC interaction loop would result in relatively many
CQs of a relatively large size.

Next, we investigate how different conformal prediction approaches perform for
varying levels of $\alpha$ in Figure 2. The top figures show that all
conformal prediction approaches enable trading off set size with coverage, a
desirable property in practice of intent classification. Looking at the
adaptivity (center figures), we see mixed results. A possible explanation for
this is in the general-purpose evaluation of adaptivity, which relies on the
minimum coverage across classes (see Eq. 6). The data sets used in our
experiments contain a relatively low number of examples for some classes and
these rare classes may have an outsized effect on the SSC metric. Looking at
the bottom figure for each data set, we see that all conformal prediction
approaches lie at or above the x=y diagonal: conformal prediction always
adheres to the coverage requirement with the marginal approach yielding the
smallest average set sizes.

##  6 Conclusion

We have proposed a framework for detecting and addressing uncertainty in
intent classification with conformal prediction. The framework empirically
determines when to ask a clarification question and how that question should
be formulated. The framework uses a moderately sized calibration set and comes
with intuitively interpretable parameters.

We have evaluated the framework in eight settings, and have found that the
framework strictly outperforms baselines across all metrics in six out of
eight cases and performs competitively in the other. The framework
additionally handles inputs that are too ambiguous for intent classification
naturally. We have additionally proposed and evaluated the usage of CICC for
out-of-scope detection and found that it is suitable for this.

We finally believe that the framework opens promising avenues for future work,
including the usage of intent groups for better adaptivity, an extension to
Bayesian models to address data drift and unsupervised OOS with CICC (Fong and
Holmes, 2021), to determine conversation stopping rules based on subsequent
questions to rephrase or clarify and to combine it with reinforcement learning
for, e.g., personalization (Den Hengst et al., 2019, 2020). We believe that
CICC and/or conformal prediction may also prove useful in various other tasks,
including entity recognition, detecting label errors (Ying and Thomas, 2022)
and to empirically identify similar intents.

## Limitations

A limitation of the framework is that it relies on a user determining values
for the hyperparameters $\alpha$ and $th$. The former balances model certainty
with CQ size. Arguably, this trade-off has to be made in any approach and CICC
makes this an explicit choice between achievable trade-offs. The threshold
$th$ must be set not to reject too many inputs as too ambiguous while avoiding
information overload in the user. We advise setting it to no more than seven
based on established insights from cognitive science Miller (1956). However,
more research on the impact of CQ size on user satisfaction in various context
is in order. Another limitation is that the approach does not include a
mechanism for stopping the dialogue. We leave the investigation of stopping
criteria based on e.g. the number and size of CQs asked during the dialogue
for future work. Furthermore, this work did not thoroughly investigate the
quality of the CQs produced by the LLM. However, we view the CQ production
component as a pluggable component and therefore believe a full-scale
evaluation on this to be out-of-scope for this work. Additionally, using CICC
for OOS detection requires the presence of OOS labels. While these can be
obtained from other data sets using the practice of open-domain outliers Zhan
et al. (2021), fully unsupervised approaches based on e.g. hierarchical
Bayesian modeling or with parameters that yield good performance across data
sets as hinted at by Table 3. A final limitation is that we applied conformal
prediction to the softmax of outputs of uncalibrated neural network outputs.
This makes results consistent across settings (including DFCX), but smaller
CQs may be achievable by applying Platt scaling prior to conformal prediction
calibration Platt et al. (1999).

## Acknowledgements

We thank Mark Jayson Doma and Jhon Cedric Arcilla for their help in obtaining
and understanding DialogflowCX model output. We kindly thank the reviewers for
their time and their useful comments, without which this work would not have
been possible in its current form.

## References

  * Acharya and Fung (2020) Shailesh Acharya and Glenn Fung. 2020.  Using optimal embeddings to learn new intents with few examples: An application in the insurance domain.  In _KDD 2020 Workshop on Conversational Systems Towards Mainstream Adoption(KDD Converse 2020)_. CEUR-WS.org.
  * Alfieri et al. (2022) Andrea Alfieri, Ralf Wolter, and Seyyed Hadi Hashemi. 2022.  Intent disambiguation for task-oriented dialogue systems.  In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, pages 5079–5080.
  * Angelopoulos et al. (2023) Anastasios N Angelopoulos, Stephen Bates, et al. 2023.  Conformal prediction: A gentle introduction.  _Foundations and Trends® in Machine Learning_ , 16(4):494–591.
  * Angelopoulos et al. (2021) Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. 2021.  Uncertainty sets for image classifiers using conformal prediction.  In _International Conference on Learning Representations_.
  * Casanueva et al. (2020) Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020.  Efficient intent detection with dual sentence encoders.  In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_ , pages 38–45.
  * Cavalin et al. (2020) Paulo Cavalin, Victor Henrique Alves Ribeiro, Ana Appel, and Claudio Pinhanez. 2020.  Improving out-of-scope detection in intent classification by using embeddings of the word graph space of the classes.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3952–3961.
  * Den Hengst et al. (2020) Floris Den Hengst, Eoin Martino Grua, Ali el Hassouni, and Mark Hoogendoorn. 2020.  Reinforcement learning for personalization: A systematic literature review.  _Data Science_ , 3(2):107–147.
  * Den Hengst et al. (2019) Floris Den Hengst, Mark Hoogendoorn, Frank Van Harmelen, and Joost Bosman. 2019.  Reinforcement learning for personalized dialogue management.  In _IEEE/WIC/ACM International Conference on Web Intelligence_ , pages 59–67.
  * DeVault and Stone (2007) David DeVault and Matthew Stone. 2007.  Managing ambiguities across utterances in dialogue.  In _Proceedings of the 11th Workshop on the Semantics and Pragmatics of Dialogue (Decalog 2007)_ , pages 49–56.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dhole (2020) Kaustubh D Dhole. 2020.  Resolving intent ambiguities by retrieving discriminative clarifying questions.  _arXiv preprint arXiv:2008.07559_.
  * Fisch et al. (2022) Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. 2022.  Conformal prediction sets with limited false positives.  In _International Conference on Machine Learning_ , pages 6514–6532. PMLR.
  * Fong and Holmes (2021) Edwin Fong and Chris C Holmes. 2021.  Conformal bayesian computation.  _Advances in Neural Information Processing Systems_ , 34:18268–18279.
  * Giovannotti and Gammerman (2021) Patrizio Giovannotti and Alex Gammerman. 2021.  Transformer-based conformal predictors for paraphrase detection.  In _Conformal and Probabilistic Prediction and Applications_ , pages 243–265. PMLR.
  * Hemphill et al. (1990) Charles T Hemphill, John J Godfrey, and George R Doddington. 1990.  The atis spoken language systems pilot corpus.  In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990_.
  * Hendrycks et al. (2020) Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020.  Pretrained transformers improve out-of-distribution robustness.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2744–2751.
  * Keyvan and Huang (2022) Kimiya Keyvan and Jimmy Xiangji Huang. 2022.  How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges.  _ACM Computing Surveys_ , 55(6):1–40.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  CLAM: Selective clarification for ambiguous questions with large language models.  In _ICML Workshop Challenges of Deploying Generative AI_.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  * Lin and Xu (2019) Ting-En Lin and Hua Xu. 2019.  Deep unknown intent detection with margin loss.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5496.
  * Liu et al. (2021) Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2021.  Benchmarking natural language understanding services for building conversational agents.  In _Increasing Naturalness and Flexibility in Spoken Dialogue Interaction: 10th International Workshop on Spoken Dialogue Systems_ , pages 165–183. Springer.
  * Maltoudoglou et al. (2020) Lysimachos Maltoudoglou, Andreas Paisios, and Harris Papadopoulos. 2020.  Bert-based conformal predictor for sentiment analysis.  In _Conformal and Probabilistic Prediction and Applications_ , pages 269–284. PMLR.
  * Miller (1956) George A Miller. 1956.  The magical number seven, plus or minus two: Some limits on our capacity for processing information.  _Psychological review_ , 63(2):81.
  * Papadopoulos et al. (2002) Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. 2002.  Inductive confidence machines for regression.  In _Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13_ , pages 345–356. Springer.
  * Plass et al. (2010) Jan L Plass, Roxana Moreno, and Roland Brünken, editors. 2010.  _Cognitive load theory._ Cambridge University Press, New York, NY, US.
  * Platt et al. (1999) John Platt et al. 1999.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ , 10(3):61–74.
  * Purver et al. (2003) Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003.  On the means for clarification in dialogue.  _Current and new directions in discourse and dialogue_ , pages 235–255.
  * Romano et al. (2020) Yaniv Romano, Matteo Sesia, and Emmanuel Candes. 2020.  Classification with valid and adaptive coverage.  _Advances in Neural Information Processing Systems_ , 33:3581–3591.
  * Sadinle et al. (2019) Mauricio Sadinle, Jing Lei, and Larry Wasserman. 2019.  Least ambiguous set-valued classifiers with bounded error levels.  _Journal of the American Statistical Association_ , 114(525):223–234.
  * Schuster et al. (2019) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019.  Cross-lingual transfer learning for multilingual task oriented dialog.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3795–3805.
  * Shafer and Vovk (2008) Glenn Shafer and Vladimir Vovk. 2008.  A tutorial on conformal prediction.  _Journal of Machine Learning Research_ , 9(3).
  * Shu et al. (2017) Lei Shu, Hu Xu, and Bing Liu. 2017.  Doc: Deep open classification of text documents.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2911–2916.
  * Siro et al. (2022) Clemencia Siro, Mohammad Aliannejadi, and Maarten de Rijke. 2022.  Understanding user satisfaction with task-oriented dialogue systems.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2018–2023.
  * van Zeelt et al. (2020) Mickey van Zeelt, Floris den Hengst, and Seyyed Hadi Hashemi. 2020.  Collecting high-quality dialogue user satisfaction ratings with third-party annotators.  In _Proceedings of the 2020 Conference on Human Information Interaction and Retrieval_ , pages 363–367.
  * Vovk et al. (1999) Volodya Vovk, Alexander Gammerman, and Craig Saunders. 1999.  Machine-learning applications of algorithmic randomness.  In _Proceedings of the Sixteenth International Conference on Machine Learning_ , pages 444–453.
  * Yan et al. (2020) Guangfeng Yan, Lu Fan, Qimai Li, Han Liu, Xiaotong Zhang, Xiao-Ming Wu, and Albert YS Lam. 2020.  Unknown intent detection using gaussian mixture model with an application to zero-shot intent classification.  In _Proceedings of the 58th annual meeting of the association for computational linguistics_ , pages 1050–1060.
  * Yilmaz and Toraman (2020) Eyup Halit Yilmaz and Cagri Toraman. 2020.  Kloos: Kl divergence-based out-of-scope intent detection in human-to-machine conversations.  In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_ , pages 2105–2108.
  * Ying and Thomas (2022) Cecilia Ying and Stephen Thomas. 2022.  Label errors in banking77.  In _Proceedings of the Third Workshop on Insights from Negative Results in NLP_ , pages 139–143.
  * Zamani et al. (2020) Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020.  Generating clarifying questions for information retrieval.  In _Proceedings of the web conference 2020_ , pages 418–428.
  * Zhan et al. (2021) Li-Ming Zhan, Haowen Liang, Bo Liu, Lu Fan, Xiao-Ming Wu, and Albert YS Lam. 2021.  Out-of-scope intent detection with self-supervision and discriminative training.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ , pages 3521–3532.
  * Zhang and Zhu (2021) Zhiling Zhang and Kenny Zhu. 2021.  Diverse and specific clarification question generation with keywords.  In _Proceedings of the Web Conference 2021_ , pages 3501–3511.
  * Zhou et al. (2021) Wenxuan Zhou, Fangyu Liu, and Muhao Chen. 2021.  Contrastive out-of-distribution detection for pretrained transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.

##  Appendix A Appendix: Implementation Details

We used python v3.10.9 with packages numpy and pandas for data manipulation
and basic calculations, matplotlib to generate illustrations, mapie for
conformal prediction and reproduced these results in Julia and the package
conformalprediction.jl. We used the huggingface API for fine tuning a version
of bert-base-uncased using the hyperparameters below. For an anonymized
version of the code and data see https://anonymous.4open.science/r/cicc-205A.

⬇

learning_rate = 4.00e-05

warmup_proportion = 0.1

train_batch_size = 32

eval_batch_size = 32

num_train_epochs = 5

###  A.1 Generative Language Model

We use the eachadea/vicuna-7b-1.1 variant of the LLAMA model using the
HuggingFace API for the experiments presented here. We here provide an example
prompt:

    Customers asked an ambiguous question. Complete each set with a disambiguation question.

    Set 1: Customer Asked: ’The terminal I paid at wouldn’t take my card. Is something wrong?’
    Option 1: ’card not working’
    Option 2: ’card swallowed’
    Disambiguation Question: ’I understand this was about you card. Was is swallowed or not working?’
    **END**

    Set 2:
    Customer Asked: ’I have a problem with a transfer. It didn’t work. Can you tell me why?’
    Option 1: ’declined transfer’
    Option 2: ’failed transfer’
    Disambiguation Question: ’I see you are having issues with your transfer. Was your transfer failed or declined?’
    **END**

    Set 3: Customer Asked: ’I transferred some money but it is not here yet’
    Option 1: ’balance not updated after bank transfer’
    Option 2: ’transfer not received by recipient’
    Disambiguation Question:

More efforts can be spent on prompt engineering and more advanced generative
LMs can be used, which we expect to improve the user satisfaction of CICC.
Alternatively, simple text templates can be used. We consider the following
alternatives and list some of their expected benefits and downsides:

Templates

a simple template-based can be used in which the user is asked to
differentiate between the identified intents. Benefits of templates include
full control over the chatbot output but a downside is that the CQs will be
less varied, possibly sounding less natural and will not refer back to the
users’ original utterance,

LM without user input

when using a LM, it is possible to not incorporate the user input $X$ in the
prompt. This has the benefit of blocking any prompt injection but the downside
of possibly unnatural CQs due to the inability to refer to the user query,

LM with user input

by incorporating the user utterance into the LM prompt for CQ generation, the
CQ can refer back to the user’s phrasing and particular question, and
therefore be formulated in a possibly more natural way.

We believe that more research is warranted to identify which of these
approaches is most applicable in which cases, and how possible downsides of
these alternatives can be mitigated in practice.

##  Appendix B Appendix: Sample ambiguous inputs

Tables 4\- 5 list inputs that are considered ambiguous by CICC in the B77 and
HWU64 data sets respectively. Some inputs could refer to multiple intents
whereas some other inputs could be considered out-of-scope.

# |  Utterance |  Label |  Prediction Set
---|---|---|---
1 |  what is the matter? |  direct debit payment not recognised |  activate my card, age limit, balance not updated after bank transfer, balance not updated after cheque or cash deposit, beneficiary not allowed, cancel transfer, card arrival, card delivery estimate, card not working, card swallowed, cash withdrawal not recognised, change pin, compromised card, contactless not working, country support, declined card payment, declined transfer, direct debit payment not recognised, exchange rate, failed transfer, get physical card, lost or stolen card, lost or stolen phone, pending card payment, pending cash withdrawal, pending transfer, pin blocked, Refund not showing up, reverted card payment?, terminate account, top up failed, top up reverted, transaction charged twice, transfer not received by recipient, transfer timing, unable to verify identity, why verify identity, wrong amount of cash received,
2 |  Can I choose when my card is delivered? |  card delivery estimate |  activate my card, card about to expire, card acceptance, card arrival, card delivery estimate, change pin, contactless not working, country support, get physical card, getting spare card, getting virtual card, lost or stolen card, order physical card, supported cards and currencies, top up by bank transfer charge, top up by card charge, visa or mastercard
3 |  My contanctless has stopped working |  contactless not working |  activate my card, apple pay or google pay, automatic top up, beneficiary not allowed, cancel transfer, card not working, card payment wrong exchange rate, contactless not working, declined card payment, disposable card limits, failed transfer, get disposable virtual card, get physical card, pending top up, pin blocked, top up failed, top up reverted, topping up by card, virtual card not working, visa or mastercard, wrong exchange rate for cash withdrawal
4 |  I misplaced my card and I dont know where the last place is where I used the card last. Can you look at my account and tell me the last place I used the card? |  lost or stolen card |  activate my card, atm support, card acceptance, card linking, card swallowed, cash withdrawal not recognised, compromised card, lost or stolen card, lost or stolen phone, order physical card, pin blocked
5 |  Is my card denied anywhere? |  card acceptance |  atm support, card acceptance, card not working, card payment fee charged, card swallowed, compromised card, contactless not working, declined card payment, lost or stolen card, lost or stolen phone, order physical card, unable to verify identity, visa or mastercard
Table 4: A sample of prediction sets on B77 of size $>th$ of seven with marginal conformal prediction on BERT outputs. Plausible labels have been highlighted with underscore. # |  Utterance |  Label |  Prediction Set
---|---|---|---
1 |  olly |  recommendation events |  calendar set, general quirky, lists createoradd, music likeness, music query, play game, play music, play radio,
2 |  this song is too good |  music likeness |  audio volume mute, general affirm, general commandstop, general joke, general negate, lists remove, music dislikeness, music likeness
3 |  do i have to go to the gym |  general quirky |  calendar query, general quirky, lists query, recommendation events, recommendation locations, transport traffic, weather query
4 |  silently adjust |  audio volume mute |  audio volume down, audio volume other, audio volume up, iot hue lightchange, iot hue lightdim, iot hue lightup, music settings
5 |  how many times does it go |  general quirky |  datetime query, general quirky, lists query, qa factoid, qa maths, transport query, transport traffic
6 |  sports head lines please |  news query |  calendar set, general quirky, iot hue lightchange, music likeness, news query, qa factoid, social post, weather query
7 |  read that back |  play audiobook |  email addcontact, email query, email querycontact, email sendemail, general quirky, lists createoradd, music likeness, play audiobook, play music, social post,
8 |  i don’t want to hear any more songs of that type |  music dislikeness |  audio volume mute, calendar remove, general commandstop, iot wemo off, lists remove, music dislikeness, music likeness
9 |  check celebrity wiki |  general quirky |  email query, general quirky, lists query, news query, qa factoid, social post, social query
10 |  Get all availables |  lists query |  email addcontact, email query, email querycontact, email sendemail, social post, social query, takeaway order,
11 |  rating |  music likeness |  cooking recipe, general quirky, lists createoradd, lists query, music likeness, music query, qa definition, qa factoid,
12 |  take me to mc donalds |  transport query |  play game, play podcasts, recommendation events, recommendation locations, recommendation movies, takeaway order, takeaway query
13 |  search |  qa factoid |  email querycontact, general quirky, lists createoradd, lists query, music query, qa definition, qa factoid,
14 |  unmute |  audio volume up |  audio volume down, audio volume mute, audio volume up, iot wemo off, music settings, play radio, transport query, transport traffic
15 |  please unmute yourself |  audio volume mute |  alarm remove, audio volume down, audio volume mute, audio volume up, iot cleaning, iot wemo on, music settings, play game
16 |  what’s the best day next week to go out for pizza |  datetime query |  calendar query, cooking recipe, general quirky, qa factoid, recommendation events, recommendation locations, takeaway query
17 |  i need a manger |  general quirky |  calendar set, cooking recipe, general quirky, lists createoradd, music likeness, play game, qa definition, qa factoid, social post,
18 |  assistant shuffle entire library |  play music |  iot cleaning, iot hue lightchange, lists createoradd, music settings, play audiobook, play game, play music
19 |  put the disco lights on |  iot hue lighton |  alarm remove, iot cleaning, iot hue lightchange, iot hue lightoff, iot hue lighton, iot hue lightup, iot wemo on
20 |  hello how are you today |  general greet |  general greet, general praise, general quirky, play radio, recommendation events, recommendation locations, recommendation movies
21 |  where does tar work currently |  email querycontact |  cooking recipe, email querycontact, general quirky, lists query, qa definition, recommendation locations, takeaway query
22 |  can you pull up jeff |  email querycontact |  general quirky, iot cleaning, news query, play game, play music, play radio, social post
23 |  show me theatre where pirates of caribbean is playing |  recommendation movies |  general quirky, play game, play music, play podcasts, recommendation events, recommendation locations, recommendation movies
24 |  spot |  play podcasts |  calendar set, music likeness, music query, play music, play radio, qa factoid, weather query,
25 |  anything i need to know |  general quirky |  calendar query, calendar set, email query, email querycontact, general quirky, lists query, news query, social query
26 |  give me some good sounds |  play music |  audio volume down, audio volume up, music likeness, music query, music settings, play music, play radio
27 |  i don’t like it |  music dislikeness |  calendar remove, general commandstop, general dontcare, general joke, general negate, lists remove, music dislikeness, music likeness, social post
28 |  what are you |  general quirky |  audio volume down, audio volume mute, audio volume up, music query, music settings, qa definition, recommendation locations, recommendation movies
29 |  i don’t like this one |  general quirky |  audio volume mute, calendar remove, general joke, general negate, iot wemo off, lists remove, music dislikeness, social post
Table 5: Prediction sets on ACID of size $>th$ of seven with marginal
conformal prediction on BERT outputs. Plausible labels have been highlighted
with underscore.

##  Appendix C Appendix: LLM results

We here present a random sample of CQs on B77 and C150.

Utterance |  Prediction Set |  LLM output
---|---|---
I hope you can help me. My account has been compromised from a different town that I haven’t been to. Can you please place a hold on my account until I can make it to the bank? |  cash withdrawal not recognised, compromised card |  I understand you are having issues with your account. Was your account compromised or your card not recognised for cash withdrawal?
Is my card denied anywhere? |  card not working, declined card payment |  I understand you are asking about your card. Was it declined or not working?
Money that I have transferred hasn’t arrived yet |  balance not updated after bank transfer, transfer not received by recipient |  I understand you are waiting for a transfer to be updated in your balance. Was the transfer not received by you or the recipient?
How long will the transfer take? |  balance not updated after bank transfer,pending transfer, transfer timing |  I understand you are asking about transfer timing. Was your transfer pending, not updated or not received?
What should I do with my stuck ATM? |  card swallowed, declined cash withdrawal |  I understand you have a stuck ATM. Was it your card that got stuck or was your cash withdrawal declined?
Table 6: Sample of LLM outputs on the B77 dataset where the user input has
been made part of the prompt.

Utterance |  Prediction Set |  LLM output
---|---|---
how many more miles can i drive on my current tank |  gas, mpg |  I understand you are asking about your car’s fuel. Are you asking about the number of miles you can drive on your current tank or the miles per gallon?
can i redeem my points for flights |  redeem rewards, rewards balance |  I understand you would like to redeem your points for flights. Are you looking to check your rewards balance or redeem your rewards?
are there any meetings between 2 and 5 pm |  schedule meeting, meeting schedule |  I understand you are looking for a schedule of meetings. Are you looking for a specific time range or a general schedule?
is the marathon on friday added to my calendar |  calendar, calendar update |  I understand you are referring to your calendar. Is it the marathon on friday that you are referring to or do you need an update on your calendar?
why didn’t my card work |  expiration date, card declined |  I understand you are having issues with your card. Is it because of the expiration date or was it declined?
Table 7: Sample of LLM outputs on the C150 dataset.

##  Appendix D Appendix: Intent distributions

We here present the intent distributions for all public datasets across train,
calibration and test splits.

Figure 3: Intent distribution in ACID data set. Figure 4: Intent distribution
in ATIS data set. Figure 5: Intent distribution in B77 data set. Figure 6:
Intent distribution in C150-IS data set. Figure 7: Intent distribution in
HWU64 data set. Figure 8: Intent distribution in IND data set. Figure 9:
Intent distribution in MTOD data set.

##  Appendix E Appendix: Unoptimized $\alpha$

This appendix contains results for an unoptimized $\alpha$ hyperparameter,
arbitrarily set at $.10$ and $.01$. We see that for most data sets, there is
no need to ask a clarification question as the model already achieves the
desired coverage. Much higher coverages (as in Table 2) are achievable for
these data sets. For some more challenging data sets such as C150, HWU64 and
IND, CICC yields small clarification questions while retaining a reasonably
large number of clarification questions of size 1.

Setting | $1-\alpha$ | $th$ |  | Cov$\uparrow$ | Single$\uparrow$ | $|\text{CQ}|\downarrow$ | Amb
---|---|---|---|---|---|---|---
ACID | .90 | 7 | CICC | .90 | .92 | $-$ | 0
|  |  | B1 | .97 | .93 | 5 | 0
|  |  | B2 | .95 | 1 | $-$ | 0
|  |  | B3 | .99 | 0 | 5 | 0
ATIS | .90 | 7 | CICC | .88 | .89 | $-$ | 0
|  |  | B1 | .99 | .93 | 5 | 0
|  |  | B2 | .98 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
B77/BERT | .90 | 7 | CICC | .98 | .79 | 2.90 | .04
|  |  | B1 | .97 | .90 | 5 | 0
|  |  | B2 | .93 | 1 | $-$ | 0
|  |  | B3 | .99 | 0 | 5 | 0
B77/DFCX | .90 | 4 | CICC | .91 | .66 | 2.63 | .02
|  |  | B1 | .95 | .71 | 4.79 | .27
|  |  | B2 | .90 | .98 | 2.26 | 0
|  |  | B3 | .97 | 0 | 5 | 1
C150 | .90 | 7 | CICC | .99 | .97 | 2.66 | 0
|  |  | B1 | .99 | .82 | 5 | 0
|  |  | B2 | .98 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
HWU64 | .90 | 7 | CICC | .90 | .97 | 2.00 | 0
|  |  | B1 | .96 | .79 | 5 | 0
|  |  | B2 | .90 | 1 | $-$ | 0
|  |  | B3 | .98 | 0 | 5 | 0
IND | .90 | 7 | CICC | .91 | .25 | 3.46 | .11
|  |  | B1 | .88 | .42 | 5 | 0
|  |  | B2 | .70 | 1 | $-$ | 0
|  |  | B3 | .91 | 0 | 5 | 0
MTOD | .90 | 7 | CICC | .90 | .90 | $-$ | 0
|  |  | B1 | .99 | .99 | 5 | 0
|  |  | B2 | .99 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
Table 8: Test set results for $1-\alpha=.90$ where underline indicates meeting coverage requirement. Bold denotes best when meeting this requirement, omitted for last column due to missing ground truth for ambiguous. Setting | $1-\alpha$ | $th$ |  | Cov$\uparrow$ | Single$\uparrow$ | $|\text{CQ}|\downarrow$ | Amb
---|---|---|---|---|---|---|---
ACID | .99 | 7 | CICC | 1 | .77 | 3.00 | .10
|  |  | B1 | .98 | .85 | 5 | 0
|  |  | B2 | .95 | 1 | $-$ | 0
|  |  | B3 | .99 | 0 | 5 | 0
ATIS | .99 | 7 | CICC | .99 | .98 | 2.54 | 0
|  |  | B1 | .99 | .73 | 5 | 0
|  |  | B2 | .98 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
B77/BERT | .99 | 7 | CICC | .98 | .79 | 2.90 | .04
|  |  | B1 | .97 | .90 | 5 | 0
|  |  | B2 | .93 | 1 | $-$ | 0
|  |  | B3 | .99 | 0 | 5 | 0
B77/DFCX | .99 | 4 | CICC | .97 | 0 | 5 | 1
|  |  | B1 | .97 | .05 | 5 | .95
|  |  | B2 | .90 | 1 | $-$ | 0
|  |  | B3 | .97 | 0 | 5 | 1
C150 | .99 | 7 | CICC | .99 | .97 | 2.66 | 0
|  |  | B1 | .99 | .82 | 5 | 0
|  |  | B2 | .98 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
HWU64 | .99 | 7 | CICC | .99 | .25 | 3.39 | .28
|  |  | B1 | .98 | .05 | 5 | 0
|  |  | B2 | .90 | 1 | $-$ | 0
|  |  | B3 | .98 | 0 | 5 | 0
MTOD | .99 | 7 | CICC | .99 | 1 | $-$ | 0
|  |  | B1 | 1 | .98 | 5 | 0
|  |  | B2 | .99 | 1 | $-$ | 0
|  |  | B3 | 1 | 0 | 5 | 0
Table 9: Test set results for $1-\alpha=.99$ where underline indicates meeting
coverage requirement. Bold denotes best when meeting this requirement, omitted
for last column due to missing ground truth for ambiguous.

##  Appendix F Appendix: Comparison results OOS detection

We here compare the results of OOS detection as reported by baselines. Note
that these results were generated on different splits of the data and (where
applicable), possibly using different open-domain samples, and that a direct
comparison between results is invalid.