# A Time-Resolved Imaging System for the Diagnosis of X-ray Self-Emission in
High Energy Density Physics Experiments

Jack W. D. Halliday<EMAIL_ADDRESS>Blackett Laboratory,
Imperial College, London SW7 2BW, United Kingdom  Simon N. Bland  Blackett
Laboratory, Imperial College, London SW7 2BW, United Kingdom  Jack. D. Hare
Department of Nuclear Science and Engineering, Massachusetts Institute of
Technology, 77 Massachusetts Avenue, Cambridge, MA 02139  Susan Parker
Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom  Lee G.
Suttle  Blackett Laboratory, Imperial College, London SW7 2BW, United Kingdom
Danny R. Russell  Blackett Laboratory, Imperial College, London SW7 2BW,
United Kingdom  Sergey V. Lebedev  Blackett Laboratory, Imperial College,
London SW7 2BW, United Kingdom

###### Abstract

A diagnostic capable of recording spatially and temporally resolved X-ray self
emission data was developed to characterise experiments on the MAGPIE pulsed-
power generator. The diagnostic used two separate imaging systems: A pinhole
imaging system with two dimensional spatial resolution and a slit imaging
system with one dimensional spatial resolution. The two dimensional imaging
system imaged light onto image plate. The one dimensional imaging system
imaged light onto the same piece of image plate and a linear array of silicon
photodiodes. This design allowed the cross-comparison of different images,
allowing a picture of the spatial and temporal distribution of X-ray self
emission to be established. The design was tested in a series of pulsed-power
driven magnetic-reconnection experiments.

††preprint: AIP/123-QED

##  I Introduction

The X-ray emission from high energy density (HED) plasmas is typically both
transient and exhibits significant spatial variation. Traditionally, X-ray
imaging data is captured either using gated X-ray detectors (for example
micro-channel-plates [1]) or time-integrated detectors (such as image plate
[2]). Time-gated detectors provide unambiguous information about the time
interval in which X-ray emission occurs however they are expensive, bulky,
introduce significant experimental complexity. Time-integrated detectors are
compelling in the sense they are simple and inexpensive, however they provide
no information about the time interval in which X-ray emission occurs and some
dynamical aspects of experiments are also obscured by motion blurring.

The information gained from time-integrated X-ray detectors is often augmented
with (time-resolved) detectors which convert X-ray flux into an electrical
signal. Examples of such detectors include PCDs [3], vacuum X-ray diodes
(XRDs)[4], and silicon photodiodes. These are however typically fielded as
spatially-integrated diagnostics.

In this paper, we describe a diagnostic which bridges the gap between the
time-integrated and time-resolved X-ray diagnostics discussed above. This
novel design made use of two separate X-ray imaging systems: A pinhole which
imaged light onto image plate (time integrating), and a slit which imaged
light onto both image plate and a linear array of silicon diodes (time-
resolved). This meant that two-dimensional time-integrated, and one-
dimensional time-resolved images were captured simultaneously. Features in the
two-dimensional and one-dimensional images could be compared.

An approach similar to the one described here has previously been used to
diagnose gas puff implosions on pulsed-power machines including Saturn and
Double-EAGLE [5, 6] however time integrating X-ray detectors were not fielded
alongside time-resolved ones.

There is an obvious analogy between the diagnostic we describe and an X-ray
streak camera (which could also be set up to provide spatial resolution in one
dimension). That said, much like MCP cameras, X-ray streaks are bulky,
complex, and expensive diagnostics to field. They also have notable issues
regarding both dynamic range and linearity of response. By contrast, the
arrangement of silicon photodiodes we present is cheap, compact, and has a
simpler response characteristic. That said, the diodes provide a coarser level
of spatial resolution than an X-ray streak camera so should be seen as a
complimentary technology which is best suited to harsh diagnostic
environments, or situations in which a compact setup is required.

The design we describe could be applied to the diagnosis of a wide variety of
HED plasmas, although it is best suited to systems which are either extended
in one dimension or have a quasi one dimensional geometry. Examples of
experiments which are extended in one dimension include gas-puff or wire-array
Z-Pinch implosions. Examples of experiments with a quasi one dimensional
geometry include the work on photoionisation fronts which currently form part
of the work done by the Z Astrophysical Plasma Properties (ZAPP) collaboration
[7].

The design we present was used to diagnose X-ray self emission in reconnection
experiments performed on the MAGPIE generator at Imperial College [8]. A
diagnostic based on the design described here is also being developed to
diagnose plasma dynamics in magnetic reconnection experiments performed on the
Z-Machine at Sandia National Laboratories [9].

##  II Optical Design of the instrument

Figure 1:  A cross section of the diagnostic which was use to image X-ray
emission onto a diode array and an image plate. Ray traces for the pinhole and
slit imaging systems are shown in blue and red respectively. The spatial
extent of the blue and red rectangles in the object and image planes are
defined by the projection of the pinhole/slit and the baffles.

Figure 1 shows a CAD model of the X-ray imaging system our diagnostic
employed. The system used two separate apertures to image radiation from the
experimental volume. The first of these was a pinhole which cast light (blue
rays in figure 1) onto a piece of image plate positioned on the same plane as
the diode array. The second was a slit imaging system which cast light (red
rays in figure 1) onto both the image plate and the diode array. The slit
imaging system provided one dimensional spatial resolution whilst the pinhole
provided two dimensional resolution.

Diagrams showing the geometry of the X-ray apertures, and the baffling which
was employed to prevent cross talk between the two imaging systems are shown
in figure 2. Both of these components were laser-cut from
$200\text{\,}\mathrm{\SIUnitSymbolMicro m}$ thick stainless steel, which was
sufficiently opaque to block the relatively soft X-ray emission encountered in
experiments performed on MAGPIE (nominal color temperature for emission is
typically $\sim$100\text{\,}\mathrm{eV}$$). Inspection of laser cut components
under a microscope revealed that features were accurate to
$\sim$20\text{\,}\mathrm{\SIUnitSymbolMicro m}$$.

Figure 2:  _Left:_ A diagram (not to scale) of the plate containing the X-ray
apertures used to image the reconnection plane. _Right:_ A diagram of the
baffle plate, which was designed to prevent cross talk between the two imaging
systems used in this setup. Both plates were laser cut from
$200\text{\,}\mathrm{\SIUnitSymbolMicro m}$ thick stainless steel.

The projection of the ray traces onto the image plane is shown in figure 3.
This diagram also shows the relative positions of the image plate and the
diode array. The diode array was oriented so that it was spatially resolving
in the same direction as the slit. The system was set up so that, for light
cast from the object plane, the X-axis of the pinhole image could be directly
related to the spatially resolved axis of the slit image. This enabled
features seen in the signal from the diode array to be related to features
seen in diagnostic images obtained in separate experiments.

Figure 3:  A diagram showing images formed on the detector plane by the setup
used for experiments with a linear photodiode array. The diagram shows the
setup formed two images, a slit image and a pinhole image. The diode array
consisted of 20 photoconducting elements, the central 10 of which were used in
experiments – with the rest shorted out.

##  III Details of X-ray detection technology

Time-integrated data was captured using Fuji-film image plate (BAS-TR). The
image plates were scanned using a Fuji-Film FLA-5000 scanner.

Figure 4:  A diagram showing the geometry of the diode array. Photoconducting
elements are indicated by diagonal shaded features in the diagram. The array
incorporated a total of 20 photoconducting elements.

The diode-array was an Opto Diode product with the model number AXUV20ELG.
This consists of 20 photoconducting elements arranged linearly, as shown in
figure 4. The size of each element is
$0.75\times$4.05\text{\,}{\mathrm{mm}}^{2}$$ and that the centre-center
separation between elements is $0.95\text{\,}\mathrm{mm}$. This detector is
silicon based and the thickness of the active layer is
$50\text{\,}\mathrm{\SIUnitSymbolMicro m}$. The spatial resolution was
dominated by the geometric factors rather than diffractive effects. The width
of the slit was chosen so the resolution was $\sim$1\text{\,}\mathrm{mm}$$
which is comparable to the separation between the photoconducting elements.

This diode array has a common cathode, and an individual anode for each
photoconducting element. The cathode was biased to $9\text{\,}\mathrm{V}$ and
the current through each element was relayed to a channel on an oscilloscope
(HP Agilent 16500B with five 16532A digitizing oscilloscope cards) via a
coaxial line terminated with a $50\text{\,}\mathrm{\SIUnitSymbolOhm}$
resistor. A passive low pass filter
($RC=$10\text{\,}\mathrm{k\SIUnitSymbolOhm}$\times$220\text{\,}\mathrm{nF}$$)
was used on the (common) cathode in order to reduce the effects of electrical
noise from the pulsed power generator and to maintain the bias voltage on the
timescale of an experiment. No filtration was used between the anode side of
the diodes and the channels of the oscilloscope.

The rise time of the detectors were empirically determined to be
$\sim$1\text{\,}\mathrm{ns}$$ by measuring their $V(t)$ response to radiation
from a $200\text{\,}\mathrm{ps}$ pulsed laser (AXUV diodes have a low-level
response in the visible). The discharge time for the capacitor is around
$$50\text{\,}\mathrm{\SIUnitSymbolOhm}$\times$220\text{\,}\mathrm{nF}$=$11\text{\,}\mathrm{\SIUnitSymbolMicro
s}$$ so the bias voltage was not significantly depleted on the timescale the
an experiment (less than $0.5\text{\,}\mathrm{\SIUnitSymbolMicro s}$).

The hydrodynamic timescale for typical experiments on MAGPIE is
$\sim$10\text{\,}\mathrm{ns}$$, so the temporal response of the detector is
sufficient to resolve dynamics in our experiments. This temporal response is
set (in part) by the bias voltage applied to the diodes and the bias voltage
we used was selected to provide a sufficiently short temporal response without
causing the diodes to breakdown.

We note that an alternative biasing scheme would have been to hold the cathode
at ground, and have the bias voltage relayed to each individual
photoconducting element, similar to the design reported in [3]. This
alternative biasing scheme would reduce the possibility of cross-talk between
the diodes in the array.

As previously stated AXUV20ELG detectors use a
$50\text{\,}\mathrm{\SIUnitSymbolMicro m}$ thick active layer of silicon. The
detector exhibits a good level of quantum efficiency from
$\sim$100\text{\,}\mathrm{eV}$$, up to $\sim$10\text{\,}\mathrm{keV}$$. For
X-rays harder than this upper limit, the penetration of the (relatively thin)
active layer limits the effective quantum efficiency.

##  IV Application pulsed-power driven magnetic reconnection experiments

To validate the utility of the design described above, we fielded a prototype
diagnostic in a series of pulsed-power driven magnetic reconnection
experiments. Results from this magnetic reconnection platform have been
previously reported in a number of publications [10, 11, 12, 13, 14]. The
setup was chosen here as the conditions of the plasmas produced are well
characterised and the experimental morphology is extended in one dimension
(rendering it suitable for diagnosis with the diode array).

Figure 5:  Illustration of a pulsed power driven magnetic reconnection
experiment. The direction of current flow is drawn in purple; the orientation
of magnetic fields are drawn in blue; and the direction of velocities are
drawn in red. Reprinted with permission from J. D. Hare et. al, Physics Review
Letters, 118, 085001 (2017). Copyright 2017 by the American Physical Society.

The experimental setup is shown in figure 5. It consists of two inverse (or
exploding) wire arrays [15] which are driven in parallel by an intense current
pulse. Each array acts as a radially diverging source of plasma carrying an
azimuthal magnetic field. These plasma flows are continually driven for the
duration of the generator’s current pulse. For experiments on MAGPIE
($1.4\text{\,}\mathrm{MA}$, $500\text{\,}\mathrm{ns}$ current pulse), the
typical conditions in the ablated plasma are $n_{e}\sim
10^{17}\;${\mathrm{cm}}^{-3}$$; $T_{e}\sim$15\text{\,}\mathrm{eV}$$;
$B_{y}\sim$5\text{\,}\mathrm{T}$$.

As shown in the figure, the two plasma flows from the two arrays collide in a
central interaction region, forming a reconnection layer. Where the two flows
meet they carry oppositely directed magnetic fields. The magnetic fields
reconnect, and magnetic energy is dissipated heating and accelerating the
plasma flows. Within the central reconnection layer, the temperature and
electron density of the plasma increase by around an order of magnitude. The
spatial scale of the reconnection layer is typically
$$1\text{\,}\mathrm{mm}$\times$20\text{\,}\mathrm{mm}$$.

Figure 6:  Diagram showing the image plane with projected positions of
features in the reconnection plane of the experiment.

Figure 6 shows the regions of the experimental volume which were imaged to the
diode array / image plate for the setup used in diagnostic tests. The figure
shows that the slit was oriented to resolve structure across the reconnection
layer (in the X-direction of the experiment). The magnification was $M=1$, so
the width of the layer on the image plane was expected to be
$\sim$1\text{\,}\mathrm{mm}$$.

Figure 7 shows some example image plate data obtained using this experimental
setup. For diagnostic tests, the pinhole was filtered with
$40\text{\,}\mathrm{\SIUnitSymbolMicro m}$ polypropylene and the slit was
filtered with $6.5\text{\,}\mathrm{\SIUnitSymbolMicro m}$ aluminium. The
polypropylene filter transmits radiation from the both the plasma inflows and
the reconnection layer whereas the aluminium transmits only the hardest
radiation produced in the experiment. This means that the slit image only
shows signal from the hottest plasma in the experiment, which is lies within
the reconnection layer.

Figure 7:  Image plate data from an experiment in which the linear photodiode
array was fielded. The extent of the slit image (which was predicted through
raytracing) is shown as a red horizontal line. The image is plotted in units
of photostimulated luminescence (PSL), which is a measure of energy deposited
in the image plate.

Figure 8 shows the diode signals obtained in the same experiment. The X-axis
of the plot is time (in nanoseconds) after current start; the signal from each
diode is mapped to a position on the vertical axis which corresponds to its
location in the array; and the colour-map indicates diode signal (in volts).
As with the image plate data, the $6\text{\,}\mathrm{\SIUnitSymbolMicro m}$
aluminium filter applied to the slit absorbed the colder emission from the
inflow region so we only expected to see a signal on the diodes which imaged
the reconnection layer.

A comparison with image plate data enabled the data from the diode array to be
related to features in the object plane. As expected, only the diode which
imaged the reconnection layer (positioned at $X=$0\text{\,}\mathrm{mm}$$)
shows a significant signal. This response persists during the interval
250-$500\text{\,}\mathrm{ns}$ after current start. The presence of multiple
peaks in the data may be related to the disruption of the reconnection layer
at late times. The amplitude of the peaks are generally as expected from other
diagnostics, however we currently lack the detailed knowledge of the emission
spectrum from the layer which is required to make this statement more
quantitative.

Figure 8:  A heatmap showing the diode signals obtained in a magnetic
reconnection experiment.

For the purpose of this paper, the key points to take away from this section
is that the emission from the layer is imaged onto the diode array and that
comparing data from the array with images captured using a time integrated
detector allows the signal on a particular diode to be related to regions in
the experimental volume, without relying on precise alignment of the
diagnostic with respect to the experimental load.

##  V Conclusions

We presented a design for a time-resolved X-ray imaging system to characterise
X-ray self emission in HED experiments. The diagnostic incorporates two
independent imaging systems, one with two-dimensional, and one with one-
dimensional spatial resolution. Both imaging systems are used to record time-
integrated X-ray self-emission images onto image plate. The one-dimensional
imaging system was also used in combination with a linear array of 20
photodiodes to record time-resolved data. A comparison between the image-plate
data from the two imaging systems enabled the features in one-dimensional
images to be understood in the context of an experimental morphology which
varied in two dimensions. This diagnostic is applicable to HED experiments
which are either extended in one dimension or have a quasi one dimensional
geometry.

## Acknowledgements

This work was supported by the U.S. Department of Energy (DOE) under Award
Nos. DE-SC0020434, DE-NA0003764, DE-F03-02NA00057, and DE-SC-0001063; and by
the U.S. Defence Threat Reduction Agency (DTRA) under award number
HDTRA1-20-1-0001.

## Data Availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * Bland  _et al._ [2004] S. N. Bland, D. J. Ampleford, S. C. Bott, S. V. Lebedev, J. B. A. Palmer, S. A. Pikuz, and T. A. Shelkovenko, Extreme ultraviolet imaging of wire array z-pinch experiments, Review of Scientific Instruments 75, 10.1063/1.1787926 (2004).
  * Meadowcroft  _et al._ [2008] A. L. Meadowcroft, C. D. Bentley, and E. N. Stott, Evaluation of the sensitivity and fading characteristics of an image plate system for x-ray diagnostics, Review of Scientific Instruments 79, 10.1063/1.3013123 (2008).
  * Spielman  _et al._ [1997] R. B. Spielman, L. E. Ruggles, R. E. Pepping, S. P. Breeze, J. S. McGurn, and K. W. Struve, Fielding and calibration issues for diamond photoconducting detectors, Review of Scientific Instruments 68, 782 (1997).
  * Dewald  _et al._ [2004] E. L. Dewald, K. M. Campbell, R. E. Turner, J. P. Holder, O. L. Landen, S. H. Glenzer, R. L. Kauffman, L. J. Suter, M. Landon, M. Rhodes, and D. Lee, Dante soft x-ray power diagnostic for national ignition facility, Review of Scientific Instruments 75, 3759 (2004).
  * Parasad  _et al._ [2004] R. R. Parasad, A. C. Crisman, S. Gensler, N. Qi, and M. Krishnan, Radiation imaging diagnostic for plasma radiation sources, DTRA Technical Report , DTRA-TR-99-42. (2004).
  * Coleman  _et al._ [2010] P. Coleman, J. Thompson, M. Krishnan, and B. L. Bures, An alternative concept for the structure of an x-ray emitting z-pinch, IEEE Transactions on Plasma Science 38, 626 (2010).
  * LeFevre  _et al._ [2021] H. J. LeFevre, M. Springstead, K. Kelso, R. C. Mancini, G. P. Loisel, P. A. Keiter, R. P. Drake, and C. C. Kuranz, The design of a photoionization front experiment using the z-machine as a driving source and estimated measurements, Physics of Plasmas 28, 093304 (2021).
  * Mitchell  _et al._ [1996] I. H. Mitchell, J. M. Bayley, J. P. Chittenden, J. F. Worley, A. E. Dangor, M. G. Haines, and P. Choi, A high impedance mega ampere generator for fiber z-pinch experiments, Review of Scientific Instruments 67, 10.1063/1.1146884 (1996).
  * Savage  _et al._ [2011] M. E. Savage, K. LeChien, M. Lopez, B. Stoltzfus, W. Stygar, D. Artery, J. Lott, and P. Corcoran, Status of the z pulsed power driver, Proceedings of the 2011 IEEE Pulsed Power Conference , 983 (2011).
  * Suttle  _et al._ [2016] L. G. Suttle, J. D. Hare, S. V. Lebedev, G. F. Swadling, G. C. Burdiak, A. Ciardi, J. P. Chittenden, N. F. Loureiro, N. Niasse, J. Wu, Q. Yang, T. Clayson, A. Frank, T. S. Robinson, R. A. Smith, and N. Stuart, Structure of a magnetic flux annihilation layer formed by the collision of supersonic , magnetized plasma flows, Physical Review Letters 225001, 10.1103/PhysRevLett.116.225001 (2016).
  * Hare  _et al._ [2017a] J. D. Hare, L. Suttle, S. V. Lebedev, N. F. Loureiro, A. Ciardi, G. C. Burdiak, J. P. Chittenden, T. Clayson, C. Garcia, N. Niasse, T. Robinson, R. A. Smith, N. Stuart, G. F. Swadling, J. Ma, J. Wu, and Q. Yang, Anomalous heating and plasmoid formation in a driven magnetic reconnection experiment, Physical Review Letters 118, 10.1103/PhysRevLett.118.085001 (2017a).
  * Hare  _et al._ [2017b] J. D. Hare, S. V. Lebedev, L. G. Suttle, N. F. Loureiro, A. Ciardi, G. C. Burdiak, J. P. Chittenden, G. F. Swadling, J. Ma, and J. Wu, Formation and structure of a current sheet in pulsed-power driven magnetic reconnection experiments, Physics of Plasmas 24, 10.1063/1.4986012 (2017b).
  * Suttle  _et al._ [2018] L. G. Suttle, J. D. Hare, S. V. Lebedev, A. Ciardi, N. F. Loureiro, G. C. Burdiak, J. P. Chittenden, T. Clayson, J. W. D. Halliday, N. Niasse, D. Russell, F. Suzuki-Vidal, E. Tubman, T. Lane, J. Ma, T. Robinson, R. A. Smith, and N. Stuart, Ion heating and magnetic flux pile-up in a magnetic reconnection experiment with super-alfvenic plasma inflows, Physics of Plasmas 25, 10.1063/1.5023664 (2018).
  * Hare  _et al._ [2018] J. D. Hare, L. G. Suttle, S. V. Lebedev, N. F. Loureiro, A. Ciardi, J. P. Chittenden, T. Clayson, S. J. Eardley, C. Garcia, J. W. D. Halliday, T. Robinson, R. A. Smith, N. Stuart, F. Suzuki-Vidal, and E. R. Tubman, An experimental platform for pulsed-power driven magnetic reconnection, Physics of Plasmas 25, 10.1063/1.5016280 (2018).
  * Harvey Thompson  _et al._ [2014] A. J. Harvey Thompson, S. V. Lebedev, S. N. Bland, J. P. Chittenden, G. N. Hall, A. Marocchino, S. C. Bott, J. B. A. Palmer, and C. Ning, Quantitative analysis of plasma ablation using inverse wire array z pinches, Physics of Plasmas 16, 10.1063/1.3077305 (2014).


# Distribution Calibration for Out-of-Domain Detection with Bayesian
Approximation

Yanan Wu1∗, Zhiyuan Zeng1∗, Keqing He2∗, Yutao Mou1 , Pei Wang1, Weiran Xu1
1Beijing University of Posts and Telecommunications, Beijing, China
2Meituan Group, Beijing, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>The first three authors contribute equally. Weiran Xu
is the corresponding author.

###### Abstract

Out-of-Domain (OOD) detection is a key component in a task-oriented dialog
system, which aims to identify whether a query falls outside the predefined
supported intent set. Previous softmax-based detection algorithms are proved
to be overconfident for OOD samples. In this paper, we analyze overconfident
OOD comes from distribution uncertainty due to the mismatch between the
training and test distributions, which makes the model can’t confidently make
predictions thus probably causing abnormal softmax scores. We propose a
Bayesian OOD detection framework to calibrate distribution uncertainty using
Monte-Carlo Dropout. Our method is flexible and easily pluggable into existing
softmax-based baselines and gains 33.33% OOD F1 improvements with increasing
only 0.41% inference time compared to MSP. Further analyses show the
effectiveness of Bayesian learning for OOD detection. 111Our code is available
at https://github.com/pris-nlp/COLING2022_Bayesian-for-OOD/.

##  1 Introduction

Detecting Out-of-Domain (OOD) or unknown intents from user queries is key for
a task-oriented dialog system Gnewuch et al. (2017); Akasaki and Kaji (2017);
Tulshan and Dhage (2018); Shum et al. (2018); Zeng et al. (2021a, b); Wu et
al. (2022). It aims to know when a user query falls outside their range of
predefined supported intents to avoid performing wrong operations. Different
from normal intent classification tasks, lack of labeled OOD examples leads to
poor prior knowledge about these unknown intents, making it challenging to
detect OOD samples.

A rich line of OOD intent detection algorithms has been developed recently,
among which softmax-based methods demonstrated promise Guo et al. (2017);
Liang et al. (2018); Zheng et al. (2020). Softmax-based methods leverage
softmax outputs extracted from an in-domain (IND) intent model and operate
under the assumption that the test OOD samples get a lower likelihood
probability than the ID data. For example, Maximum Softmax Probability (MSP)
Hendrycks and Gimpel (2017) detects a test query as OOD if its max softmax
probability is lower than a fixed threshold. However, all these models make a
strong distributional assumption of the practical OOD probability being
uniform, which has been proven wrong because neural networks can produce over-
confidently high softmax scores even for OOD samples Guo et al. (2017).
Therefore, solving the overconfidence issue is still challenging for OOD
detection.

Figure 1: Multiple predicted probability distributions of an OOD sample under
different random seeds. We train four identical models on the same data but
only use different random seeds. Fig (a) displays each output distribution of
an OOD input and Fig (b) shows the averaged output distribution.

In this paper, we study the overconfidence issue from the perspective of
Bayesian learning Gal and Ghahramani (2016). Essentially, the reason for
overconfidence is that a model cannot confidently make predictions on the
input OOD utterances (unknow-unknow) due to the lack of prior knowledge of OOD
data. In other words, even given the same input OOD intent, the predicted
probability distributions using different random seeds are completely
different, uniform, sharp, or any distribution. Fig 1 show an example. We find
models with different initialization seeds can output diverse distributions
for OOD input, maybe cause overconfidence in several in-domain classes. But
the averaged output is close to a uniform distribution. We also find models
with different seeds are more robust to IND input and obtain consistent
outputs (see Appendix C.2). Therefore, one direct way to solve the
distribution uncertainty is to train multiple models independently and
assemble their outputs for the final result. But this method is not applicable
to practical scenarios for large training cost. In this paper, we propose a
Bayesian OOD detection framework to calibrate distribution uncertainty.
Specifically, we firstly train an in-domain intent classifier using IND data,
then in the test stage, we perform multiple stochastic forward passes with a
certain dropout rate (like 0.7) and average the output normalized logits as a
final probability. Without increasing any new parameters, we calibrate
distribution uncertainty by tending to expectation uniform distribution via
Monte-Carlo Dropout Gal and Ghahramani (2016). Our method can be easily
extended to existing softmax-based OOD detection methods and gain significant
OOD improvements with only increasing little inference time compared to
baselines, even outperform the state-of-the-art distance-based methods like
LOF Lin and Xu (2019) and GDA Xu et al. (2020). Our contributions are two-
fold: (1) We analyze the intrinsic reason of overconfidence issue via
distribution uncertainty and propose a Bayesian OOD detection framework to
calibrate this uncertainty using Monte-Carlo Dropout. (2) We provide
theoretical and empirical analysis to demonstrate the effectiveness of our
Bayesian OOD method.

Figure 2: A distribution over distributions where each dot represents a
softmax prediction distribution for a test OOD sample and all the dots denote
a distribution over distributions. We also display softmax prediction
distributions of several dots on the dot line in Fig 1.2.

##  2 Method

###  2.1 Understanding OOD Detection

Problem Definition We refer to training data $D$ as IND data. We aim to detect
the input utterances $x$ belonging to OOD and correctly classify the
utterances belonging to IND utilizing a well-calibrated classifier trained
only on finite IND data $D$.

The predictive uncertainty of a classification model $P(\upsilon|x,D)$ is
commonly divided into _data uncertainty_ (aleatoric), _distribution
uncertainty_ and _model uncertainty_ (epistemic)Kiureghian and Ditlevsen
(2009); Malinin and Gales (2018):

$P(\upsilon|x,D)=\iint{\underbrace{P(\upsilon|\mu)}_{\text{data}}\underbrace{P(\mu|x,\theta)}_{\text{distribution}}\underbrace{P(\theta|D)}_{\text{model}}d\mu\\!\,d\theta}$
(1)

The model uncertainty is described by the posterior distribution over model
parameters $\theta$, and it can be lowered by increasing the amount of data
and simplifying the model complexity. The data uncertainty is described by the
posterior distribution over classes, where $\upsilon$ is the predicted
distribution of all possible in-domain intent classes for OOD detection. It
arises from the natural complexity of the data, such as class overlap, label
noise and homoscedastic noise. It is a property of the world, and cannot be
changed. The distribution uncertainty is modeled with a distribution over
distribution, where $\mu$ is the categorical distribution over simplex. It
arises due to the mismatch between the training and test distributions. We
give an example in Fig 2 which displays a distribution over distributions on a
simplex (Dirichlet distribution Malinin and Gales (2018)) where each dot
represents a softmax prediction distribution for a test OOD sample and all the
dots denote a distribution over distributions. For an input utterance $x$,
softmax-based detection algorithms like MSP assume that the distribution of
OOD utterances ($v_{ood}$) should be very close to the uniform distribution
(the yellow dots in Fig 2) and the distribution of IND utterances ($v_{ind}$)
should be very close to one-hot distribution (e.g. Fig 2(a)-(d)). However, the
practical OOD samples (green dots) exactly yield a sparse distribution over
the simplex where each OOD sample may get a sharp softmax prediction
distribution (like one-hot distribution) or a flat softmax prediction
distribution (like uniform distribution). Due to the lack of prior knowledge
of OOD data, the model cannot confidently make predictions on the input OOD
utterances (unknow-unknow) which is the essential reason why the predicted
probability distributions of the same OOD sample are completely different
under different random seeds and even get very high max softmax scores (Fig
1). In other words, distribution uncertainty could lead to overconfidence in
the prediction of OOD samples.

Therefore, how to alleviate the distribution uncertainty is the key to solving
the overconfidence problem in OOD detection.

###  2.2 Bayesian Approximation

In order to alleviate the distribution uncertainty, we consider marginalizing
out $\theta$ in Eq 1:

$P(\upsilon|x,D)=\int{P(\upsilon|\mu)P(\mu|x,D)d\mu}\vspace{-0.09cm}$ (2)

This yields expected estimates of data and distributional uncertainty given
model uncertainty. Marginalization is intractable in deep neural networks,
thus we consider using $q(\omega)$ to approximate the intractable posterior
though Monte-Carlo Sampling algorithm Tsymbalov et al. (2020); Gal and
Ghahramani (2016):

$P(\upsilon|x,D)=\int{P(\upsilon|\omega)q(\omega)d\omega}$ (3)

where $\omega=\\{W_{i}\\}_{i=1}^{l}$ is the random variables for a model with
$l$ layers. We define $q(\omega)$ as:

$W_{i}=M_{i}\cdot diag(\,\\![\alpha_{i,j}\\!\,]_{i=1}^{k_{i}})$ (4)
$\alpha_{i,j}\sim Bernoulli(p_{i})$ (5)

Where $p_{i}$ and $M_{i}$ are the variational parameters. The binary variable
$\alpha_{i,j}$ indicates whether unit $j$ of the $i-1$ layer will be passed to
the next layer. Specifically, we sample $N$ sets of independent random vectors
of realisations from the Bernoulli distribution
$\\{\alpha_{1}^{n},...,\alpha_{l}^{n}\\}_{n=1}^{N}$ with
$\\![\alpha_{i,j}\\!]_{i=1}^{k_{i}}$ giving
$\\{W_{1}^{n},...,W_{L}^{n}\\}_{n=1}^{N}$. Then we average the output:

$E_{q(\upsilon|x,D)}\\!(\upsilon\\!)\approx\frac{1}{N}\sum_{n=1}^{N}\hat{\upsilon}{(x,D,W_{1}^{n},...,W_{l}^{n})}$
(6)

According to the Law of Large Numbers Yao and Gao (2016), when N is large
enough, the predicted distribution will converge in expected uniform
distribution. That is, we can calibrate the practical sparse OOD distribution
(green dots) over the simplex into ideal dense OOD distribution (yellow dots)
by Bayesian approximation to mitigate the overconfidence issue, which is
verified in the following empirical experiments.

###  2.3 OOD Detection with Bayesian Learning

Fig 3(a) shows the overall architecture of our proposed OOD detection model.
The part in the dashed box is a well-trained feature extractor based on Bi-
LSTM Hochreiter and Schmidhuber (1997) or BERT Devlin et al. (2019). It is
trained on labeled in-domain data using cross-entropy loss. Fig 3(b) shows the
Bayesian approximation process for distribution calibration. We adopt Monte-
Carlo Dropout and average the output normalized logits from multiple
stochastic forward passes:
$\boldsymbol{\bar{v}}=\frac{1}{N}\sum_{i=1}^{N}\boldsymbol{v_{i}}$. In this
way, we calibrate the softmax distribution to the expected distribution, which
close to a uniform distribution. Then, we apply two softmax-based metrics for
OOD detection, which is $m_{MSP}=\max(\boldsymbol{\bar{v}})$ and
$m_{Entropy}=-\sum_{i=1}^{c}\bar{v}_{i}\log\bar{v}_{i}$. We further apply a
empirical threshold to distinguish IND and OOD data.

Model | CLINC-Full | CLINC-Imbal
---|---|---
OOD | IND | OOD | IND
F1 | Recall | F1 | ACC | F1 | Recall | F1 | ACC
LSTM | LOF Lin and Xu (2019) | 59.28 | 58.32 | 86.08 | 85.87 | 55.37 | 51.03 | 80.51 | 82.79
GDA Xu et al. (2020) | 65.79 | 64.14 | 87.90 | 86.83 | 61.38 | 63.80 | 85.35 | 84.20
MSP Hendrycks and Gimpel (2017) | 50.13 | 45.60 | 87.73 | 87.25 | 44.93 | 41.10 | 84.96 | 84.16
MSP+Bayes.(ours) | 70.05 | 68.38 | 88.91 | 88.57 | 61.70 | 57.50 | 85.92 | 85.65
Entropy Zheng et al. (2020) | 68.05 | 67.96 | 88.97 | 88.68 | 64.45 | 63.80 | 86.07 | 85.71
Entropy+Bayes.(ours) | 72.02 | 71.70 | 89.10 | 88.73 | 68.32 | 67.61 | 86.34 | 86.11
BERT | MSP | 52.79 | 50.50 | 87.81 | 87.46 | 48.76 | 46.70 | 85.87 | 85.65
MSP+Bayes.(ours) | 71.25 | 69.58 | 89.10 | 89.56 | 64.32 | 62.00 | 86.39 | 85.87
Entropy | 68.97 | 68.83 | 89.13 | 88.72 | 65.25 | 64.89 | 86.21 | 85.94
Entropy+Bayes.(ours) | 72.85 | 72.42 | 89.47 | 88.94 | 69.11 | 68.49 | 86.74 | 86.42

Table 1: Performance comparison between our method and baselines on CLINC-Full
and CLINC-Imbal datasets (p <0.01). Bayes. represents our proposed Bayesian
approximation via Monte-Carlo Dropout.

Figure 3: The overall architecture of our method.

##  3 Experiments

###  3.1 Datasets

CLINC | Full | Imbal
---|---|---
Avg utterance length | 9 | 9
Intents | 150 | 150
Training set size | 15100 | 10625
Training samples per class | 100 | 25/50/75/100
Training OOD samples amount | 100 | 100
Development set size | 3100 | 3100
Development samples per class | 20 | 20
Development OOD samples amount | 100 | 100
Testing Set Size | 5500 | 5500
Testing samples per class | 30 | 30
Development OOD samples amount | 1000 | 1000

Table 2: Statistics of the CLINC datasets.

We perform experiments on two public benchmark OOD
datasets222https://github.com/clinc/oos-eval, CLINC-Full and CLINC-Imbal
Larson et al. (2019). We show the detailed statistic of these datasets in
Table 2. They both contain 150 in-domain intents across 10 domains. The only
difference is that, for CLINC-Imbal, there are either 25, 50, 75 or 100
training queries per in-scope intent, rather than 100. Note that all the
datasets we used have a fixed set of labeled OOD data but we don’t use it for
training.

###  3.2 Metrics

We report both OOD metrics: Recall and F1-score(F1) and in-domain metrics:
F1-score(F1) and Accuracy(ACC). Since we aim to improve the performance of
detecting out-of-domain intents from user queries, OOD Recall and F1 are the
main evaluation metrics in this paper.

###  3.3 Baselines

For detection algorithms, we use LOF, GDA, MSP and Entropy, none of them need
OOD supervised training. For the feature extractor, we use LSTM and BERT. We
provide a more comprehensive comparison and implementation details of these
models in the Appendix.

###  3.4 Main Results

Table 1 shows our main results on two benchmarks. Our Bayesian method
significantly outperforms softmax-based baselines including MSP and Entropy,
even distance-based SOTA GDA on OOD metrics. Specifically, on CLINC-Full,
Bayes improves 19.92% and 3.97% OOD F1 compared to MSP and Entropy using LSTM,
which proves MSP suffers from severe overconfidence and our method helps
calibrate OOD distribution. The performance gap between MSP and Entropy is
because Entropy based on softmax output distribution can better capture
distinguished information for OOD than MSP based on a single value of softmax
distribution. We find similar improvements under the BERT setting on CLINIC-
Imbal dataset.

Figure 4: Effect of Bayesian approximation on MSP and Entropy confidence
distributions of IND and OOD.

##  4 Analysis

###  4.1 Effect of Bayesian approximation

Fig 4 shows the MSP and Entropy confidence distributions of IND and OOD test
data using Bayesian to verify the effect of our method. Due to the over-
confidence issue of OOD, we find IND and OOD curves overlap a lot in the
original confidence scores. The overlapping part of Entropy is less, which
confirms its better OOD detection performance. After calibration using Bayes,
the overlap part of both methods is reduced, making it easier to distinguish
between IND and OOD.

###  4.2 Analysis of Distribution Calibration

Table 3 shows the effect of Bayes on OOD Dirichlet Distribution. We calculate
the KL-divergence between the predicted averaged softmax distribution and the
uniform distribution of each test OOD sample and report the mean and median
values on the whole test set. With the increase of sampling, we observe a
larger drop on OOD mean and median KL values than INDs. It proves that Bayes
can gradually calibrate the $v_{ood}$ to a uniform distribution and thus make
the sparse OOD Dirichlet distribution dense but not affect IND. Besides, we
find N = 100 already achieves good performance to reduce inference cost. We
provide an efficiency comparison in Section 4.3 and find 33.33% OOD F1
improvements only increase 0.41% time.

###  4.3 Analysis of Cost-effectiveness

We show the comparison between the time consumption and the corresponding
performance improvement in Table 4 on CLINC-Full which has 15100 training data
and 5500 test data. We find that when the number of samples N is 10, our
method can improve the performance by 33.33% while only increasing the time by
0.41%, which proves that our proposed method is very cost-effective. Besides,
we also find that more sampling times lead to more improvements, demonstrating
that more accurate calibration significantly boosts OOD detection. In terms of
time consumption and performance improvement, N = 100 is the most appropriate
sampling parameter. When the sampling time is 1000, the cost-effectiveness is
not as high as when $N=10$. We consider that methods such as model
distillation and pruning can reduce the time consumption, and we will leave it
to future work. In general, we can choose the appropriate number of samples
according to the computing resources.

$\lg$ $N$ | Statistical Indicators
---|---
| OOD | IND
| mean | median | mean | median
0 | 3.63 | 3.60 | 4.74 | 4.95
1 | 2.59 | 2.47 | 4.36 | 4.58
2 | 2.27 | 2.15 | 4.31 | 4.54
3 | 2.24 | 2.12 | 4.30 | 4.54

Table 3: KL-divergence between predicted distribution and uniform distribution
on CLINC-Full. The smaller value is better for OOD. $N$ is the number of
dropout.

lgN | Time(s) | OOD F1 | Increased
---|---|---|---
Time(%) | OOD F1(%)
0 | 240.00 | 50.13 | - | -
1 | 240.98 | 66.84 |  $\uparrow$ 0.41 |  $\uparrow$ 33.33
2 | 252.41 | 70.05 |  $\uparrow$ 5.17 |  $\uparrow$ 39.74
3 | 388.36 | 70.82 |  $\uparrow$ 61.82 |  $\uparrow$ 41.27

Table 4: Time consumption and corresponding performance improvement of
Bayesian approximation based MSP.

###  4.4 Analysis of Parameters

Table 5 reports the OOD F1 under different dropout probability and sampling
times. Within a range between 0.3 to 0.7, the larger dropout probability leads
to better OOD detection performance. This is because OOD data is more
vulnerable to feature loss and its averaged softmax prediction distribution
tends to be more uniform. Besides, more sampling times lead to improvements,
demonstrating that more accurate calibration significantly boosts OOD
detection. We also find that the performance on p=0.7, N=10 is better than the
performance on p=0.3, N=1000. This prompts us to choose a higher p (e.g. 0.7),
which can effectively reduce the time consumption (1000->10). In addition, OOD
F1 is not sensitive to excessive sampling times.

$\lg$ $N$ | Dropout Probability
---|---
0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8
1 | 57.21 | 60.97 | 61.80 | 64.27 | 66.84 | 64.01
2 | 60.78 | 64.38 | 65.84 | 68.87 | 70.05 | 69.03
3 | 63.32 | 65.35 | 67.51 | 69.33 | 70.82 | 69.69

Table 5: Effect of Bayesian approximation with different parameters on OOD
F1-score.

##  5 Conclusion

In this paper, we conduct an analysis of why previous softmax-based detection
algorithms like MSP or Entropy suffer from the overconfidence issue. We find
OOD samples exactly yield a sparse distribution over the simplex and evenly
distribute over the whole space. Therefore, we propose a simple but strong
Bayesian approximation method to calibrate OOD distribution. Experiments prove
the effectiveness of our method. We hope to provide new guidance for future
OOD detection work.

## Acknowledgements

We thank all anonymous reviewers for their helpful comments and suggestions.
This work was partially supported by MoE-CMCC "Artifical Intelligence" Project
No. MCM20190701, National Key R&D Program of China No. 2019YFF0303300 and
Subject II No. 2019YFF0303302, DOCOMO Beijing Communications Laboratories Co.,
Ltd.

## References

  * Akasaki and Kaji (2017) Satoshi Akasaki and Nobuhiro Kaji. 2017.  Chat detection in an intelligent assistant: Combining task-oriented and non-task-oriented spoken dialogue systems.  _ArXiv_ , abs/1705.00746.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 1050–1059, New York, New York, USA. PMLR.
  * Gnewuch et al. (2017) Ulrich Gnewuch, S. Morana, and A. Maedche. 2017.  Towards designing cooperative and social conversational agents for customer service.  In _International Conference on Information Systems_.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International Conference on Machine Learning_ , pages 1321–1330. PMLR.
  * Hendrycks and Gimpel (2017) Dan Hendrycks and Kevin Gimpel. 2017.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _ArXiv_ , abs/1610.02136.
  * Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber. 1997.  Long short-term memory.  _Neural Computation_ , 9:1735–1780.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kiureghian and Ditlevsen (2009) Armen Der Kiureghian and O. Ditlevsen. 2009.  Aleatory or epistemic? does it matter?  _Structural Safety_ , 31:105–112.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Liang et al. (2018) Shiyu Liang, Yixuan Li, and R. Srikant. 2018.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _arXiv: Learning_.
  * Lin and Xu (2019) Ting-En Lin and Hua Xu. 2019.  Deep unknown intent detection with margin loss.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5491–5496.
  * Malinin and Gales (2018) A. Malinin and M. Gales. 2018.  Predictive uncertainty estimation via prior networks.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , page 7047–7058.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Shum et al. (2018) H. Shum, X. He, and Di Li. 2018.  From eliza to xiaoice: challenges and opportunities with social chatbots.  _Frontiers of Information Technology & Electronic Engineering_, 19:10–26.
  * Tsymbalov et al. (2020) Evgenii Tsymbalov, K. Fedyanin, and Maxim Panov. 2020.  Dropout strikes back: Improved uncertainty estimation via diversity sampled implicit ensembles.  _ArXiv_ , abs/2003.03274.
  * Tulshan and Dhage (2018) Amrita S Tulshan and Sudhir Namdeorao Dhage. 2018.  Survey on virtual assistant: Google assistant, siri, cortana, alexa.  In _International symposium on signal processing and intelligent recognition systems_ , pages 190–201.
  * Wu et al. (2022) Yanan Wu, Keqing He, Yuanmeng Yan, QiXiang Gao, Zhiyuan Zeng, Fujia Zheng, Lulu Zhao, Huixing Jiang, Wei Wu, and Weiran Xu. 2022.  Revisit overconfidence for OOD detection: Reassigned contrastive learning with adaptive class-dependent threshold.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4165–4179, Seattle, United States. Association for Computational Linguistics.
  * Xu et al. (2020) Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. 2020.  A deep generative distance-based classifier for out-of-domain detection with mahalanobis space.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 1452–1460, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Yao and Gao (2016) Kai Yao and Jinwu Gao. 2016.  Law of large numbers for uncertain random variables.  _IEEE Transactions on Fuzzy Systems_ , 24:615–621.
  * Zeng et al. (2021a) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a.  Modeling discriminative representations for out-of-domain detection with supervised contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 870–878, Online. Association for Computational Linguistics.
  * Zeng et al. (2021b) Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Hong Xu, and Weiran Xu. 2021b.  Adversarial self-supervised learning for out-of-domain detection.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5631–5639, Online. Association for Computational Linguistics.
  * Zheng et al. (2020) Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020.  Out-of-domain detection for natural language understanding in dialog systems.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:1198–1209.

##  Appendix A Baseline Details

We compare many types of unsupervised OOD detection models. For detection
algorithms, we use LOF(Local Outlier Factor)Lin and Xu (2019), GDA(Gaussian
Discriminant Analysis)Xu et al. (2020), MSP(Maximum Softmax
Probability)Hendrycks and Gimpel (2017) and Entropy. For feature extractor, we
use LSTM(Long Short Term Memory)Hochreiter and Schmidhuber (1997) and
BERT(Bidirectional Encoder Representations from Transformers)Devlin et al.
(2019).

MSP (Maximum Softmax Probability)Hendrycks and Gimpel (2017) uses maximum
softmax probability as the confidence score and regards an intent as OOD if
the score is below a fixed threshold.

LOF (Local Outlier Factor)Lin and Xu (2019) A detecting unknown intents in the
utterance algorithm with local density. It Assumes that unknown intents’ local
density is significantly lower than its k-nearest neighbor’s.

GDA (Gaussian Discriminant Analysis) Xu et al. (2020) A generative distance-
based classifier for OOD detection with Euclidian space. For avoiding over-
confidence problems, they estimate the class-conditional distribution on
feature spaces of DNNs via Gaussian discriminant analysis. GDA is the state-
of-the-art detection method till now, our proposed method using Bayesian
approximation still significantly outperforms GDA. We also compare our method
on two feature extractors for further study.

LSTM (Long Short Term Memory)Hochreiter and Schmidhuber (1997) A neural
network that was proposed with the motivation of an analysis of Recurrent
Neural Nets, which found that long time lags were inaccessible to existing
architectures because backpropagated error either blows up or decays
exponentially.

BERT (Bidirectional Encoder Representations from Transformers)Devlin et al.
(2019) A neural network that is trained to predict elided words in the text
and then fine-tuned on our data. Note that they both trained only on labeled
in-domain data using cross-entropy loss.

Figure 5: Effect of Bayesian approximation on softmax distribution of OOD
sample.

##  Appendix B Implementation Details

We use the public pre-trained 300 dimensions GloVe embeddings Pennington et
al. (2014)333https://github.com/stanfordnlp/GloVe or bert-base-uncased Devlin
et al. (2019)444https://github.com/google-research/bert model to embed tokens.
We use a two-layer BiLSTM as a feature extractor and set the dimension of
hidden states to 128. We use Adam optimizer Kingma and Ba (2014) to train our
model. We set a learning rate to 1E-03 for GloVe+LSTM and 1E-04 for BERT. In
the training stage, We set the dropout probability to 0.5 and set the training
epoch up to 200 with an early stop. We train only on in-domain labeled data.
We use the best F1 scores on the validation set to calculate the detection
method’s threshold adaptively. For our proposed Bayesian approximation, we set
the dropout probability to 0.7, and the dropout sampling times to 100. Each
result of the experiments is tested 10 times under the same setting and gets
the average value. The training stage of our model lasts about 4 minutes using
GloVe embeddings, and 12 minutes using Bert-base-uncased, both on a single
Tesla T4 GPU(16 GB of memory). The average value of the trainable model
parameters is 3.05M. We will release our code after blind review.

##  Appendix C Visualization of softmax prediction distribution.

###  C.1 Visualization of OOD samples

In Fig 5, we give a 150-dimensional class distribution of an OOD sample to
help understand our calibration process. The upper part of the figure is the
distribution obtained by using the primary feature extractor. The softmax
prediction distribution has obvious over-confidence in a particular IND
category. The lower half of the figure presents the distribution after
calibration by Bayesian approximation, which is flatter and meets the
expectations of the OOD sample. When applying softmax-based detection methods,
the latter will more easily recognized as OOD.

###  C.2 Visualization of IND samples

Figure 6: Multiple predicted probability distributions of an IND sample under
different random seeds.

Figure 7: Multiple predicted probability distributions of an IND sample under
different random seeds.

Corresponding to Fig 1, Fig 6 and Fig 7 show predicted probability
distributions of two IND sample under different random seeds over 150 classes.
We train four identical models on the same data but only use different random
seeds. Fig (a) displays each output distribution of an IND input and Fig (b)
shows the averaged output distribution. Specifically, Fig 6 shows when the
input utterance is ’ _block my american saving bank for now_ ’, the model
obtains the maximum prediction probability on ground truth (_freeze_account_)
under four random seeds and averaged output. Specifically, The maximum
probabilities of prediction are 0.77, 0.96, 0.90 and 0.84 under different
random seeds, and 0.87 under averaged output. In the experiments, we find that
most IND samples present the state of Fig 6, that is, under different random
samples setting, the model is very confident to give the input IND utterances
with high confidence probability in the ground-truth category. We guess that
this is because the model has seen some IND data in the training phase, and is
familiar with IND classification, that is, the distribution uncertainty of IND
is not serious. We also show another example in Fig 7 which the input
utterance is ’ _where is improve the credit score_ ’ and the corresponding
true label is _improve_credit_score_. However, we find that under one random
sampling setting, the model mispredicts into _credit_score_ category with a
probability of 0.61. We argue this is due to the fact that the two are easily
confused with each other. Under this random sampling parameter setting, the
model has not learned the feature ability to accurately distinguish these two
categories. In addition, we also find that although there are wrong
predictions, most of the IND predictions are accurate and have a high
prediction probability, so that the highest prediction probability can still
be obtained on the ground-truth label after averaging the distributions. This
also reveals that our method will not damage the performance of IND
classification, and can even avoid misjudgment among some confusing IND
categories.