# Exploring the Advantages of Dense-Vector to One-Hot Encoding of Intent
Classes in Out-of-Scope Detection Tasks

Claudio Pinhanez  Paulo Cavalin

###### Abstract

This work explores the intrinsic limitations of the popular one-hot encoding
method in classification of intents when detection of out-of-scope (OOS)
inputs is required. Although recent work has shown that there can be
significant improvements in OOS detection when the intent classes are
represented as dense-vectors based on domain specific knowledge, we argue in
this paper that such gains are more likely due to advantages of dense-vector
to one-hot encoding methods in representing the complexity of the OOS space.
We start by showing how dense-vector encodings can create OOS spaces with much
richer topologies than one-hot encoding methods. We then demonstrate
empirically, using four standard intent classification datasets, that
knowledge-free, randomly generated dense-vector encodings of intent classes
can yield massive, over 20% gains over one-hot encodings, and also outperform
the previous, domain knowledge-based, SOTA of one of the datasets. We finish
by describing a novel algorithm to search for good dense-vector encodings and
present initial but promising experimental results of its use.

##  1 Introduction

Dense representations of inputs to machine learning (ML) models, often
referred to as input embeddings, have been one of the key drivers of the
massive improvements of performance of most NLP applications in the last 10
years. However, the use of similar dense-vector representations for the output
classes, or output embeddings (Yu & Aloimonos, 2010; Rohrbach et al., 2011;
Kankuekul et al., 2012; Akata et al., 2016), is quite uncommon, except in
scenarios of zero-shot learning (Romera-Paredes & Torr, 2015) where embeddings
are used to encode non-observed latent classes.

The most used method to represent $c>0$ different intent classes is still to
encode them as one-hot vectors , i.e. $c$-dimensional vectors which are all
filled with 0s (zeros) except in the position corresponding to the class of
the intent label, which is filled with. Therefore classes are represented as
equally-distant points in a $c$-dimensional space and classification is
performed by computing the distance to the closest one-hot vector, often using
the $softmax$ function.

Recent works (Cavalin et al., 2020; Pinhanez et al., 2021) have shown that
dense-vector representations of intent classes based on domain knowledge can
improve intent classification accuracy. In particular, such representations
have shown to produce impressive gains in the detection of the inputs to the
machine learning system which are outside of the scope of the intent classes,
often referred to as out-of-scope (OOS) or out-of-domain (OOD) samples (Lee et
al., 2018b; Vyas et al., 2018; Lee et al., 2018a; Chen et al., 2020). Notice
that a classification task without OOS detection in fact assumes a closed
world scenario, while the full version, which detects both the correct class
for in-scope (IS) samples or whether they are OOS samples, corresponds to the
open world case. Most real applications are in open world contexts, including
almost any practical use of intent classification. Here we consider only the
case where no examples of the OOS class are provided for training, although
some are available for testing.

The main argument of this paper is that such large accuracy gains in OOS
detection are more likely due to advantages of dense-vector to one-hot
encoding of intent classes than to the use of domain knowledge. We start by
showing that, in OOS detection, the complexity of the spaces representable by
one-hot encodings is quite limited when compared to dense-vector ones. This
argument is made based on the number of different topological spaces enabled
by each type of encoding. We show that one-hot encodings of $c$ classes can
create only $c$ types of topologically-distinct spaces, while the number of
different spaces enabled by dense-vector encodings is larger than $c^{2}$.

We follow by presenting the results of some experiments in four intent
classification public datasets, where dense-vectors yielded 20% to 40%
improvements in the accuracy of OOS detection. We did this by randomly
generating dense-vector encodings. Our experiments also achieved better
results than a previous SOTA method based on domain knowledge which encode
intent classes using word graphs (Cavalin et al., 2020). We did not obtain
similar improvements in IS intent classification, where the one-hot softmax-
based technique was as good as any dense-vector encoding.

However, those encodings were found through random trial-and-error, a highly
inefficient process. We close the paper proposing an algorithm which searches
the space of encodings, looking for an optimal combination of a set of dense-
vectors and a classification neural network. In initial experiments, the
algorithm improved by 10% the OOS detection rate of the best random dense-
vector encoding used in the previous experiment, and found a representation as
good as one-hot softmax for intent classification.

Given those results, we propose that intent classification with and without
OOS detection should be regarded as quite different problems with distinct
methods to solve them. This work makes the case that the use of more powerful
representation systems, i.e., dense-vectors, may have formidable performance
impacts in the classification with OOS detection case, and that a possible
explanation for the success is their ability to represent more complex
topological spaces.

##  2 Related Work

This paper focuses on the task known as _out-of-domain sample detection_ (Tan
et al., 2019) or _out-of-distribution sample detection_ (Lee et al., 2018b;
Vyas et al., 2018; Lee et al., 2018a; Chen et al., 2020); a good survey can be
found in (Yang et al., 2021). Many of the existing approaches for OOS
detection rely on adapting the training algorithm by changing the loss
function of a neural network (Lee et al., 2018a); by generating ensembles of
classifiers (Vyas et al., 2018; Shalev et al., 2018); by exposing the model to
dversarial, crafted inlier and outlier examples (Chen et al., 2020; Li et al.,
2021); or by including additional OOS examples either in an unsupervised way
(Yu & Aizawa, 2019; Tan et al., 2019) or by generating them (Vernekar et al.,
2019). Another approach is to transform and improve the softmax outputs
(Hendrycks & Gimpel, 2017; Liang et al., 2018; Techapanurak & Okatani, 2019);
or to generate an additional classifier to measure the confidence of the ML
classifier, following an assumption that processing OOS samples have low-
confidence scores (Ryu et al., 2017, 2018; DeVries & Taylor, 2018; Lee et al.,
2018b).

Instead, our approach focuses on changing the representation of the output
layer to match it better with the characteristics of the space of intent
classes. In many ways, we explore in the output layer one of the most
important advances in machine learning, which is the use of input embeddings
(Turian et al., 2010; Mikolov et al., 2013; Pennington et al., 2014; Dos
Santos & Gatti, 2014). In particular, we look into a new use for output or
class embeddings, which have been explored before in other contexts. In
particular, in  _zero-shot learning_ (Romera-Paredes & Torr, 2015), class
embeddings have been used as a tool that makes it possible building a solution
for the problem. Zero-shot learning is based on the identification and
addition of new classes to a classifier with no reliance in input samples.
With class embeddings, new classes can be added to the system by simply
generating an embedding with the proper configuration of an unseen class, to
encapsulate the knowledge of the new concepts (Palatucci et al., 2009; Socher
et al., 2013; Akata et al., 2015b, 2016). Zero-shot learning is a problem
closely related to OOS detection but they differ on the criterion of success:
in the former, the accuracy of assigning inputs to the previously unknown
classes; in the latter, the accuracy of identifying inputs which do not belong
to any of the known classes. We focus here solely in the OOS detection problem
but some of our arguments may also be valid for zero-shot learning.

Recent research has focused in using class embeddings to enhance a ML
classifier by encapsulating additional high-level knowledge related to the
classes, such as in (Cavalin et al., 2020; Pinhanez et al., 2021). In (Cavalin
et al., 2020) the classes were represented by a keyword extracted from the
class training examples followed by the embedding of the corresponding word
graph. However, word graphs tend to repeat the class examples with a different
structure, thus are far from ideal to produce proper class embeddings. In
(Pinhanez et al., 2021) the hierarchical taxonomy of the classes, as
understood by the system developers, was mined from the documentation of the
system and used to create class embeddings. Although the latter approach seems
promising, such taxonomies might not be available in many cases, what limit
its applicability. Notice that those two approaches excelled particularly in
OOS detection.

This work explores further the use of class embeddings by looking into the
properties of dense-vectors themselves, independent of the presence of
knowledge. Our key baseline for comparison is the traditional one-hot encoding
methods. Some previous works have explored the difference between one-hot and
dense-vectors, such as (Rodríguez et al., 2018), which found higher rates of
convergence for the latter. Output embeddings have also been explored in the
context of multi-class classification problems (Amit et al., 2007; Weinberger
& Chapelle, 2009; Weston et al., 2010; Akata et al., 2015a) and large-scale
recognition (Srivastava & Salakhutdinov, 2013; Deng et al., 2014; Xiao et al.,
2014; Yan et al., 2015; Lin et al., 2015). We are not aware of works focusing
on OOS detection using knowledge-free class embeddings as described in this
paper.

##  3 Class Encoding for OOS Detection

Following the notation of (Cavalin et al., 2020), an _intent classification_
method is a function $D$ which maps a set of sentences (potentially infinite)
$S=\\{s_{1},s_{2},...\\}$ into a finite set of classes
$\Omega=\\{\omega_{1},\omega_{2},...,\omega_{c}\\}$:

$D:S\rightarrow\Omega\hskip 14.22636ptD(s)=\omega_{i}$ (1)

In many practical situations of intent classification, it is also necessary to
determine whether a sentence $s$ does not belong to any of the classes, what
is often referred to as out-of-scope (OOS) detection or out-of-domain (OOD)
detection. This can be represented by expanding $\Omega$ to the set
$\bar{\Omega}=\Omega\cup\\{o\\}$ where $o$ represent the class OOS of samples.
Following, an intent classification with OOS detection method
$\bar{D}:S\rightarrow\bar{\Omega}$ is defined by:

$\bar{D}(s)=\left\\{\begin{array}[]{ll}D(s)=\omega_{i}&\mbox{~{}if~{}in-
scope}\\\ o&\mbox{~{}if~{}out-of-scope}\end{array}\right.$ (2)

An input embedding $\xi:S\rightarrow\mathbb{R}^{n}$ is often used, mapping the
space of sentences $S$ into a vector space $\mathbb{R}^{n}$, and defining a
classification function $\bar{E}:\mathbb{R}^{n}\rightarrow\bar{\Omega}$ such
as $\bar{D}(s)=\bar{E}(\xi(s))$. In typical intent classifiers, $\bar{E}$ is
usually composed of a function $M$ which computes the
$z=(z_{1},z_{2},...,z_{c})$ likelihood of $s$ being in each class
$\omega_{i}$, typically in a finite range such as $[-1,1]$, followed by a
class encoding function $\bar{C}$ which maps the likelihood results into the
classes in $\bar{\Omega}$.

$S\overset{\xi}{\rightarrow}\mathbb{R}^{n}\overset{M}{\rightarrow}[-1,1]^{c}\overset{\bar{C}}{\rightarrow}\bar{\Omega}$
(3)

A common way to implement $\bar{C}$, denoted here as $\bar{C}_{max}$, is to
verify whether any coordinate $z_{i}$ is greater than a threshold
$0\leq\theta<1$, and, if so, to map it into the $\omega_{i}$ associated with
the maximum $z_{i}$ value; otherwise, $\bar{C}_{max}$ maps it into $o$.

$\bar{C}_{max}(z,\theta)=\left\\{\begin{array}[]{ll}\mbox{argmax~{}}z_{i}&\mbox{~{}if~{}}\max
z_{i}>\theta\\\ o&\mbox{~{}otherwise}\end{array}\right.$ (4)

Probably the most common method for class encoding is to use the $softmax$
function where the likelihood components are normalized with the exponential
funcion, denoted here as $\bar{C}_{softmax}$.

$\sigma(z)_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{c}e^{z_{j}}}\mbox{~{}~{}~{}thus~{}~{}~{}}\sum_{i=1}^{c}\sigma(z)_{i}=1$
(5)
$\bar{C}_{softmax}(z,\theta)=\left\\{\begin{array}[]{ll}\mbox{argmax~{}}\sigma(z)_{i}&\mbox{~{}if~{}}\max\sigma(z)_{i}>\theta\\\
o&\mbox{~{}otherwise}\end{array}\right.$ (6)

Notice that $\bar{C}_{max}$ and $\bar{C}_{softmax}$ can be seen as computing
the $min$-based distance $d_{min}$ of $z=(z_{1},z_{2},...z_{c})$ to one-hot
vectors $h_{i}=(0,0,....,1,...,0,0)$ where the $1$ value is in the i-th
position of $h_{i}$. In this paper we compare those methods with approaches
based on Euclidean distance. If instead of $min$ we use the Euclidean distance
$d$ to the one-hot vectors $h_{i}$, we obtain a function which we call
$\bar{C}_{d}$:

$\bar{C}_{d}(z,\theta)=\left\\{\begin{array}[]{ll}\mbox{argmin~{}}d(z_{i},h_{i})&\mbox{if~{}}\min
d(z_{i},h_{i})\leq\theta\\\ o&\mbox{otherwise}\end{array}\right.$ (7)

Since all previous methods consider which intent class is closer, for a
particular distance, to the one-hot vectors, we refer to them as one-hot class
encoding methods. An alternative approach is to consider each class
$\omega_{i}$ as represented by the points closer to a given dense-vector
$r_{i}=(r_{i1},r_{i2},...r_{ic})\in[-1,1]^{c}$. We call this function
$\bar{C}_{r}$, defining a typical dense-vector encoding method.

$\bar{C}_{r}(z,\theta)=\left\\{\begin{array}[]{ll}\mbox{argmin~{}}d(z_{i},r_{i})&\mbox{if~{}}\min
d(z_{i},r_{i})\leq\theta\\\ o&\mbox{otherwise}\end{array}\right.$ (8)

The use of dense-vector encoding opens up the exploration of different
dimensions beyond $c$-dimensional encodings as the output of the likelihood
function $M$. In fact, any dimension $p>0$ can be used, and in this case each
class $\omega_{i}$ is represented by the points closer to
$r_{i}^{p}=(r_{i1}^{p},r_{i2}^{p},...r_{ic}^{p})\in[-1,1]^{p}$, defining the
function $\bar{C}_{r}^{p}$.

$\bar{C}_{r}^{p}(z^{p},\theta)=\left\\{\begin{array}[]{ll}\mbox{argmin~{}}d(z_{i}^{p},r_{i}^{p})&\mbox{if~{}}\min
d(z_{i}^{p},r_{i}^{p})\leq\theta\\\ o&\mbox{otherwise}\end{array}\right.$ (9)

Dense-vector encoding methods can use a variety of ways to generate the
$r_{i}$ points to represent the intent classes. We explore experimentally in
this paper both random methods to generate such points and the knowledge-
informed method based on word graphs described in (Cavalin et al., 2020). But
to better understand the differences between using one-hot and dense-vector
encodings, we first discuss the different representational topological
expressivity of each class encoding method to handle different cases of
component connectiveness of the OOS space.

##  4 Class Encoding Topologies

In this section we investigate the differences of the class encoding methods
by considering first a simplified 2D scenario where the goal is to determine
whether a sentence $s$ belongs to one of two classes, $\omega_{1}$ or
$\omega_{2}$, or is out-of-scope ($o$). The generalization to high-dimensional
spaces uses similar arguments and is detailed in Appendix A.

To simplify the analysis we do not consider here special, limit cases where
the intersection of two components degenerates to a single point or to a
tangent line. Considering this, let us start by examining the simplest
situations where there is no OOS detection. This corresponds to the case where
$\theta=0$ in the previous formulas. Figure 1 shows that, in those cases, all
the different functions $\bar{C}$ described before can only generate the same
topology in which the $\omega_{1}$ and the $\omega_{2}$ components are
connected.

Figure 1: The same topology is generated by $\bar{C}_{max}$,
$\bar{C}_{softmax}$, $\bar{C}_{d}$, and $\bar{C}_{r}$ in 2D when there is no
OOS detection ($\theta=0$).

However, when there is OOS detection, that is, $\theta>0$, a different
topological landscape emerges. The leftmost 2D space of fig. 2 illustrates the
case of the $\bar{C}_{max}$ function where the class $\omega_{1}$ maps into a
trapezium positioned between $\theta\leq z_{1}\leq 1$, bound by the main
diagonal of the first quadrant as shown. The class $\omega_{2}$ similarly maps
into a reflected trapezium and the OOS class $o_{1}$ occupies a square defined
by $-1\leq z_{1}<\theta$ and $-1\leq z_{2}<\theta$. We represent schematically
the topology of the connected components defined by $\bar{C}_{max}$ as the
leftmost bottom diagram of fig. 2. It shows that the three components are
pair-wise connected, that is, share a non-trivial border in this space.

Figure 2: The only $\bar{C}_{max}$ topology and the two topologies of
$\bar{C}_{softmax}$ in 2D.

Although quite similar to $\bar{C}_{max}$, the $\bar{C}_{softmax}$ function
can represent two distinct topologies, as shown in the central and rightmost
parts of fig. 2. This is an effect of the normalization process which maps all
points into a triangle in the first quadrant. The first topology is identical
to the case of $\bar{C}_{max}$, and it happens if $0<\theta\leq 1/2$. However,
if $\theta>1/2$, the square of the OOS component $o_{1}$ divides the two
intent classes into two non-connected triangles, yielding a new topological
configuration where the $\omega_{1}$ and $\omega_{2}$ are not connected as
shown in fig. 2.

Continuing the exploration of one-hot class encodings, fig. 3 shows the
effects of substituting $max$ with the Euclidian distance $d$, which basically
maps $w_{1}$ and $w_{2}$ into circles of $1-\theta$ radius centered on the
one-hot vectors $(1,0)$ and $(0,1)$. If $1-1\sqrt{2}<\theta<1$ we obtain the
case depicted on the center of fig. 3, a topology similar to the second case
of $\bar{C}_{softmax}$ where the two intent classes are disconnected and
connected to a common OOS component $o_{1}$. However, when $0<\theta\leq
1-1/\sqrt{2}$ a new topological configuration is enabled, created by a new
unconnected OOS component $o_{2}$ corresponding to points with high values of
both $z_{1}$ and $z_{2}$, as seen in the leftmost part of fig. 3. Also, as
seen in the rightmost part of fig. 3, it is not possible to have this case a
topological configuration where the two classes are pair-wise connected but
the OOS space is not split into two components. That would require that the
threshold $\theta$ to be greater than $1$ what is not possible. Notice that
although the sofmax and Euclidean distance $d$ in one-hot encoding
representations allow for different topologies of OOS spaces, both afford just
two different topologies which depend on the value of $\theta$.

Figure 3: The topologies of $\bar{C}_{d}$ in 2D. The rightmost topology is not
allowed because $\theta\leq 1$. Figure 4: The twelve topologies of
$\bar{C}_{r}$ in 2D.

Figure 4 shows how the use of dense-vector encoding enables many more and
richer representations of the output space than the one-hot methods. As
depicted, dense-vectors in 2D allow 12 different topological configurations,
and the split of the OSS space into 2, 3, and even 4 disconnected components.
Also, dense-vector encoding allows for situations where one OOS component is
connected to just one of the intent classes, such as in the 8 cases of the two
central columns of fig. 4.

This analysis of the 2D scenario shows how potentially limiting is the use of
one-hot encoding methods, especially in their ability to represent more
complex topological configurations of the OOS space. More important, such
analysis holds for higher dimensions of $c$ as discussed in detail in Appendix
A. For the main argument of this paper, it suffices to say that, for any
number of classes $c\geq 2$, the one-hot encoding function $\bar{C}_{max}$
defines only one topology, and $\bar{C}_{softmax}$ and $\bar{C}_{d}$ define
exactly $c$ different topologies. However, for dense-vector encoding methods
such as $\bar{C}_{r}$, the number of different topologies increases at least
quadratically with $c$, as demonstrated in Appendix A.

##  5 Empirical Evaluations of the Encodings

In the previous section we established that dense-vector encoding methods can
represent much more complex OOS components and output spaces than one-hot
encoding. We now present empirical evidence that dense-vector methods can
outperform significantly one-hot encoding methods in OOS detection tasks and
that in tasks without OOS detection the gains are small or non-existent.

###  5.1 The Algorithms

We used only the _Universal Sentence Embeddings (USE)_ (Cer et al., 2018) as
the main classifier in our experiments. We consider the baseline in our
experiments the traditional classification methods which represent a given
class symbol in the one-hot encoding format. For simplicity, in our
experiments we considered only the _softmax_ function $\bar{C}_{softmax}$ for
the final step of the classification. We refer to this algorithm as 1-hot
softmax.

Next, we used the 1-hot distance method which implements one-hot encodings
using Euclidean distance, corresponding to the function $\bar{C}_{d}$. The
main reason for this is to evaluate the impact of switching from $softmax$ to
Euclidean distance, which, as we saw, creates more topologically complex OOS
spaces though not increasing the topological count.

As for dense-vector encodings, we evaluated both knowledge-based and random
methods. For domain knowledge-based dense-vector encodings, we used the
algorithm described by (Cavalin et al., 2020), called here word graph.
Basically, the graph embedding algorithm _DeepWalk_ (Perozzi et al., 2014) was
used to generate the dense-vector class embeddings based on a graph composed
of nodes to represent the classes linked to keywords extracted from the
examples in the training set. The resulting class encodings were the graph
embeddings associated to the class nodes.

For the random sampling dense-vector encodings, we employed algorithms
identical to the one used in word graph except that they used $N$-dimensional
random dense-vector class embeddings instead of the word graph embeddings.
Classes were represented by $N$-dimensional vectors filled with values which
were randomly sampled from an uniform distribution. We refer to the
corresponding algorithms as R(N). For evaluation purposes, we computed a set
of random samples for each R(N) and considered both the average and minimum
value of the metrics across the set of samples.

###  5.2 Evaluation Metrics

To evaluate the different methods, we used an uniform way to select the key
$\theta$ value. We considered the fairest way to do so is to use the _equal
error rate (EER)_ evaluation metrics employed in (Ryu et al., 2017, 2018; Tan
et al., 2019; Cavalin et al., 2020; Pinhanez et al., 2021). In this evaluation
metrics, the threshold $\theta$ is set based on the value where the curves of
_false acceptance rate (FAR)_ and _false rejection rate (FRR)_ intersect. The
former metric corresponds to number of accepted OOS samples divided by the
total of OOS samples; the latter represents the ratio between the number of
wrongly rejected in-scope (IS) cases and the total of IS samples.
Additionally, we also took into account _in-scope error rate (ISER)_ , which
corresponds to the error considering only IS samples and no rejection, i.e.
$\theta=0$, similar to the class error rate in (Tan et al., 2019).

###  5.3 Experiments and Results

The different algorithms were evaluated in four distinct intent classification
datasets, all of them containing tagged OOS examples. These datasets comprise
the three HINT3 intent recognition problems (Arora et al., 2020), consisting
of challenging small and unbalanced datasets; and the balanced dataset known
as _CLINC150_ (Larson et al., 2019), referred here as Larson, which is
considerably larger both in the number of classes and examples.

Figure 5: Charts of FAR (min) and ISER (min) results on each of the HINT3
datasets. The difference percentage from 1-hot softmax is shown to 1-hot
distance, to word graph, and to the best $R(N)$ value, highlighted with
diagonal stripes.

####  5.3.1 HINT3 Datasets

The three HINT3 intent recognition problems (Arora et al., 2020) consist of
datasets containing actual user queries and real OOS examples. Each dataset is
related to a single and unique domain, and the datasets possess real-world
difficulties such as small and unbalanced training sets and a small number of
intents. For the experiments, we considered only the _full_ version of each of
the three HINT3 datasets:

SOFTMattress: 21 intents, 328 training samples, and 231 IS and 166 OOS test
samples;

Curekart: 28 intents, 600 training samples, and 452 IS and 539 OOS test
samples;

Powerplay11: 59 intents, 471 training samples, and 275 IS and 708 OOS test
samples.

Table 1: Summary of the EER, FAR, and ISER results on the HINT3 datasets
showing the minimum (min) values. For each metric, the best result is in bold
typeface; and, among the dense-vector encodings $R(N)$, in italic. Full
results available in Appendix B.

The algorithms were configured as follows. In all methods, the neural network
architecture had an input size of 512, the dimension of the USE vectors; one
hidden layer with 768 neurons and dropout rate of 0.1; parameters trained with
the Adam optimizer; and minimizing categorical cross-entropy. For the word
graph method, we created the graph by finding the common words of the training
examples as in (Cavalin et al., 2020). We then used DeepWalk with the class
embedding size set to 200 and walk sizes of 20. We used a two-layer neural
network with 800 hidden neurons trained for 50 epochs with the Adam optimizer.

We evaluated 10 different implementations of the $R(N)$ algorithm, with nine
$N$ values ranging from 10 to 150, and one $N$ equal to the number of classes
$c$. For each $R(N)$, we trained 500 different randomly generated class
encodings to better explore the space of class encodings. Those systems were
compared to one training of the 1-hot softmax, 1-hot distance, and word graph
algorithms. We are aware that differences in results can occur since the
neural networks weights are randomly initialized but we found such differences
negligible. Moreover, if we had to perform different training runs for the
other three algorithms we would also need to perform different training runs
for each of the 500 random class encodings.

Table 1 shows a summary of the main results of the experiments on the HINT3
datasets, considering the minimum value of $R(N)$ in each metrics. More
detailed results, including the average and standard deviation for the 500
random samplings, are available from table 3 in Appendix B. The best result
for each metric is marked in bold typeface and the best result for the $R(N)$
algorithms in italic.

Regarding the minimum EER values reached by the $R(N)$ methods, the results in
table 1 show that random sampling is promising towards finding good dense-
vector encodings. For SOFTMattress, Curekart, and Powerplay11, the best
overall EER values were 0.186, 0.344, and 0.292, respectively from $R(25)$,
$R(10)$, and $R(10)$. Those correspond to improvements in EER, compared to
1-hot softmax, of -28%, -27%, and -23%, respectively.

The improvements in EER seemed to be chiefly due to improvements in the in the
FAR metric of OOS detection. Table 1 shows that gains in FAR happened in the
three datasets for every $N$-dimension of $R(N)$, and in $R(30)$, $R(10)$, and
$R(10)$ we obtained gains in FAR of -23%, -32%, and -26% to 1-hot softmax,
respectively. This is highlighted by the top part of fig. 5 which clearly
shows that the FAR metrics of the best $R(N)$ encodings were not only
significantly better than 1-hot softmax, but also beat easily 1-hot distance
and word graph in the three datasets.

The same did not repeat for ISER, as seen in table 1. When considering only
the accuracy of classifying IS input in the intent classes, dense-vector
encodings were able to perform slightly better in Curekart and Powerplay11,
with improvements of -5% and -4% but not in SOFTMattress. In the lower part of
fig. 5 we can clearly see the difficulty for almost all algorithms to be
better than 1-hot softmax.

Table 2: Summary of the EER, FAR, and ISER results on the Larson dataset
showing the minimum (min) values. For each metric, the best result is in bold
typeface; and, among the dense-vector encodings $R(N)$, in italic. Full
results available in Appendix B.

Figure 6: Charts of FAR (min) and ISER (min) results on the Larson dataset.
The difference percentage from 1-hot softmax is shown to 1-hot distance, to
word graph, and to the best $R(N)$ value, highlighted with diagonal stripes.

####  5.3.2 Larson Dataset

We also explored the publicly-available Larson dataset (Larson et al., 2019),
a set specifically designed for the evaluation of the OOS detection. Unlike
the HINT3 datasets, it contains a much larger number of samples with a total
of 18,000 training samples and 5,500 test samples (4,500 IS and 1,000 OOS),
and the number of examples per class is balanced. Moreover, the number of
classes in this dataset is quite larger than in the HINT3 with 150 classes,
and comprises five different domains unlike the single-domain HINT3 datasets.
Also, the error rates reported in Larson are considerably lower, thus it is
more challenging to achieve improvements to 1-hot softmax. Configuration-wise,
we used a slightly modified set of values for $N$, starting in 10 and ending
in 300, due to the larger number of classes.

Table 2 shows a summary of the main results of the experiments on the Larson
dataset, considering the minimum value of $R(N)$ in each metrics. More
detailed results are available from table 4 in Appendix B. In terms of EER,
$R(20)$ was the best encoding, improving 1-hot softmax by -35% and 1-hot
distance and word graph by -13%. The improvement in FAR was even more
impressive, from 0.165 of 1-hot softmax to 0.095 in $R(30)$, a massive gain of
-42%, although it was only slightly better than word graph as seen in fig. 6.
The ISER metrics was 3% worse and for small dimensions it was twice as bad.

##  6 Towards Optimal Class Encodings

The results of our experiments indicate that there exist dense-vector
encodings able to achieve better OOS detection results than one-hot encodings
and in some cases, than knowledge-based class encodings. This section proposes
an algorithm to search for those “optimal” dense-vector encodings
systematically.

###  6.1 The Class Encoding Search (CES) Algorithm

We propose a Generative Adversarial Network (GAN)-inspired algorithm which
improves class encodings after a finite number of iterations. The algorithm,
called Class Encoding Algorithm (CES) and detailed in Algorithm 8, takes as
input a given initial class encoding computed, for instance, by a random
sampling. The goal is to find an optimal encoding for $R(N=p)$, i.e. a
instance of the $\bar{C}_{r}^{p}$ function, which is accurate at both OOS
detection (FAR) and in-scope sample classification (ISER).

Considering that $W_{\bar{C}}$ represents the set of weights used by
$\bar{C}_{r}^{p}$ and that during training there is no OOS example available,
our GAN-inspired algorithm consists of continuously updating $W_{\bar{C}}$ and
$R(p)$ to achieve two distinct but complementary objectives: to create a
function class $\bar{C}_{r}^{p}$ which projects the sentences in $S$ onto
compact spheres, while considering class encodings where each point is the
farthest way from the other ones. That is achieved by computing $W_{\bar{C}}$
by minimizing the mean squared error and by updating $R$ by maximizing the
distance of each point to the others. A similar dual-objective optimization
has shown promising results in (Zeng et al., 2021). For this we use a $\mu(X)$
function which computes the mean 1D vector from a set of 1D vectors with the
same dimension represented by $X$.

Figure 7: FAR and ISER results of the Class Embedding Search algorithm on the
SOFTMattress dataset, during 1,000 iterations, $restart{\\_}weights=true$.
Figure 8: The Class Encoding Search (CES) algorithm.

###  6.2 Initial Experiments with SOFTMattress

We applied the CES Algorithm on the SOFTMattress dataset, considering $R(N=c)$
as the base class encoding approach. Class encodings were updated to 1,000
iterations, and we experimented the $restart{\\_}weights$ argument set to both
$true$ and $false$. In the first case, a new $W_{\bar{C}}$ set of weights for
$\bar{C}_{r}^{p}$ is trained from scratch at the beginning of each iteration.
In the second one, $W_{\bar{C}}$ is only initialized in the first iteration,
and the weights are only updated in the subsequent iterations. In our
experiments, the learning rate $\lambda$ was set to 0.0001 after some
preliminary experiments.

The results for $restart\\_weights=true$ are depicted in fig. 7. For the FAR
metric we observed a considerable impact, with the algorithm finding a class
embedding at iteration $619$ where the FAR value dropped from the previous
best result of 0.215 with $R(30)$ to 0.194, a reduction of about 10%. However,
we did not see a pattern of convergence to a minimum which would be highly
desirable. For the ISER metric, the algorithm found at iteration $394$ an ISER
value of 0.198, as good as the best ISER value obtained by the method using
the 1-hot softmax encoding. Notice that this was a reduction of about 7% over
the best previous $p-$dimensional spatial encoding, $R(20)$. We see both
results as very promising about the ability of finding good class encodings by
the CES algorithm but acknowledge that further development is needed to assure
that the results converge to the minimum values.

##  7 Conclusions

In this work we presented an investigation of dense-vector class embeddings in
the context of intent classification with OOS detection. We first showed that
dense-vector encodings allow much more complex OOS spaces in terms of the
number of different topologies they can represent. We then performed
experiments in four datasets for OOS detection, where we found that we could
find good dense-vector representations by random sampling. Using the FAR
metric of OOS detection, we saw improvements between 20% and 40% over typical
one-hot softmax encodings of outputs and obtaied better results than the
knowledge-based SOTA method, but got no significant gains in-scope error rate
(ISER). We then presented a GAN-inspired search algorithm and showed, in
preliminary experiments, that it could find dense-vector encodings which were
better than the ones we had found by random sampling.

We believe both the theoretical exploration and the experimental results
support the importance of using dense-vector encoding of intent classes in
classification with OOS detection, that is, in open world scenarios of intent
classification. At the same time, these results question the need of
knowledge-based class encoding as argued in (Cavalin et al., 2020; Pinhanez et
al., 2021). Of course, if such knowledge is available, it should be used, and
this work has shown only indications that it can be significantly improved.
But when such knowledge is not immediately present or minable, our proposed
algorithm can search simultaneously the space of encodings and neural networks
to find a good encoding.

More broadly, our results suggest that ML systems built using one-hot encoding
may be improved, perhaps dramatically, in the case of open world scenarios by
switching to dense-vector encodings of the output layer, with or without
domain knowledge. Investigating further this possibility is imperative,
especially in domains other than intent classification, as well as
understanding its underlying mechanisms.

## References

  * Akata et al. (2015a) Akata, Z., Perronnin, F., Harchaoui, Z., and Schmid, C.  Label-embedding for image classification.  _IEEE transactions on pattern analysis and machine intelligence_ , 38(7):1425–1438, 2015a.

  * Akata et al. (2015b) Akata, Z., Reed, S., Walter, D., Lee, H., and Schiele, B.  Evaluation of output embeddings for fine-grained image classification.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015b.
  * Akata et al. (2016) Akata, Z., Perronnin, F., Harchaoui, Z., and Schmid, C.  Label-embedding for image classification.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 38(7):1425–1438, 2016.
  * Amit et al. (2007) Amit, Y., Fink, M., Srebro, N., and Ullman, S.  Uncovering shared structures in multiclass classification.  In _Proceedings of the 24th international conference on Machine learning_ , pp. 17–24, 2007.
  * Arora et al. (2020) Arora, G., Jain, C., Chaturvedi, M., and Modi, K.  HINT3: Raising the bar for intent detection in the wild.  In _Proceedings of the First Workshop on Insights from Negative Results in NLP_ , pp. 100–105, Online, November 2020. Association for Computational Linguistics.
  * Cavalin et al. (2020) Cavalin, P., Ribeiro, V. H. A., Appel, A., and Pinhanez, C.  Improving out-of-scope detection in intent classification by using embeddings of the word graph space of the classes.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 3952–3961, 2020.
  * Cer et al. (2018) Cer, D., Yang, Y., Kong, S.-y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Strope, B., and Kurzweil, R.  Universal sentence encoder for English.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 169–174, Brussels, Belgium, November 2018. Association for Computational Linguistics.
  * Chen et al. (2020) Chen, J., Wu, X., Liang, Y., Jha, S., et al.  Robust out-of-distribution detection in neural networks.  _arXiv preprint arXiv:2003.09711_ , 2020.
  * Deng et al. (2014) Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H., and Adam, H.  Large-scale object classification using label relation graphs.  In _European conference on computer vision_ , pp. 48–64. Springer, 2014.
  * DeVries & Taylor (2018) DeVries, T. and Taylor, G. W.  Learning confidence for out-of-distribution detection in neural networks.  _arXiv preprint arXiv:1802.04865_ , 2018.
  * Dos Santos & Gatti (2014) Dos Santos, C. and Gatti, M.  Deep convolutional neural networks for sentiment analysis of short texts.  In _Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers_ , pp. 69–78, 2014\.
  * Hendrycks & Gimpel (2017) Hendrycks, D. and Gimpel, K.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _Proceedings of International Conference on Learning Representations_ , 2017.
  * Kankuekul et al. (2012) Kankuekul, P., Kawewong, A., Tangruamsub, S., and Hasegawa, O.  Online incremental attribute-based zero-shot learning.  In _2012 IEEE conference on computer vision and pattern recognition_ , pp. 3657–3664. IEEE, 2012.
  * Larson et al. (2019) Larson, S., Mahendran, A., Peper, J. J., Clarke, C., Lee, A., Hill, P., Kummerfeld, J. K., Leach, K., Laurenzano, M. A., Tang, L., and Mars, J.  An evaluation dataset for intent classification and out-of-scope prediction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 1311–1316, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * Lee et al. (2018a) Lee, K., Lee, H., Lee, K., and Shin, J.  Training confidence-calibrated classifiers for detecting out-of-distribution samples.  In _Sixth International Conference on Learning Representations - ICLR 2018_ , 2018a.
  * Lee et al. (2018b) Lee, K., Lee, K., Lee, H., and Shin, J.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 31, pp. 7167–7177. Curran Associates, Inc., 2018b.
  * Li et al. (2021) Li, X., Li, J., Sun, X., Fan, C., Zhang, T., Wu, F., Meng, Y., and Zhang, J.  $k$Folden: $k$-fold ensemble for out-of-distribution detection.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 3102–3115, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * Liang et al. (2018) Liang, S., Li, Y., and Srikant, R.  Enhancing the reliability of out-of-distribution image detection in neural networks.  In _Sixth International Conference on Learning Representations - ICLR 2018_ , 2018.
  * Lin et al. (2015) Lin, K., Yang, H.-F., Hsiao, J.-H., and Chen, C.-S.  Deep learning of binary hash codes for fast image retrieval.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pp. 27–35, 2015.
  * Mikolov et al. (2013) Mikolov, T., Chen, K., Corrado, G., and Dean, J.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ , 2013.
  * Palatucci et al. (2009) Palatucci, M., Pomerleau, D., Hinton, G. E., and Mitchell, T. M.  Zero-shot learning with semantic output codes.  In Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C., and Culotta, A. (eds.), _Advances in Neural Information Processing Systems_ , volume 22. Curran Associates, Inc., 2009.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C. D.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pp. 1532–1543, 2014.
  * Perozzi et al. (2014) Perozzi, B., Al-Rfou, R., and Skiena, S.  Deepwalk: Online learning of social representations.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 701–710, 2014.
  * Pinhanez et al. (2021) Pinhanez, C., Cavalin, P., Alves Ribeiro, V. H., Appel, A., Candello, H., Nogima, J., Pichiliani, M., Guerra, M., de Bayser, M., Malfatti, G., and Ferreira, H.  Using meta-knowledge mined from identifiers to improve intent recognition in conversational systems.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 7014–7027, Online, August 2021. Association for Computational Linguistics.
  * Rodríguez et al. (2018) Rodríguez, P., Bautista, M. A., Gonzalez, J., and Escalera, S.  Beyond one-hot encoding: Lower dimensional target embedding.  _Image and Vision Computing_ , 75:21–31, 2018.
  * Rohrbach et al. (2011) Rohrbach, M., Stark, M., and Schiele, B.  Evaluating knowledge transfer and zero-shot learning in a large-scale setting.  In _CVPR 2011_ , pp. 1641–1648. IEEE, 2011.
  * Romera-Paredes & Torr (2015) Romera-Paredes, B. and Torr, P.  An embarrassingly simple approach to zero-shot learning.  In Bach, F. and Blei, D. (eds.), _Proceedings of the 32nd International Conference on Machine Learning_ , volume 37 of _Proceedings of Machine Learning Research_ , pp. 2152–2161, Lille, France, 07–09 Jul 2015\. PMLR.  URL https://proceedings.mlr.press/v37/romera-paredes15.html.
  * Ryu et al. (2017) Ryu, S., Kim, S., Choi, J., Yu, H., and Lee, G. G.  Neural sentence embedding using only in-domain sentences for out-of-domain sentence detection in dialog systems.  _Pattern Recogn. Lett._ , 88(C):26–32, March 2017.  ISSN 0167-8655.
  * Ryu et al. (2018) Ryu, S., Koo, S., Yu, H., and Lee, G. G.  Out-of-domain detection based on generative adversarial network.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 714–718, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
  * Shalev et al. (2018) Shalev, G., Adi, Y., and Keshet, J.  Out-of-distribution detection using multiple semantic label representations.  In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems_ , volume 31, pp. 7375–7385. Curran Associates, Inc., 2018.
  * Socher et al. (2013) Socher, R., Ganjoo, M., Manning, C. D., and Ng, A.  Zero-shot learning through cross-modal transfer.  In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q. (eds.), _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc., 2013.
  * Srivastava & Salakhutdinov (2013) Srivastava, N. and Salakhutdinov, R. R.  Discriminative transfer learning with tree-based priors.  In Burges, C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. (eds.), _Advances in Neural Information Processing Systems_ , volume 26. Curran Associates, Inc., 2013.
  * Tan et al. (2019) Tan, M., Yu, Y., Wang, H., Wang, D., Potdar, S., Chang, S., and Yu, M.  Out-of-domain detection for low-resource text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3566–3572, Hong Kong, China, November 2019. Association for Computational Linguistics.
  * Techapanurak & Okatani (2019) Techapanurak, E. and Okatani, T.  Hyperparameter-free out-of-distribution detection using softmax of scaled cosine similarity.  _arXiv:1905.10628_ , 2019.
  * Turian et al. (2010) Turian, J., Ratinov, L., and Bengio, Y.  Word representations: a simple and general method for semi-supervised learning.  In _Proceedings of the 48th annual meeting of the association for computational linguistics_ , pp. 384–394, 2010.
  * Vernekar et al. (2019) Vernekar, S., Gaurav, A., Abdelzad, V., Denouden, T., Salay, R., and Czarnecki, K.  Out-of-distribution detection in classifiers via generation.  In _Neural Information Processing Systems (NeurIPS 2019), Safety and Robustness in Decision Making Workshop_. https://sites.google.com/view/neurips19-safe-robust-workshop, https://sites.google.com/view/neurips19-safe-robust-workshop, 12/2019 2019.
  * Vyas et al. (2018) Vyas, A., Jammalamadaka, N., Zhu, X., Das, D., Kaul, B., and Willke, T. L.  Out-of-distribution detection using an ensemble of self supervised leave-out classifiers.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * Weinberger & Chapelle (2009) Weinberger, K. Q. and Chapelle, O.  Large margin taxonomy embedding for document categorization.  In _Advances in Neural Information Processing Systems_ , pp. 1737–1744, 2009.
  * Weston et al. (2010) Weston, J., Bengio, S., and Usunier, N.  Large scale image annotation: learning to rank with joint word-image embeddings.  _Machine learning_ , 81(1):21–35, 2010.
  * Xiao et al. (2014) Xiao, T., Zhang, J., Yang, K., Peng, Y., and Zhang, Z.  Error-driven incremental learning in deep convolutional neural network for large-scale image classification.  In _Proceedings of the 22nd ACM international conference on Multimedia_ , pp. 177–186, 2014.
  * Yan et al. (2015) Yan, Z., Zhang, H., Piramuthu, R., Jagadeesh, V., DeCoste, D., Di, W., and Yu, Y.  Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2740–2748, 2015.
  * Yang et al. (2021) Yang, J., Zhou, K., Li, Y., and Liu, Z.  Generalized out-of-distribution detection: A survey.  _arXiv preprint arXiv:2110.11334_ , 2021.
  * Yu & Aizawa (2019) Yu, Q. and Aizawa, K.  Unsupervised out-of-distribution detection by maximum classifier discrepancy.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * Yu & Aloimonos (2010) Yu, X. and Aloimonos, Y.  Attribute-based transfer learning for object categorization with zero/one training example.  In _European conference on computer vision_ , pp. 127–140. Springer, 2010.
  * Zeng et al. (2021) Zeng, Z., He, K., Yan, Y., Liu, Z., Wu, Y., Xu, H., Jiang, H., and Xu, W.  Modeling discriminative representations for out-of-domain detection with supervised contrastive learning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 870–878, Online, August 2021. Association for Computational Linguistics.

##  Appendix A Class Encoding Topologies in Higher Dimensions

In section 4 we explored the different topologies in 2D afforded by the class
encodings described section 3. In this section we show how those results
extend to greater dimensions, that is, $c>2$. As before, we do not consider
here special, limit cases where the intersection of two components degenerates
to a single point or to a tangent line, and only the cases where OOS detection
is needed, that is, $\theta>0$.

Topologies of $\mathbf{\bar{C}_{max}}$: let us consider the $\bar{C}_{max}$
function, with $c>2$ classes $\omega_{1},\omega_{2},...\omega_{c}$ being
recognized, or the symbol $o$ of a OOS input being produced. It is easy to see
that the OOS class occupies a $c$-dimensinal hypercube and each class
$\omega_{i}$ a $c$-dimensional hyper-trapezium. Moreover, each pair of
components corresponding to the $\omega_{i}$ and $\omega_{j}$ classes are in
contact through the diagonal of the plane defined by the axis of the
coordinates $z_{i}$ and $z_{j}$. Similarly, each triad $\omega_{i}$,
$\omega_{j}$, and $\omega_{k}$ classes share the diagonal of the hyper-cube
defined by the coordinates $z_{i}$, $z_{j}$, and $z_{k}$. And the same is
valid for every subset $\\{\omega_{i_{1}},\omega_{i_{2}},...\omega_{i_{t}}\\}$
of $\Omega$, $2\leq t\leq c$. Therefore, all pair of classes are connected to
each other, and so all triads of classes, and so on, in a $c$-dimensional
complete hypergraph, and thus, like in the 2D case, the $\bar{C}_{max}$
function defines only 1 topology for any number of classes $c\geq 2$.

Topologies of $\mathbf{\bar{C}_{softmax}}$: let us consider the case of the
$\bar{C}_{softmax}$ function. In the 2D case, there were two possible
topologies, defined by $0<\theta\leq 1/2$ and $1/2<\theta\leq 1$. We show here
that, for any $c$, the $\bar{C}_{softmax}$ defines exactly $c$ different
topologies, corresponding to the intervals $0<\theta\leq 1/c$, $1/c<\theta\leq
1/(c-1)$, … , $1/2<\theta\leq 1$. We showed in section 4 that in the case
where $c=2$, there are exactly two topologies, one where all classes are
connected when $0<\theta\leq 1/2$ and one where all classes are disconnected
when $1/2<\theta\leq 1$. Using induction, let us assume that it is true that
the number of topologies in the case of $c>2$ classes is exactly $c$, defined
by the intervals $0<\theta\leq 1/c$, $1/c<\theta\leq 1/(c-1)$, … ,
$1/2<\theta\leq 1$. Let us consider the case of $c+1$. First, let us observe
that the $1/2<\theta\leq 1$ in the $c$ case generates a topology where all $c$
classes are disconnected. It is easy to see that this holds in the
$c+1$-dimensional space, and so it goes to all intervals until $1/c<\theta\leq
1/(c-1)$. While the interval $0<\theta\leq 1/c$ in the $c$ case generated a
topology where all classes are fully connected, that is not true in the case
in $c+1$. It is necessary to split the interval $0<\theta\leq 1/c$ into
$0<\theta\leq 1/(c+1)$ and $1/(c+1)<\theta\leq 1/c$. In the interval
$1/(c+1)<\theta\leq 1/c$ all subsets of $c$ components or less are
hyperconnected, but only when $0<\theta\leq 1/(c+1)$, all $c+1$ classes have
points in common, corresponding to the additional topology. Since each of the
$c+1$ intervals is associated to a different topology, there are $c+1$
topologies, completing the induction. Therefore the $\bar{C}_{softmax}$
function defines $c$ different topologies when the number of classes is $c$.

Topologies of $\mathbf{\bar{C}_{d}}$: The case of the one-hot softmax is
analogous to the one-hot encoding using the Euclidian distance. While in the
former the intervals are defined by $1/c$, in the latter they are defined by
$1/\sqrt{c}$, or $\sqrt{c}/c$. The same reasoning used in the case of
$\bar{C}_{softmax}$ can show that $\bar{C}_{d}$ defines exactly $c$ different
topologies, corresponding to the intervals $0<\theta\leq 1/\sqrt{c}$,
$1/\sqrt{c}<\theta\leq 1/\sqrt{c-1}$, … , $1/\sqrt{2}<\theta\leq 1$. Therefore
the $\bar{C}_{d}$ function defines $c$ different topologies when the number of
classes is $c$.

Topologies of $\mathbf{\bar{C}_{r}}$: In section 4 we showed that the
$\bar{C}_{r}$ affords 12 different topologies when $c=2$. For the sake of the
argument of this paper, that is, that dense-vector class encodings enable
richer representations of the space of classes, it is not necessary to provide
an exact estimation of the topologies defined by $\bar{C}_{r}$ in the case of
$c$ classes. We here simply shows that the number of different topologies in
this case grows quadratically with the number of classes by demonstrating that
the number of different topologies $M_{c}$ is equal or greater than the sum
$\sum_{m=1}^{c}m=m(m+1)/2$. We have shown that this is true in the case of
$c=2$, which defines 12 topologies, clearly greater than
$\sum_{m=1}^{2}m=1+2=3$. Using again induction, let us assume that if the
inequality holds for $c$, that is, if $\bar{C}_{r}$ defines at least
$M_{c}\geq\sum_{m=1}^{c}m$ different topologies when there are $c$ classes,
then it is also true for $c+1$. To do so, first observe that all $M$
topologies can be extended to the $c+1$ dimension by considering a value of
$-1$ for the $c+1$ coordinate of each $r_{i}$ dense vector, $1\leq i\leq c$.
If we define the $r_{c+1}$ to be the one-hot vector of $c+1$ dimension, it is
easy to see that the $c+1$ component is disjoint with all other components,
since $\theta<1$. Therefore there are at least $M_{c}$ different topologies
defined by the $c+1$-dimensional $\bar{C}_{r}$, all of them with the
$\omega_{c+1}$ component disjoint from the other components. Therefore, we
just need to show that there are $c+1$ other topologies to complete this
demonstration. We first show that it is possible to define $c$ different
vectors $r_{c+1}$, denoted here as $r_{c+1}^{i}$, in such a way that its
associated component is connected to only one of the other vectors. In fact,
given $r_{i}=(r_{i1},r_{i2},...r_{ic})$, $r_{ij}\in[-1,1]^{c}$, we can extend
all $r_{i}$ to the $c+1$ dimension by setting the $c+1$ coordinate to
$-\theta_{i}$, $\theta_{i}<1$, and for each $r_{i}$, consider a vector
$r_{c+1}^{i}$ such as $r_{c+1}^{i}=(r_{i1},r_{i2},...r_{ic},\theta_{i})$. It
is easy to see that each of the $c$ spaces combining the original
$r_{1},r_{2},...r_{c}$ dense-vectors to each of the $r_{c+1}^{i}$ dense-vector
has at least one new topology where the component $\omega_{c+1}$ is connected
to one component $\omega_{i},i\leq c$. We can always finding a $\theta_{i}$
which is large enough so the component $\omega_{c+1}$ is connected only to the
component $\omega_{i}$, there warranting that all the $c$ encodings
constructed this way have different topologies among themselves and also
different from any the previous $M_{c}$ topologies (where the component of
$\omega_{c+1}$ was always not connected to any other component). We need only
to demonstrate that we can create one more topology, different from all of the
previous ones, to finish. To do so, let us extend all $r_{i}$ to the $c+1$
dimension by setting the $c+1$ coordinate to $0$. If we define $r_{c+1}$ at
the center of the space, that is, with all coordinates as $0$, it is easy to
notice that there is a $0<\theta<1$ which the hyper-dimension “ball” of radius
$\theta$ centered in $r_{c+1}$ intersects with at least two other “balls” of
radius $\theta$ defined by $r_{i}$ and $r_{j}$, $1\leq i,j\leq c$. Therefore
the set of dense vectors defined this way is guaranteed to define a topology
where the component $\omega_{c+1}$ is connected to at least two other
components, and thus is different from the ones we constructed before. Thus,
$M_{c+1}\geq M_{c}+c+1$, but since $\sum_{i=1}^{c+1}=\sum_{i=1}^{c}+(c+1)$, we
obtain $M_{c+1}\geq\sum_{i=1}^{c+1}$. As observed before,
$\sum_{i=1}^{c+1}>(c+1)^{2}$, yielding $M_{c+1}\geq(c+1)^{2}$, and completing
the demonstration. Therefore the $\bar{C}_{r}$ function defines more than
$c^{2}$ different topologies when the number of classes is $c$.

This completes the demonstration that, for any number of classes $c\geq 2$,
the one-hot encoding function $\bar{C}_{max}$ defines only one topology; the
$\bar{C}_{softmax}$ and $\bar{C}_{d}$ define exactly $c$ different topologies;
and for dense-vector encoding methods such as $\bar{C}_{r}$, the number of
different topologies increases at least quadratically with $c$, as stated in
section 4.

##  Appendix B Detailed Experiment Results

Table 3: Full experimental results of the EER, FAR, and ISER metrics on the
HINT3 datasets showing the average (avg), its standard deviation (std),
andminimum (min) values. For each metric, the best result is in bold typeface;
and, among the dense-vector encodings $R(N)$, in italic typeface. These resuls
are summarized in table 1

This section presents more detailed results of the experiment described in
section 5.3.

Table 3 expands table 1 showing all the results associated to the experiments
with the HINT3 datasets, including the average results for each metric and its
associated standard deviation. We can see that, most often, the dense-vector
encoding with minimum value of a metric produced a considerably smaller value
for that metric. In other words, finding a dense-vector encoding which
minimizes the metrics is not a trivial task.

Table 4: Full experimental results of the EER, FAR, and ISER metrics on the
Larson dataset showing the average (avg), its standard deviation (std),
andminimum (min) values. For each metric, the best result is in bold typeface;
and, among the dense-vector encodings $R(N)$, in italic typeface. These resuls
are summarized in table 2.

Table 4 expands table 2 showing all the results associated to the experiments
with the Larson datasets, including the average results for each metric and
its associated standard deviation. Similar to the HINT3 datasets, most often,
the dense-vector encoding with minimum value of a metric produced a
considerably smaller value for that metric. In other words, finding a dense-
vector encoding which minimizes the metrics is not a trivial task.


Primary Author Email<EMAIL_ADDRESS>
# Characterization of a Longwave HgCdTe GeoSnap Detector

Rory Bowens  The University of Michigan, 500 S State Street, Ann Arbor, USA
Michael R. Meyer  The University of Michigan, 500 S State Street, Ann Arbor,
USA  Taylor L. Tobin  The University of Michigan, 500 S State Street, Ann
Arbor, USA  Eric Viges  The University of Michigan, 500 S State Street, Ann
Arbor, USA  Dennis Hart  The University of Michigan, 500 S State Street, Ann
Arbor, USA  John Monnier  The University of Michigan, 500 S State Street, Ann
Arbor, USA  Jarron Leisenring  The University of Arizona, 1200 E University
Boulevard, Tucson, USA  Derek Ives  European Southern Observatory, Karl-
Schwarzschild-Straße 2, Garching bei München, Germany  Roy van Boekel  Max
Planck Institute for Astronomy, Königstuhl 17, Heidelberg, Germany

###### Abstract

New longwave HgCdTe detectors are critical to upcoming plans for ground-based
infrared astronomy. These detectors, with fast-readouts and deep well-depths,
will be key components of extremely large telescope instruments and therefore
must be well understood prior to deployment. We analyze one such HgCdTe
detector, a Teledyne Imaging Sensors GeoSnap, at the University of Michigan.
We find that the properties of the GeoSnap are consistent with expectations
from analysis of past devices. The GeoSnap has a well-depth of 2.75 million
electrons per pixel, a read noise of 360 e-/pix, and a dark current of 330,000
e-/s/pix at 45 K. The device experiences 1/f noise which can be mitigated
relative to half-well shot noise with modest frequency image differencing. The
GeoSnap’s quantum efficiency is calculated to be 79.7 $\pm$ 8.3 % at 10.6
microns. Although the GeoSnap’s bad pixel fraction, on the order of 3%, is
consistent with other GeoSnap devices, close to a third of the bad pixels in
this detector are clustered in a series of 31 ”leopard” spots spread across
the detector plane. We report these properties and identify additional
analyses that will be performed on future GeoSnap detectors.

###### keywords:

Instrumentation, Infrared, ELT

##  1 INTRODUCTION

Ground-based astronomy will be revolutionized by extremely large telescopes,
such as the European Southern Observatory’s Extremely Large Telescope
(henceforth ELT). These facilities offer unprecedented spatial resolution and
light gathering power that will be utilized by a powerful suite of
instruments. The Mid-infrared ELT Imager and Spectrograph (METIS) will be the
ELT’s window into the 3-13 micron regime [1]. METIS will be able to probe a
range of science topics including protoplanetary disks, forming planets,
atmospheres of mature planets, young stellar clusters, evolved stars, the
galactic center, and active galactic nuclei. This includes the potential for a
first direct image of a small, temperate exoplanet around one of the nearest
stars [2].

METIS will utilize a GeoSnap detector with a B0 readout integrated circuit
(ROIC), developed by Teledyne Imaging Sensors (TIS). GeoSnap is a 2048x2048
HgCdTe (MCT) detector with wavelength range from $<$ 3 to $>$ 13 microns, high
quantum efficiency (QE), and favorable noise properties compared to existing
longwave Si:As blocked-impurity-band detectors[3]. The ROIC utilizes a
capacitance transimpedance amplifier (CTIA). CTIAs are useful for mid-IR
applications as they can achieve high well depths (order of 2 million
electrons) and should not suffer from persistence[4] which is present in the
source follower method employed in other state-of-the-art detectors such as
the HAWAII-2RG[5]. GeoSnap can operate at 86.7 Hz in standard operational mode
(and up to 143.2 Hz with certain settings) and has an 18 micron pixel pitch.
At the University of Michigan (UM), we have begun testing potential candidate
GeoSnap detectors for use in METIS, starting with the GeoSnap SN23260. SN23260
represents the first of at least two candidate detectors that we are testing
for the METIS/ESO team. We focused on testing core properties of SN23260
including its cosmetics, gain, and dark current. We compared SN23260 with an
A0 ROIC GeoSnap detector (henceforth UM GeoSnap) previously characterized by
the UM and the University of Arizona (UA)[6]. The UM GeoSnap has only one of
its four quadrants hybridized with MCT material. The MCT material originated
from long wavelength development programs at the University of Rochester [7].
We anticipate a different dark current owing to a different long wavelength
cutoff (see the empirical Rule 07 [8]) and a different read noise owing to
different well depths between the two instruments.

We begin the paper with a description of the testing conditions at UM (Section
2). We then describe the results for SN23260 (Section 3), covering the
cosmetics, gain, well-depth, read noise, bias, dark current, linearity, 1/f
noise, and quantum efficiency. Finally, we summarize our results and suggest
future work (Section 4).

##  2 Testing Conditions

###  2.1 MITTEN Cryostat

We tested SN23260 in the Michigan Infrared Test Thermal ELT N-band (MITTEN)
Cryostat at UM. Details of MITTEN were reported in Bowens et al. 2020[9] but
we briefly summarize its major features and new additions here. MITTEN’s
interior volume is a cylinder that is 36.3 cm tall and 67 cm in diameter. The
optical bench and surrounding shield reach temperatures $<$ 20 K while other
interior surfaces reach temperatures $<$ 60 K. The system has two thermal
shrouds and no windows and is cooled via a pulse-tube cryo-cooler.

A photograph of the interior layout can be seen in Figure 1. The testing
environment uses 1:1 reimaging via an $f/11$ Offner relay to fit the detector
and thermal source within the compact interior space. The detector mount is a
molybdenum block whose temperature is controlled by two heaters, encased in
6061 aluminum on the backside of the mount. They can maintain the detector
mount to a precision of $\pm$ 0.001 K within the range of 30-70 K. However,
the detector introduces approximately 0.8 W of heating (which can vary
slightly depending on selected frame rate). In an environment where the user
is changing frame rate on the order of minutes (such as during our testing),
it is more accurate to say the mount can be maintained to a precision of $\pm$
0.03 K.

Figure 1: Photographs of the MITTEN Optical Bench. The layout of the optical
bench is similar to the description in Bowens et al. 2020 [9] with two
important changes. First, the baffling on the Offner relay is shorter in
length but larger in diameter, in an effort to illuminate the now four times
larger detector surface. Second, the heaters behind the cryostat mount are now
encased in an aluminum enclosure to reduce thermal contamination.

Adjacent to the detector is the light source. The source is a thermally
isolated copper block with a centrally located heating element (and offset
temperature diode) on the backside. The front-side is coated with ultra-black
nano particles to improve the emissivity. During testing, the source was held
within $\pm$ 0.03 K of the set-point temperature. The entire copper source
block is enclosed in a cold aluminum case. Light exits via a 2.03 cm diameter
exit hole.

The exit hole is flush to a slide that can be used to restrict the field of
view to a series of small pinholes. Then the light travels through a six
position filter wheel which includes one blank position and one open position.
These structures are all enclosed except their co-aligned 2.03 cm diameter
entrances and exits to reduce stray light. The filter wheel represents the
greatest uncertainty in alignment but should maintain a center within 1 mm of
source’s center. Finally, the light exits the filter wheel enclosure, travels
through the first set of baffling, travels through the Offner relay, and then
the second set of baffling to a focus on the detector. The co-aligned equal
diameter exits at the source result in vignetting but a mask at the secondary
mirror is used to define the system as f/11. The source was originally
designed for testing the UM GeoSnap used for MIRAC-5 [6, 10] which only uses
one of the four quadrants. As such, it is undersized for the testing of the
full 2048x2048 GeoSnaps and can illuminate approximately 25% of the array. It
can be moved laterally across the detector surface but not vertically. Due to
a gradient across the source (see Section 3.2), we choose to leave the source
at a fixed position near the center of the detector.

There are three operational heaters within the cryostat. One attached to the
source and two attached to the detector mount. Several diodes are located
throughout the cryostat to monitor temperature. Past testing has revealed that
even when the source is heated to 300 K, the exterior of the source box does
not exceed 40 K. A test filter and the filter wheel were left exposed to a 300
K source and did not exceed 62 K after 6 hours.

Baffling has been increased from 2.9 cm diameter to 6.3 cm diameter to
eliminate vignetting of the f/11 beam from the source at its location. The
lengths of the baffles had to be decreased due to their larger radii in order
to avoid collisions with other interior structures. This results in an
approximately two inch gap from the filter wheel enclosure to the first baffle
and a one inch gap from the second baffle to the detector surface. A small
strip of baffling was inserted on the side of the detector to block potential
scattered light from the source.

The filter wheel for MITTEN has also been updated. Descriptions of the new
filters are given in Table 1. One of the six filter positions provides an
opaque ”blank” position. Another position provides a 2.03 cm diameter ”open”
position. Only the ”ammonia” (named for its scientific goal of measuring the
10.6 micron ammonia feature) and 11 micron filters with their narrow
bandpasses and blocking across the whole wavelength range over which the
detector is sensitive, are used for QE estimates. Both of these filters
display obscuration in a small edge region due to oversized ”washers” used to
hold them in place. All filters have measured transmission curves from vendors
at 77 K111These filters are expected to have temperatures of 30 to 40 K in the
cryostat. Conversations with Materion while acquiring the ammonia filter
suggested shifts of about 0.05 microns due to temperature change. which are
used throughout the paper. All transmission curves are from at least 2 to 14
microns, even if the blocking covers a smaller range.

Table 1: The MITTEN filter wheel has six positions 60 degrees in separation from each other. Each position is 2.03 cm in diameter. Transmission curves are known from at least 2 to 14 microns for all filters. Filter | Central $\lambda$ | Full Width Half Max | Peak Trans. | Blocking Trans. | Blocking Range
---|---|---|---|---|---
Name | (microns) | (microns) | (%) | (%) | (microns)
Blank | - | - | - | - | -
Open | - | - | - | - | -
5 Micron | 5.03 $\pm$ 0.7 | 2.0 $\pm$ 0.7 |  $\geq$ 80% |  $\leq$ 1% | 0.4 to 5.5
8 Micron | 8.87 $\pm$ 0.14 | 2.23 $\pm$ 0.14 |  $\geq$ 80% |  $\leq$ 1% | 1.5 to 12.0
Ammonia | 10.59 $\pm$ 0.05 | 0.64 $\pm$ 0.07 |  $\geq$ 80% |  $\leq$ 0.1% | 1.0 to 13.0
11 Micron | 11.300 $\pm$ 0.057 | 0.226 $\pm$ 0.045 |  $\geq$ 70% |  $\leq$ 0.30% | 0.300 to $\geq$ 15.000

###  2.2 Tuning

We tuned SN23260 at 83 Hz for the MITTEN cryostat following instructions from
TIS. SN23260 was held to 45 K $\pm$ 0.03 K. The analog-to-digital converter
(ADC) gain and offset were defined such that in a bias image and in a
saturated image, a Gaussian fit to the pixel value histogram was 3-sigma from
the 0 ADU value and 16,384 ADU value, respectively. This is necessary to
ensure that the full dynamic range of the device is utilized without allowing
a significant number of pixels to under or overflow their registers. We
display a histogram of our ”bias” (spatial histogram of the mean of ten 83 Hz,
100% Tint images with the blank filter in front of the 225 K source) and
saturated (spatial histogram of the mean of ten 83 Hz, 100% Tint images of the
225 K source) in Figure 2 (see Section 3 for an explanation of Tint). These
data utilize a conservative 10 Hz bad pixel mask described in 3.1 and region A
(to coincide with a smooth region of the undersized source) described in 3.2.

Figure 2: Histograms and fits for a 200 x 200 pixel subregion of a 83 Hz dark
(approximately the bias at this frame rate) and 83 Hz saturated image. The
dark data is well fit by a Gaussian with mean 1142.0 ADU/pix and sigma 246.5
ADU/pix, over 3-sigma from the 0 threshold. Slight margin is allotted for the
difference between the true bias and the 83 Hz dark. Likewise, the saturated
data is well fit by a Gaussian with mean of 15270 ADU/pix and sigma of 319
ADU/pix; over 3-sigma from the 16384 threshold. Both thresholds are marked by
dashed, vertical lines.

##  3 Results

We summarize the series of tests performed on SN23260. Unless otherwise noted,
SN23260 is kept at 45 K $\pm$ 0.03 K. Exposures are taken in sets of 128
frames which are then combined via a temporal mean.

GeoSnap can be read with variable integration time, called Tint. GeoSnap
”Tint” is different from the typical use of the word Tint: for GeoSnap it is a
percentage of the frame time (which is defined by the frame rate). Thus, a 10
Hz frame with 50% Tint is only actually exposed for $\sim$0.05 seconds,
followed by $\sim$0.05 seconds of dead time with pixels shorted before the
next frame begins. We utilize Tint ramps throughout the analysis of SN23260,
taking ramps from 0% to 100% Tint in 5% Tint steps. At each step 128 frames
are acquired. There are some intricacies to GeoSnap’s exposure time with
regards to Tint that deserve explanation. The minimum exposure time for
GeoSnap is defined by the master clock frequency, typically set to 80 MHz.
Tint 0% actually results in an exposure time of 320 $\mu$s, regardless of user
defined imaging frame rate. Likewise, the maximum exposure time is actually
the frame time subtracted by the minimum reset time of 140.5 $\mu$s. Thus,
Tint 100% of a 50 Hz frame has a max exposure time of 19.8595 ms, not 20 ms.
The actual exposure time at all Tints are slightly different from the simple
estimate of Tint multiplied by frame time. At 10 Hz, these differences are on
the order of 0.14 ms and are fixed for different Tint/frame rate combos. The
process is further complicated by GeoSnap swapping between Integrate then Read
(ITR) and Integrate while Read (IWR) modes depending on the frame rate and
Tint222METIS will utilize its own electronics to operate its GeoSnap and
therefore these trends are exclusive to our testing environment using the TIS
GeoSnap electronics.. As the device needs enough time to read out the data, it
will perform IWR at higher Tint levels where it no longer has enough time to
wait in between planned frames. The swap also results in bias offsets. The
transition between the modes occurs at
$t_{trans}=(t_{max\\_exposure\\_time}-11.53\ ms)/t_{max\\_exposure\\_time}$
where 11.53 ms is the minimum readout time (based on the 86.7 Hz max frame
rate) of the detector in 2048x2048 mode. So for a 50 Hz frame rate, the change
would be visible at about 43% Tint.

###  3.1 Fraction of Operable Pixels

Before analyzing the data, we produce a bad pixel mask that will be used in
all future data analysis. SN23260’s cold mask occults pixels close to the
detector’s edges. Furthermore, Fresnel diffraction occurs directly adjacent to
this. In all of our analyses, we exclude the 100 pixels directly adjacent to
the edges, effectively limiting our device to a 1848x1848 device. These pixels
are not inherently bad but the physical layout of the device prevents their
use. We refer to this as the ”edge” mask. Furthermore, a horizontal strip due
to metadata processing during readout (which will be fixed in future imaging
software updates) appears at y = 1024 on the array. These pixels make up 0.05%
of the array and are also excluded from the analysis.

After excluding the edge mask and horizontal metadata strip, we begin making a
bad pixel mask, starting with a mean dark frame made from the 128 $\times$
100% Tint 10 Hz dark images (i.e., the source cold and the filter wheel in the
blank position). This dark is the temporal mean of these 128 dark images,
restricted to the inner 1848x1848 pixel region, and based on it we apply two
criteria to exclude pixels from analysis. First, we remove pixels with dark
currents greater than 20 million e-/s/pix. For 10 Hz data and assuming the
gain and bias are approximately 200 e-/ADU and 1000 ADU, respectively, we
consider all pixels with values over 11,000 ADU to exceed this dark current
limit (hereafter ”hot” pixels). We also mask all pixels whose median absolute
deviation (MAD) across the 128 frame equals 0. We call these pixels ”zero-
variance” and they may result from being dead or saturated due to dark
current. We find 1.01% of pixels in the 1848x1848 region are hot and 2.22% are
zero-variance. The criteria are not mutually exclusive so we find that the
total number of bad pixels is 3.17%. The mask for the 10 Hz data is shown in
Figure 3. SN23260 suffers from 29 regions of concentrated bad pixels and 2
regions of excess dark current, which we refer to as ”leopard” and ”glow”
spots, respectively. The two glow spots are regions of excess dark current
which decrease in size as exposure time decreases. For consistency, we utilize
the 10 Hz mask for all data with frame rates greater than 10 Hz, as this
should always cover the two glow spots. In cases where we perform 1 or 5 Hz
integrations, we use a 1 or 5 Hz bad pixel mask (constructed in an analogous
way), respectively.

Figure 3: The edge mask and 10 Hz bad pixel mask for SN23260. The two ”glow”
spots are visible as the two large diameter circles on the mask. A horizontal
strip due to metadata processing during readout (which will be fixed in future
imaging software updates) appears at y = 1024.

We analyze the leopard and glow spots, focusing on their location and
stability for the leopard spots. Analysis of data taken in February 2023 at 85
Hz found the locations of the 29 leopard spots and determined that they could
be accurately fit with top-hat functions. We repeat that analysis with our own
85 Hz data from April 2024 generated from a mean of 128 frames. We find that,
with the exception of one spot (which is composed of a majority dead pixels),
all spots can be fit by top-hat functions with (x,y) positions and radii
within 2 pixels of their previously estimated values estimated. The two glow
spots likewise remained in the same location between the two data sets and had
FWHM 4 pixels larger than their prior values at the same temperature and frame
rate. The glow spots are the result of excess dark current and therefore their
change in size may be representative of a slightly higher dark current in our
testing environment.

Approximately 12,500 pixels are contained within the leopard spots, and, at 10
Hz, 11,000 pixels across both glow spots. This represents approximately 0.7%
of the 1848x1848 working area. The hot and zero-variance pixel percentage
excluding the edge mask and the spots then equals approximately 2.5%.

###  3.2 Gain and Well Depth

To measure the gain, we set the source to 200 K and look through the ammonia
filter. We produce ramps in 5% Tint steps. At each step, we generate a mean
frame from the time series of 128 frames, after applying the appropriate bad
pixel mask. As noted, the source is undersized for our detector and exhibits a
lateral gradient across it which may be due to the aging of the coating. We
utilize the region within (1250:1450,500:700) pixels (henceforth region A), as
marked in Figure 4, for our analysis. The accompanying plot in Figure 4 shows
the median x-value of each column within the region and slightly beyond.

Figure 4: (Top): A picture of the masked, 200 K source at 80% Tint, 20 Hz,
with the ammonia filter in place (including one of its slightly oversized
rectangular ”washers”). We select a small region of the detector for use in
the gain calculation and other tests. (Bottom): The median pixel value within
each column for region A and columns adjacent to it. A one-pixel wide spike
(not visible in the top picture at this scale) intercepts our region, causing
the outlier near pixel 1230.

We utilize the region A data to estimate the gain of SN23260. For each Tint
step, we calculate a standard deviation per pixel by comparing the values of
each pixel across all 128 frames. For each pixel, we perform a linear fit of
the pixel’s variance versus the median across the first 14 Tint steps (0%
through 60% Tint). Our gain per pixel is then the inverse of the slope. We
select the specific Tint range to avoid any non-linearity as we approach
saturation. A Gaussian fit to the gain per pixel histogram returns a mean of
194.7 e-/ADU and sigma of 40.5 e-/ADU (although there are obvious non-Gaussian
tails).

We crosscheck our gain result with a different approach that avoids using Tint
ramps. We define region B at (750:1500,500:1100), i.e., a rectangular region
covering most of the source. We estimate median pixel values and pixel
standard deviations across 128 $\times$ 20 Hz and 20% Tint frames. We solve
for the gain per pixel as the median pixel value divided by the pixel variance
(excluding bad pixels). We then fit a Gaussian to the histogram, recovering a
mean gain of 193.4 e-/ADU and sigma of 31.6 e-/ADU. We use 194.7 e-/ADU gain
for the remainder of the paper as it is based on a larger data volume.

Utilizing the fit to the saturated data from Figure 2, we can also estimate
the detector’s well depth. For a gain of 194.7 e-/ADU and mean saturated pixel
value of 14,130 ADU/pix (after removing the offset), our detector has a mean
well depth of 2.75 million electrons per pixel.

###  3.3 Read Noise

To determine the read noise, we take 128 frames at 85 Hz, 0% Tint with the
source off (T $\sim$ 45 K). We subtract adjacent frames, i.e., frame 1 by
frame 2 and then frame 3 by frame 4. We then calculated a standard deviation
for each pixel across its 64 values for all 1848x1848 pixels. We plot a
histogram of standard deviations, divided by square-root of two (such that
they represent the read noise of a single, non-subtracted frame) in Figure 5.
We fit a Gaussian to the data, recovering a mean read noise of 360 e-/pix
(1.85 ADU/pix) with sigma of 78 e-/pix (0.40 ADU/pix). The expected shot noise
due to dark current at 0.320 ms (the 0% Tint minimum exposure time) is 16
e-/pix (0.2% of the total). We also repeat the analysis with 100% Tint frames
at different frame rates. Accounting for shot noise, we find that read noise
does vary depending on the frame rate and Tint combination utilized. At 10 Hz
and 100% Tint, we estimate a read noise of 485 e-/pix (2.49 DN/pix) via this
method. The read noise appears to be stable for a given Tint and frame rate.

Figure 5: A histogram of pixel noise across 64 pair subtracted frames at 85
Hz, 0% Tint.

###  3.4 Bias and Dark Current

We estimate the bias and dark current for SN23260. For bias, we assume that
all signal contributions at 0% Tint are negligible (true exposure time is
0.320 ms). We create a mean frame from 128 $\times$ 0% Tint frames at 1 Hz. We
fit a Gaussian to the bias signal per pixel histogram and recover a mean of
1017.0 $\pm$ 0.1 ADU/pix.

To estimate the dark current, we repeat this process with 100% Tint frames
subtracted by the 0% Tint frames. We also multiply by the gain to convert into
e-/s/pix. We perform the analysis on a region behind the cold mask and then
across the unmasked area. First, we calculate the dark current in an ”edge”
region from (20:2028,10:40). We recover a Gaussian fit with mean of 330,000
e-/s/pix and sigma of 52,600 e-/s/pix, shown in Figure 6. We then perform the
analysis across the whole detector and with the bad pixel mask (which blocks
the edge region among other locations). This region could receive ambient
radiation originating from light leaks into the cryostat or reflected emission
from the warm capacitors on the back of SN23260. We recover a Gaussian fit
with a mean of 840,000 e-/s/pix and sigma of 131,000 e-/s/pix at 1 Hz.

For both the edge region and masked array, we recalculate dark current with 10
Hz data, recovering approximately 264,000 e-/s/pix and 777,000 e-/s/pix,
respectively. We believe the discrepancy is because Tint 0% is not truly an
IWR bias image and may be subject to both differences between a) ITR and IWR
offsets and b) effective dark current during the minimum exposures. As longer
exposures will be less impacted by the latter problem, we report 330,000
e-/s/pix as the device’s 45 K dark current.

Figure 6: A histogram of individual pixel dark currents with SN23260 at 45 K
within the cold masked edge of the device. This dark current is calculated via
a subtraction of the 100% and 0% Tint 1 Hz data.

We repeat the process in the edge region for 10 Hz dark data taken with the
detector at 47.5 K and 50 K. We find a dark current at 47.5 K of 506,000
e-/s/pix and a dark current at 50 K of 1.107 million e-/s/pix. Tennant et al.
determined an empirical expression for the dark current in MCT devices,
referred to as ”Rule 07” [8]. We plot our dark currents for 45 K, 47.5 K, and
50 K in A/cm2 alongside Rule 07 in Figure 7, scaled to the 50 K result. The
slope of our dark current is shallower than predicted by Rule 07 which may
indicate that SN23260 is hitting an electronics glow or ambient background
limited floor as it approaches 45 K. However, more data are needed at lower
temperatures to confirm.

Figure 7: The cold masked edge dark currents for SN23260 at three temperatures
versus the empirical prediction, Rule 07, as derived by Tennant et al. 2008
[8]. Our device displays a shallower dark current slope than Rule 07 predicts
which may be due to warm electronics glow providing a limiting dark current.

###  3.5 Linearity

To assess linearity, we utilize the same data used in the gain calculations
but work from a region C (850:1050, 500:700), located 400 pixels shortwards in
x from Region A’s center. This is done to place the zone of interest in a
location that will saturate at 70% Tint rather than 95% Tint. We produce a
mean image for each set of 128 frames and we then examine the behavior for
individual pixels.

We first define a pixel’s maximum as its value in the 100% (saturated) frame
and then quote its data as % of full-well. We use the data points below 40% to
fit a linear trend to the data as signal level versus exposure time. We
calculate the non-linear residual for each well-depth greater than 40%. In
Figure 8, we present a scatterplot of non-linear residuals versus their
corresponding well-depth. We fit a line to the non-linear residuals and find
that most pixels experience $<$ 2% non-linearity at $<$ 83% well-depth. While
not yet quantified, there is no reason to suspect variations in this behavior
over time, suggesting it can be calibrated.

Figure 8: A scatterplot of non-linear residual versus well-depth for all
pixels in region C over a series of integrations that culminate with
saturation.

###  3.6 1/f Noise

GeoSnap suffers from 1/f noise, indicating that greater time between exposures
result in more noise between frames. As described in Leisenring et al. 2023
[6], it is more accurate to say that this noise scales with the number of
frames. Thus, two images taken an equal number of frames apart will have the
same 1/f noise, regardless of the frame rates at which they were taken.
Leisenring et al. also reported that as the signal onto the detector
increased, the 1/f noise slope decreased.

We verify this 1/f behavior in SN23260 by taking 1,000 dark images at 10 Hz
and 50 Hz. We also take an additional 1,000 half-well images at 10 Hz using
the ammonia filter. Frames are subtracted from other frames in specific
integer spacings to replicate sampling at all accessible frequencies and we
work from region B. We display the data in terms of the ”frequency (per
frame)” in Figure 9. Frequency (per frame) is the sampling frequency divided
by the frame rate and is a means of showing different frame rate curves in
terms of the same number of images between a subtraction. So 0.1 frequency
(per frame) at 10 Hz means there is 1 second (10 images) between subtractions
while at 50 Hz it is 0.2 seconds (10 images) between subtractions. Similar to
the UM GeoSnap in MIRAC-5[6], SN23260 reaches the shot noise limited floor
with high frequency sampling. The behaviors of the 1/f are nearly identical
when viewed in terms of number of frames though their floors differ due to the
shot noise expected from the three different signal levels. The 10 Hz half-
well data has a far greater shot floor and demonstrates that as the signal
onto the detector increases, the 1/f noise slope decreases. In all cases, with
a frequency (per frame) greater than 0.2, the detector is shot noise
dominated.

Figure 9: 1/f behavior for SN23260 for different frame rates and well-depths.
Dashed-lines are used to show the theoretical noise limited floor for each
data set, assuming the only sources of noise in the individual frames are read
noise and shot noise added in quadrature. The significance of 1/f noise is
negligible if one modulates the signal at least every twenty frames.

###  3.7 Quantum Efficiency

To verify the quantum efficiency of our detector, we can utilize a flux
measurement taken at the detector plane. We can write the average signal per
pixel ($\mu_{pix}$) in SN23260 as:

$\mu_{pix}=\frac{f(\lambda)*QE(\lambda)*A_{pix}*t}{g*E_{\gamma}}$

In the above, $f(\lambda)$ is the flux at the detector plane, $QE(\lambda)$
the quantum efficiency in a narrow bandpass, $A_{pix}$ the pixel area, $t$ the
exposure time, $g$ the detector gain, and $E_{\gamma}$ the average photon
energy in the bandpass. By working only with narrow bandpass filters such as
the ammonia or 11 micron filter, we simplify our estimate as the anticipated
blackbody emission, throughput, photon energy, and quantum efficiency should
all be close to constant within the measurement regime.

To calculate quantum efficiency, we must solve this equation for
$QE(\lambda)$. The only unknown variable is $f(\lambda)$, the flux at the
detector plane. To determine $f(\lambda)$, we utilize a small single-cell
photovoltaic (PV) MCT detector from TIS: J19D13. The J19D13 detector is well-
calibrated with a known QE from 2 to 16 microns. J19D13 has a peak
responsivity of 5.70 A/W with 10% systematic uncertainty (via discussion with
TIS). It operates at 77 K and is 1.0 mm by 1.0 mm square detector active
region. We mount the J19D13 to the same position as SN23260 such that is
should occupy a 55.55 by 55.55 pixel space centered at pixel (1024,1024).
Thus, the pixel region 1000:1048 (exclusive) in both X and Y should experience
the same net flux on it as the calibrator detector.

To obtain our throughput estimate, we measure the median pixel signal in the
J19D13 region on SN23260 for several temperatures and filters, as shown in
Table 2. We warm the cryostat, swap in J19D13, and then cool the cryostat
again, leaving all other elements untouched between the thermal cycles. We
repeat the same combination of tests. The voltage across J19D13 is measured
via a voltmeter. The voltage is then converted to a current using the known
total resistance of 10,259.7 Ohms. Then, this current is multiplied by the
corresponding responsivity ($\pm$ 10%) in the bandpass, according to the
relative responsivity curve provided by TIS and the peak responsivity, to get
a wattage onto the detector. Using the active area of the detector, we
calculate the flux onto J19D13. We tested the uncertainty of our J19D13
voltages due to drifts in its bias value, stability of the source, and
variations across cooldowns. We found that systematic uncertainty across
cooldowns (of 3%) dominated over all other sources of uncertainty, with the
exception of very low flux measurements (such as a 200 K source measured at 11
micron filter measurements). Therefore, all J19D13 voltages are taken to have
a minimum 3% uncertainty.

For both SN23260 and J19D13, we take ”ambient dark” data (i.e., with the
source on but the filter wheel on the blank slot), and subtract that value
from the image and measured voltage, respectively. This is necessary since
even in the blank filter position the detector senses some light.

We provide the estimated flux (and corresponding throughputs) based on the
J19D13 data in Table 2. For the SN23260, we based our choice of frame rate and
Tint on remaining within the linear regime for the data set. We then calculate
the necessary SN23260 QE to match the measurement pairs. Working with the 300
K data, we find that the ammonia (10.6 micron) filter results in a QE of 79.7
$\pm$ 8.3 % while the 11 micron filter results in a QE of 79.0 $\pm$ 8.0 %.
For comparison, TIS estimated QEs of 88% at 8.69 microns, 76% at 10.77
microns, and 65% at 12.02 microns.

Table 2: SN23260 and J19D13 warm source measurements for several filters and source temperatures. Net values are found for both detectors after ”ambient” dark subtraction. Source | Filter | Net | Net | SN23260 | Measured | Estimated Flux | QE
---|---|---|---|---|---|---|---
Temp. |  | J19D13 | J19D13 | Exp. | SN23260 | SN23260 |
|  | Signal | Flux | Time | Signal | Signal |
(K) |  | (mV) | (erg/cm2/s) | (ms) | (e-/pix/frame) | ($\gamma$/pix/frame) | (%)
200 | Amm. | 0.32 $\pm$ 0.02 | 5.41 $\pm$ 0.60 | 19.860 | 1.66E6 $\pm$ 9E3 | 1.86E6 $\pm$ 2E5 | 89.5 $\pm$ 9.9
200 | 11 $\mu$m | 0.07 $\pm$ 0.01 | 1.19 $\pm$ 0.24 | 19.860 | 4.18E5 $\pm$ 2E3 | 4.35E5 $\pm$ 9E4 | 96.2 $\pm$ 19.4
|  |  |  |  |  |  |
250 | Amm. | 1.48 $\pm$ 0.02 | 24.8 $\pm$ 2.6 | 3.146 | 1.04E6 $\pm$ 5E3 | 1.35E6 $\pm$ 1E5 | 76.7 $\pm$ 8.0
250 | 11 $\mu$m | 0.33 $\pm$ 0.02 | 5.4 $\pm$ 0.60 | 11.624 | 8.46E5 $\pm$ 4E3 | 1.15E6 $\pm$ 1E5 | 73.8 $\pm$ 8.7
|  |  |  |  |  |  |
300 | Amm. | 3.57 $\pm$ 0.10 | 60.0 $\pm$ 6.2 | 0.885 | 7.32E5 $\pm$ 4E3 | 9.18E5 $\pm$ 1E5 | 79.7 $\pm$ 8.3
300 | 11 $\mu$m | 0.71 $\pm$ 0.01 | 11.7 $\pm$ 1.2 | 5.972 | 1.01E6 $\pm$ 5E3 | 1.28E6 $\pm$ 1E5 | 79.0 $\pm$ 8.0

##  4 Conclusion

We analyzed the performance of the 2048x2048 SN23260 GeoSnap with a B0 ROIC.
The proper quantification of this device is paramount as it is one of the
candidate detectors for METIS on the ELT. We measured various properties of
the detector, finding:

  1. 1.

At 10 Hz and in the inner 1848x1848 pixels, there are 3.17% bad pixels, 0.7%
of which are contained within twenty-nine ”leopard” spots and two ”glow” spots
on the detector.

  2. 2.

The detector has an average gain of 194.7 e-/ADU and mean well depth of 2.75
million e-/pix.

  3. 3.

The detector has an average read noise of 360 e-/pix.

  4. 4.

The detector has a dark current of approximately 330,000 e-/s/pix at 45 K in
the cold masked region measured at 1 Hz frame rate.

  5. 5.

The detector achieves $<$ 2% non-linearity at $<$ 83% well-depth.

  6. 6.

The detector displays 1/f noise matching trends observed with the previously
tested UM GeoSnap.

  7. 7.

The detector has a QE of 79.7 $\pm$ 8.3 % at 10.6 microns.

SN23260’s performance matched expectations based on our previous experience.
In the future, we would like to further explore possible detector glow or
ambient background. By obtaining information above 50 K and below 45 K, we can
check how the device performs relative to Rule 07. Although persistence is not
expected for CTIA devices, verification of this statement would be ideal. We
will also verify the temporal stability of all detector parameters, especially
linearity. Although the main testing for SN23260 is complete, more thorough
tests for it and other METIS candidate GeoSnap will be completed in the
future.

###### Acknowledgements.

We thank Vincent Douence from TIS for his major contributions in tuning and
operating of GeoSnap SN23260 and other equipment. We also thank Henry Yuan and
David Bond from TIS for their assistance with planning and utilizing the
J19D13 detector for the QE assessments. We are grateful to the University of
Michigan for their support of the infrared detector test lab.

## References

  * [1] Brandl, B., Bettonvil, F., van Boekel, R., Glauser, A., Quanz, S., Absil, O., Amorim, A., Feldt, M., Glasse, A., Güdel, M., Ho, P., Labadie, L., Meyer, M., Pantin, E., van Winckel, H., and METIS Consortium, “METIS: The Mid-infrared ELT Imager and Spectrograph,” The Messenger 182, 22–26 (Mar. 2021).
  * [2] Bowens, R., Meyer, M., Delacroix, C., Absil, O., van Boekel, R., Quanz, S., Shinde, M., Kenworthy, M., Carlomagno, B., Orban de Xivry, G., Cantalloube, F., and Pathak, P., “Exoplanets with ELT-METIS I: Estimating the Direct Imaging Exoplanet Yield Around Nearby Stars within 6.5 Parsecs,” in [European Planetary Science Congress ], EPSC2021–79 (Sept. 2021).
  * [3] McMurtry, C., Lee, D., Beletic, J., Chen, C.-Y. A., Demers, R. T., Dorn, M., Edwall, D., Fazar, C. B., Forrest, W. J., Liu, F., Mainzer, A. K., Pipher, J. L., and Yulius, A., “Development of sensitive long-wave infrared detector arrays for passively cooled space missions,” Optical Engineering 52, 091804 (Sept. 2013).
  * [4] Smith, R. M., Zavodny, M., Rahmer, G., and Bonati, M., “A theory for image persistence in HgCdTe photodiodes,” in [High Energy, Optical, and Infrared Detectors for Astronomy III ], Dorn, D. A. and Holland, A. D., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 7021, 70210J (July 2008).
  * [5] Beletic, J. W., Blank, R., Gulbransen, D., Lee, D., Loose, M., Piquette, E. C., Sprafke, T., Tennant, W. E., Zandian, M., and Zino, J., “Teledyne Imaging Sensors: infrared imaging technologies for astronomy and civil space,” in [High Energy, Optical, and Infrared Detectors for Astronomy III ], Dorn, D. A. and Holland, A. D., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 7021, 70210H (July 2008).
  * [6] Leisenring, J. M., Atkinson, D., Bowens, R., Douence, V., Hoffmann, W. F., Meyer, M. R., Auyeung, J., Beletic, J., Cabrera, M. S., Greenbaum, A. Z., Hinz, P., Ives, D., Forrest, W. J., McMurtry, C. W., Pipher, J. L., and Viges, E., “Evaluating the GeoSnap 13‑$\mu$m cutoff HgCdTe detector for mid‑IR ground‑based astronomy,” Astronomische Nachrichten 344, e20230103 (Oct. 2023).
  * [7] Cabrera, M. S., McMurtry, C. W., Dorn, M. L., Forrest, W. J., Pipher, J. L., and Lee, D., “Development of 13-$\mu$m cutoff HgCdTe detector arrays for astronomy,” Journal of Astronomical Telescopes, Instruments, and Systems 5, 036005 (July 2019).
  * [8] Tennant, W. E., Lee, D., Zandian, M., Piquette, E., and Carmody, M., “MBE HgCdTe Technology: A Very General Solution to IR Detection, Described by “Rule 07”, a Very Convenient Heuristic,” Journal of Electronic Materials 37, 1406–1410 (Sept. 2008).
  * [9] Bowens, R., Viges, E., Meyer, M. R., Atkinson, D., Monnier, J., Morgenstern, M., Leisenring, J., and Hoffmann, W., “The Michigan infrared test thermal ELT N-band (MITTEN) cryostat,” in [Ground-based and Airborne Instrumentation for Astronomy VIII ], Evans, C. J., Bryant, J. J., and Motohara, K., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 11447, 1144737 (Dec. 2020).
  * [10] Bowens, R., Leisenring, J., Meyer, M., Montoya, M., Hoffman, W., Morzinski, K., Hinz, P., Monnier, J. D., Bergin, E., Viges, E., Calissendorff, P., Forrest, W., McMurtry, C., Pipher, J., and Cabrera, M., “MIRAC-5: a ground-based mid-IR instrument with the potential to detect ammonia in gas giants,” in [Ground-based and Airborne Instrumentation for Astronomy IX ], Evans, C. J., Bryant, J. J., and Motohara, K., eds., Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series 12184, 121841U (Aug. 2022).