# Convolutional Autoencoder-Based Phase Shift Feedback Compression for
Intelligent Reflecting Surface-Assisted Wireless Systems

Xianhua Yu, Dong Li, Yongjun Xu and Ying-Chang Liang  This work was supported
in part by The Science and Technology Development Fund, Macau SAR, under
Grants 0003/2019/A1, 0018/2019/AMJ, 0009/2020/A1 and 0110/2020/A3, and in part
by the Natural Science Foundationof China, under Grants 61601071 and 62071078.
(Corresponding author: Dong Li.) Xianhua Yu and Dong Li are with the Faculty
of Information Technology, Macau University of Science and Technology, Macau
999078, China (e-mails<EMAIL_ADDRESS>dli@must.edu.mo).
Yongjun Xu is with the School of Communication and Information Engineering,
Chongqing University of Posts and Telecommunications, Chongqing 400065, China
(e-mail: xuyj@cqupt.edu.cn). Ying-Chang Liang is with the Center for
Intelligent Networking and Communications (CINC), University of Electronic
Science and Technology of China, Chengdu 611731, China, and also with the
Faculty of Information Technology, Macau University of Science and Technology,
Macau 999078, China (e-mail: liangyc@ieee.org).

###### Abstract

In recent years, intelligent reflecting surface (IRS) has emerged as a
promising technology for 6G due to its potential/ability to significantly
enhance energy- and spectrum-efficiency. To this end, it is crucial to adjust
the phases of reflecting elements of the IRS, and most of the research works
focus on how to optimize/quantize the phase for different optimization
objectives. In particular, the quantized phase shift (QPS) is assumed to be
available at the IRS, which, however, does not always hold and should be fed
back to the IRS in practice. Unfortunately, the feedback channel is generally
bandwidth-limited, which cannot support a huge amount of feedback overhead of
the QPS particularly for a large number of reflecting elements and/or the
quantization level of each reflecting element. In order to break this
bottleneck, in this letter, we propose a convolutional autoencoder-based
scheme, in which the QPS is compressed on the receiver side and reconstructed
on the IRS side. In order to solve the problems of mismatched distribution and
vanishing gradient, we remove the batch normalization (BN) layers and
introduce a denosing module. By doing so, it is possible to achieve a high
compression ratio with a reliable reconstruction accuracy in the bandwidth-
limited feedback channel, and it is also possible to accommodate existing
works assuming available QPS at the IRS. Simulation results confirm the high
reconstruction accuracy of the feedback/compressed QPS through a feedback
channel, and show that the proposed scheme can significantly outperform the
existing compression algorithms.

###### Index Terms:

Intelligent reflecting surface, quantized phase shift, feedback compression,
autoencoder, convolutional neural network

Figure 1: An IRS-assisted wireless system

##  I Introduction

Driven by the explosive growth of wireless applications like high-resolution
streaming, video calls, virtual reality, and augmented reality (AR), there is
an increasing demand for high data-rate transmission in wireless transmission.
Existing wireless solutions (see, e.g., [1] and [2]), such as ultra-dense
network (UDN), massive multiple-input multiple-output (MIMO), and the
millimeter-wave (mmWave), are able to support massive access and achieve high
spectral efficiency. However, they consume more power and increase hardware
costs. Specifically, compared with the traditional cellular networks, the UDN
is equipped with a higher density of base stations (BSs) and access points
(APs), which increases the hardware investment, maintenance cost, and severe
interference. Besides, extending the fundamental sub-6 GHz to mmWave frequency
in massive MIMO needs more complex signal processing and costly energy-
consuming hardware.

Intelligent reflecting surface (IRS), as a promising innovative technology,
has recently attracted much attention to supporting spectrum-efficient and
energy-efficient wireless communications (see, e.g., [3] and [4]). To be
specific, the IRS is a two-dimensional human-made surface consisting of plenty
of low-cost passive reflecting elements with adjustable phases, where the IRS
is connected with a BS/AP via a smart controller. Thus, in the IRS-assisted
system, it is crucial to control the phases of reflecting elements for
performance enhancement, and extensive studies have been paid to compute
optimal/sub-optimal phases for different optimization objectives (see, e.g.,
[5, 6, 7]). However, in most of the existing works, it is assumed that the
quantized phase shift (QPS) is available at the IRS side, which is difficult
to obtain due to the bandwidth/data-rate limitation of feedback channels as
indicated, for instance, in the long-term evolution (LTE) standard. Thus, a
natural question arises: how to deliver the QPS accurately through the band-
limited feedback channel in the IRS-assisted wireless system particularly when
there are a huge number of reflecting elements and/or a large number of
quantization levels for each reflecting element? It should be noted that this
problem is neglected in the existing works, and remains unsolved.

Motivated by this observation, in this letter, we propose an autoencoder-based
scheme named phase shift compression and denoising network (PSCDN) in the IRS-
assisted wireless system. Specifically, the PSCDN-encoder stochastically maps
the QPS information on the receiver (user) side to a code with a smaller
dimension in the feature space, and sends it to the IRS side through a
feedback channel. The PSCDN-decoder learns to recover the original QPS
information from the corrupted code with noise. In particular, we build the
PSCDN with multiple convolutional neural network (CNN) layers to reduce the
computational complexity[8]. The contributions of this letter are summarized
as follow:

  * •

Due to the batch normalization (BN) layers [9], the decoded QPS follows the
Gaussian-like distribution [10], which violates the uniform distribution of
the original QPS. In this case, the loss between them is hard to be minimized.
Thus, in order to solve this problem, we remove BN layers to keep the
distributions of the model’s input and output consistent with the uniform
distribution.

  * •

However, by removing the BN layers, the problem of vanishing gradient arises.
To this end, we introduce a denoising module to solve this problem.

  * •

Simulation results demonstrates that our PSCND with above two modifications
can achieve a reliable reconstruction accuracy but with a low computational
complexity by comparing it with existing algorithms.

It should be noted that traditional autoencoder-based compression works (see,
e.g., [11, 12, 13]) cannot be readily applied to this work due to the above
two modifications (i.e., removing the BN layers and introducing the denoising
module). This makes our work significantly different from previous works.
Besides, in most of previous works, the channel state information (CSI) was
considered to be fed back, in which the feedback overhead is, however,
proportional to the number of transmit antennas. In this work, there is no
such issue in the feedback of the QPS, which only depends on the number of
reflecting elements and quantization levels (as shown in the following
section). Furthermore, it is worth mentioning that the feedback of QPS is
unique in the IRS-assisted systems, and there is no attention that has been
paid to this problem to the best of the authors’ knowledge.

##  II System Model and Problem Formulation

###  II-A System Model

As shown in Fig. 1(a), we consider a typical IRS-assisted wireless system,
where the IRS is equipped with $M$ reflecting elements ($\forall
m\in\mathcal{M}=\left\\{1,2,\ldots,M\right\\}$) to assist the communication
between the AP and the user/receiver.111Note that, in this letter, we assume
that the QPS at the user side is obtainable, and we only focus on the feedback
issue. In this regard, the feedback compression method developed in this work
can be easily extended to scenarios with multiple antennas and/or multiple
users assuming the available QPS, which can be obtained by using the methods
developed by, e.g., [4] and [7]. The IRS is attached with a controller which
is allowed to adjust each reflecting element’s phase. The optimized/computed
phases are assumed to be available in the IRS, and we consider a band-limited
feedback channel to deliver the QPS. By denoting
$\textbf{h}_{sr}\in\mathcal{C}^{M\times 1}$,
$\textbf{h}_{rd}\in\mathcal{C}^{M\times 1}$, $h_{sd}\in\mathcal{C}$ as the
channel coefficients between the AP and the IRS, the IRS and the user, the
user and the AP, respectively, and $P$ and $s$ as the transmit power and the
transmitted signal of $S$, the received signal at the user is given by

$y_{d}=\sqrt{P}\textbf{h}^{T}_{rd}\Phi\textbf{h}_{sr}s+\sqrt{P}h_{sd}s+u_{d},$
(1)

where
$\Phi=\rho\rm{diag}(e^{j\theta_{1}},e^{j\theta_{2}},\ldots,e^{j\theta_{M}})\in\mathcal{C}^{M\times
M}$ is the phase matrix with $\rho\in(0,1]$ and
$\left\\{\theta_{m}\right\\}^{M}_{m=1}$ denotes the reflection coefficient and
phases of the IRS, and $u_{d}\sim\mathcal{CN}(0,\sigma_{d}^{2})$ is the
additive white Gaussian noise (AWGN). Besides, in (1), the optimal phase shift
can be given by
$\theta^{*}_{m}=\arg(h_{sd})-\arg([\textbf{h}_{sr}]_{m}[\textbf{h}_{rd}]_{m})$
for capacity maximization (see, e.g., [14, 15]). Besides, due to the finite
resolution of the IRS, $\theta_{m}$ can only take a finite number of discrete
values (see, e.g., [14, 15]), i.e., $2^{K}$ quantization levels and $K$
denotes the number of quantization bits. The phase $\theta_{m}$ can be
uniformly quantized and the set of QPS is given by
$\left\\{0,\frac{2\pi}{2^{K}},\cdots,\frac{(2^{K}-1)2\pi}{2^{K}}\right\\}$.
Thus, $\theta_{m}$ can be represented by $K$ bits by using, for instance, the
phase shift keying (PSK) mapping.

(a) Traditional autoencoder-based compression

(b) PSCDN (PSCDN without the denoising module is denoted as PSCN)

Figure 2: The architectures of the traditional autoencoder-based compression
method and the proposed PSCDN

###  II-B Problem Formulation

In this letter, our goal is to reduce the feedback overhead by designing a new
convolutional autoencoder, i.e., PSCDN, as illustrated in Fig. 1(b). The
autoencoder consists of two parts: an encoder and a decoder[8]. The encoder
stochastically maps the input $x$ to the feature space and then outputs a code
$c$, i.e., $c=f(x)$. The decoder reconstructs the estimated input $\hat{x}$
from a feature space code $c$, i.e., $\hat{x}=g(c)$. Furthermore, the
autoencoder whose code dimension is less than the input dimension is called
under complete. Learning an under complete representation forces the
autoencoder to capture the most important features of the input data. The
learning process is tantamount to minimizing a loss function, i.e.,
$L(x,\hat{x}=g(f(x)))$, which penalizes $\hat{x}$ for being dissimilar from
$x$. The PSCDN-encoder learns to map
$\Theta=[\theta_{1},\theta_{2},\cdots,\theta_{M}]$ to a code (with a low-
dimension data), and the PSCDN-decoder learns to recover the phase information
from the code. Specifically, the processes of the encoder and the decoder are
given by $C=f(\Theta)$ and $~{}\hat{\Theta}=g(C)$, where $C$ denotes the code
in the feature space.

##  III Neural Network For Phase Compression

The CNN is a state of art architecture in deep learning which has achieved
tremendous success in image processing because of the properties of CNN (parse
interactions, parameter sharing, and equivariant representations)[8]. Our
feedback compression task is similar to the super-resolution and image
denoising in image processing, which aims at recovering the complete
information from the partial one and alleviating the Gaussian noise effect.
Due to the success of CNN in the above area and its low computational
complexity, we chose CNN to build our model.

###  III-A Architecture of PSCDN

The architecture of PSCDN is given in Fig. 2(b). In the proposed PSCDN, there
are three types of CNN layers: (i) Conv+ReLU: the kernel size of the
convolutional layer is 3 and the convolutional layer uses $N$ filters to
generate $N$ feature maps. The activation function is the rectified linear
units (ReLU) that transforms the obtained linear features into nonlinear
features; (ii) Conv: In the encoder, the kernel size and the filter size of
the convolutional layer are 1 and $C$, respectively. In the decoder, the
filter size of the convolutional layer is equal to $N$; (iii) Conv+sigmoid:
the kernel size and the number of filters of the convolutional layer are 3 and
$K$, respectively, where the sigmoid activation function is given by
$\phi_{s}(z)=\frac{1}{1+e^{-z}}$.

Similar to traditional compression methods (see, e.g., [11, 12, 13]), we build
the PSCDN with 4 layers in the encoder and 11 layers in the decoder (except
the reshape layer and the concatenation layer). In the PSCDN-encoder, the
input data size is $K\times 1$, the reshape layer reshapes the data size to
$1\times KN$. The last layer will map the data to the feature space with the
size $1\times C$. The compression ratio (CR) is defined as ${\rm
CR}=\frac{C}{K}$. In the PSCDN-decoder, the last Conv+sigmoid layer uses $K$
filters to reconstruct the QPS and regularized its value to $[0,1]$ by the
sigmoid function.

Fig. 2 shows the architectures of a traditional autoencoder-based compression
method and the proposed PSCDN. Comparing with these two architectures, the
PSCDN has made two improvements: 1) we abandon using BN layers (the red block
in Fig. 2(a)); 2) we introduce a novel denoising module (the fuchsia block in
Fig. 2(b)), which will be elaborated in the following sections.

###  III-B Removing the Batch Normalization Layers for Uniformly Distributed
QPS

In recent years, the BN has been widely used in deep learning, which is used
to make artificial neural networks faster and more stable through the
normalization of the layers’ inputs. At each hidden layer, the BN process is
given by

$z^{(i)}_{norm}=\frac{z^{(i)}-\overline{z^{(i)}}}{\sqrt{\sigma_{SD}^{2}+\epsilon}},$
(2)

where $z^{(i)}$ is the mini-batch input,
${\overline{z^{(i)}}=\frac{1}{n}\sum_{i=1}^{n}z^{(i)}}$ is the mean of the
mini-batch input, and
$\sigma_{SD}=\frac{1}{n}\sum_{i=1}^{n}(z^{(i)}-\overline{z^{(i)}})$ is the
standard deviation. Besides, $z^{(i)}_{norm}$ is the normalized mini-batch
input, $\epsilon$ is an arbitrarily small constant that is involved in the
denominator for numerical stability. Thus, $z^{(i)}_{norm}$ has zero mean and
unit variance, which follows a Gaussian-like distribution[10].

As mentioned earlier, our goal is to minimize the loss between the original
QPS and its reconstruction. It can be found that, the loss is minimized if the
distributions of $\hat{\Theta}$ and $\Theta$ are the same. The input data
follows the uniform distribution, but the BN process will transform the input
into a Gaussian-like distribution that differs from the uniform distribution.
In order to circumvent this problem, we use two types of layers, (i)
CONV+ReLU; (ii) CONV+BN+ReLU, to evaluate whether we need BN layers in our
model. The two types of layers can be formulated as

$\displaystyle\hat{\Theta}$ $\displaystyle=\phi_{r}(W(\Theta)),$ (3)
$\displaystyle\hat{\Theta}$ $\displaystyle=\phi_{r}(BN(W(\Theta))),$ (4)

where $\phi_{r}(\cdot)$ is the activation function ReLU, and $W$ is the weight
of the convolutional layer and $BN$ is the BN process. If compared with the
first type, the second type encourages faster convergence and a more smooth
training procedure, but with a higher computational complexity.

For ease of illustration, let us first consider the PSCDN without denoising
module (denoted as PSCN), which only consists of multiple CONV layers. For
comparison, we evaluate six types of PSCN with multiple BN layers: (a) There
is no BN layer in both the encoder and the decoder; (b)-(e) The number of type
(ii) layers in the encoder ranges from 1 to 4; (f) all CONV layers belong to
type (ii) except reshape layers. We evaluate the performance by using the
normalized mean square error (NMSE), which is defined as

${\rm NMSE}=\frac{\|\Theta-\hat{\Theta}\|^{2}_{2}}{\|\Theta\|^{2}_{2}}.$ (5)

With the empirical results in [16], we train and test the models under SNR 10
dB, $\rm{CR}=\frac{2}{9}$ and $\rm{CR}=\frac{3}{9}$, respectively. The NMSE
results are shown in Table I. With the increasing number of BN layers, the
NMSE performance decreases. This verifies our analysis on the impact of the
BN, i.e., the BN will break the QPS’s uniform distribution during the training
process. The performance of the PSCN without BN layers is worse than the PSCN
with one or two BN layers, since the deep model is often accompanies by the
vanishing gradient problem.

TABLE I: NMSE performance of Different Types Of PSCN CR Type | (a) | (b) | (c) | (d) | (e) | (f)
---|---|---|---|---|---|---
$\frac{2}{9}$ | 0.119 | 0.114 | 0.117 | 0.131 | 0.204 | 0.269
$\frac{3}{9}$ | 0.0428 | 0.0381 | 0.0390 | 0.0442 | 0.113 | 0.159

###  III-C Introducing Denoising Module to Combat the Vanishing Gradient
Problem

Although the problem of mismatched distribution can be solved by deleting the
BN layers, it brings a new problem, i.e., the vanishing gradient problem.
Inspired by the success in image denoising by using the deep CNN and the
residual learning operation[17], we consider the denoising module to mitigate
the vanishing gradient problem. The denoising module is shown in Fig 2(b),
which consists of a concatenation layer, a CONV layer with the kernel size of
1, and a residual learning operation.

In the PSCDN-decoder, after extracting the noise information from the previous
Conv+ReLU layer, we use a concatenation layer to concatenate the noisy input
of the decoder and the observed noise feature. The concatenation operation
produces the critical noise information. After concatenation, we use the CONV
layer to obtain more accurate features, where the output size is the same as
the noisy QPS. Then the residual learning operation is used to subtract the
input of the decoder from the observed noise feature. Therefore we will
receive a less noisy QPS, and the vanishing gradient problem will be
alleviated because of the residual learning operation.

To evaluate the impact of the denoising module, we apply PSCDN to reproduce
the previous experiment here. The NMSE performance of PSCDN with
$\rm{CR}=\frac{2}{9}$ and $\rm{CR}=\frac{3}{9}$ are 0.113 and 0.0354, which
perform much better than their counterparts in Table I.

###  III-D Training Procedure

To train the PSCDN, we use end-to-end training for all weights and biases by
computing the loss. In the training state, we quantize the phase information
to $K=9$ bits. The training samples are taken as the input and the ground
truth of the PSCDN. The noise will be added to the output of the encoder in
the training stage. The training process is given by

$\hat{\Theta}=f_{dec}(g\cdot f_{enc}(\Theta)+n),$ (6)

where $\hat{\Theta}$ is the output of PSCDN, and $g$ is the constant channel
coefficient [18] and $n$ is the AWGN.

We chose the mean squared error (MSE) as the loss function, which can be
expressed as

$L=\frac{1}{M}\sum^{M}_{m=1}\|\hat{\theta}_{m}-\theta_{m}\|^{2}_{2},$ (7)

where $\hat{\theta}_{m}$ and $\theta_{m}$ represent the $m$-th column of
$\hat{\Theta}$ and $\Theta$, respectively. We chose Adam algorithm[19] as the
optimizer to update the parameters.

Furthermore, we use the exponential decay as the learning rate schedule to
gradually decrease the learning rate during the training state. The
exponential decay is given by

$lr=lr_{0}*\rm{decay~{}rate^{\frac{total~{}steps}{decay~{}steps}}},$ (8)

where $lr$ and $lr_{0}$ are the current learning rate and the initial learning
rate, respectively.

##  IV Simulation Results And Analysis

We adopt the NMSE to evaluate the performance of the proposed PSCDN. The PSCDN
is trained with SNR 10 dB, and the training epochs are 1000. Without loss of
generality, we set the channel coefficient $g$ to be 1. We generate training
samples, validation samples, and test samples according to the uniform
distribution. The sizes of training samples, validation samples, and test
samples are 100000, 30000 and 100000, respectively. We use decay rate=0.99,
decay steps=1000, which means the learning rate multiplied by a decay rate of
0.99 after every 1000 steps.

Figure 3: NMSE performance (dB) comparison for different compression
algorithms.

For comparison, we consider the following autoencoder-based compression
methods the CsiNet[11], the RecCsiNet[12] and the DS-NLCsiNet[13] for
performance comparison. Since our data type is with one dimension, the kernel
size in CsiNet, RecCsiNet and DS-NLCsiNet will switch to $3$, and other
hyperparameter settings are the same as the PSCDN. We not only evaluate the
NMSE performance for different compression methods through the feedback
channel, but also evaluate the model size and the computational complexity by
calculating the number of parameters and examining the test running time,
respectively.

Fig. 3 shows the performance of the PSCDN, the CsiNet, the RecCsiNet and the
DS-NLCsiNet with CRs $\frac{2}{9},~{}\frac{3}{9},~{}\frac{4}{9}$ and
$\frac{5}{9}$ through the feedback channel. As can be seen from Fig. 3, the
performance of all compression methods increases with the increasing
compression ratio as expected. Besides, for each compression ratio, it is
always observed that the proposed PSCDN outperforms three benchmark schemes.
The observation above can be explained in the following. The code in the
feature space will learn the representation of the information bits by several
neurons through the encoder. It is well-known that the neural network with
more neurons usually accompanies a satisfactory performance. Therefore, a
higher dimension code is easier to learn more representation of the QPS than a
lower one. As a result, the compression methods with a higher CR achieve
better performance with more representation learned. Besides, the denoising
module in the PSCDN-decoder is able to mitigate the gradient vanishing
problem, which enhances the performance of the PSCDN.

Table II shows the comparison of model size and computational complexity for
different compression methods with compression ratio $\frac{2}{9}$ and
SNR=10dB. The CsiNet shows the smallest model size and the lowest
computational complexity but has the worst NMSE. However, the model size and
computational complexity of the proposed PSCDN are slightly larger than the
CsiNet, but the PSCDN achieves the best NMSE performance, as shown in Fig. 3.
On the other hand, although the DS-NLCsiNet achieves almost the same NMSE
performance as the proposed PSCDN, its model size and computational complexity
are much larger than that of the PSCDN. This indicates that the proposed PSCDN
has better performance in practice.

TABLE II: Comparison of Computational Complexity and Model Size Test Methods | CsiNet | RecCsiNet | DS-NLCsiNet | PSCDN
---|---|---|---|---
model size | 79235 | 1417883 | 302979 | 85699
running time | 0.339s | 0.451s | 0.506s | 0.378s

##  V Conclusion

In this letter, we investigate and analyze the problem of the feedback
compression for the QPS in the IRS-assisted wireless system by proposing a
novel autoencoder-based model, i.e., PSCDN. In the proposed PSCDN, we make two
major modifications compared with traditional convolutional autoencoder-based
schemes by removing the BN layers and introducing the denoising module. As a
result, the problems of the mismatched distribution and the vanishing gradient
can be solved. Simulation results demonstrated that the PSCDN can achieve a
much better performance but with a low computational complexity compared with
existing algorithms.

## Acknowledgment

Xianhua Yu would like to extend his sincere thanks to Professor Zhengdao Wang
from Iowa State University for his helpful comments during the preparation of
this letter.

## References

  * [1] F. Boccardi et al., “Five disruptive technology directions for 5G,” _IEEE Commun. Mag._ , vol. 52, no. 2, pp. 74-80, Feb. 2014.
  * [2] Z. Yang et al., “Federated learning for 6G: Applications, challenges, and opportunities,” _arXiv preprint._ , arXiv:2101.01338, 2021.
  * [3] S. Gong et al., “Towards smart wireless communications via intelligent reflecting surfaces: A contemporary survey,” _IEEE Commun. Surveys Tuts._ , vol. 22, no. 4, pp. 2283-2314, 4th Quart., 2020.
  * [4] Z. Yang et al., “Energy-efficient wireless communications with distributed reconfigurable intelligent surfaces,” _IEEE Trans. Wireless Commun._ , doi: 10.1109/TWC.2021.3098632., 2021
  * [5] Z. Gao, Y. Xu, Q. Wang, Q. Wu and D. Li, “Outage-constrained energy efficiency maximization for RIS-assisted WPCNs,” _IEEE Commun. Lett._ , doi:10.1109/LCOMM.2021.3101657., 2021
  * [6] C. Pan et al. “Multicell MIMO communications relying on intelligent reflecting surfaces” _IEEE Trans. Wireless Commun._ , vol. 19, no. 8, pp. 5218-5233, Aug. 2020.
  * [7] Y. Xu et al., “RIS-enhanced WPCNs: Joint radio resource allocation and passive beamforming optimization,” _IEEE Trans. Veh. Technol._ , doi: 10.1109/TVT.2021.3096603., 2021
  * [8] Goodfellow et al., “Deep Learning,” _MIT Press._ , 2015.
  * [9] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proc. ICML._ , vol. 37, pp. 448-456, 2015.
  * [10] H. Daneshmand, A. Joudaki and F. Bach, “Batch normalization orthogonalizes representations in deep random networks,” _arXiv preprint._ , arXiv:2106.03970, 2021.
  * [11] C. Wen et al., “Deep learning for massive MIMO CSI feedback,” _IEEE Wireless Commun. Lett._ , vol. 7, no. 5, pp. 748-751, Oct. 2018.
  * [12] C. Lu et al., “MIMO channel information feedback using deep recurrent network,” _IEEE Commun. Lett._ , vol. 23, no. 1, pp. 188–191, Jan. 2019.
  * [13] X. Yu et al., “DS-NLCsiNet: Exploiting non-local neural networks for massive MIMO CSI feedback,” _IEEE Wireless Commun. Lett._ , vol. 24, no. 12, pp. 2790-2794, Dec. 2020.
  * [14] D. Li, “Ergodic capacity of intelligent reflecting surface-assisted communication systems with phase errors,” _IEEE Commun. Lett._ , vol. 24, no. 8, pp. 1646-1650, Aug. 2020.
  * [15] D. Li, “Fairness-aware multiuser scheduling for finite-resolution intelligent reflecting surface-assisted communication,” _IEEE Commun. Lett._ , vol. 25, no. 7, pp. 2395-2399, Jul. 2021.
  * [16] J. Guo et al., “Convolutional neural network-based multiple-rate compressive sensing for massive MIMO CSI feedback: Design, simulation, and analysis,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 4, pp. 2827-2840, Apr. 2020.
  * [17] K. Zhang et al., “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” _IEEE Trans. Image Process._ , vol. 26, no. 7, pp. 3142-3155, Jul. 2017.
  * [18] D. J. Love et al., “An overview of limited feedback in wireless communication systems,” _IEEE J. Sel. Areas Commun._ , vol. 26, no. 8, pp. 1341-1365, Oct. 2008.
  * [19] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR._ , pp. 1–41, 2015.


# Calibrated Interpretation: Confidence Estimation in Semantic Parsing

Elias Stengel-Eskin
Johns Hopkins University
<EMAIL_ADDRESS>Benjamin Van Durme
Johns Hopkins University
<EMAIL_ADDRESS>
###### Abstract

Sequence generation models are increasingly being used to translate natural
language into programs, i.e. to perform executable semantic parsing. The fact
that semantic parsing aims to predict programs that can lead to executed
actions in the real world motivates developing safe systems. This in turn
makes measuring calibration – a central component to safety – particularly
important. We investigate the calibration of popular generation models across
four popular semantic parsing datasets, finding that it varies across models
and datasets. We then analyze factors associated with calibration error and
release new confidence-based challenge splits of two parsing datasets. To
facilitate the inclusion of calibration in semantic parsing evaluations, we
release a library for computing calibration metrics.111Models/Analysis/Data:
https://github.com/esteng/calibration_miso, Metric:
https://github.com/esteng/calibration_metric

##  1 Introduction

When probabilistic models are used for decision-making or interaction, we not
only want the model to be accurate but also to be _calibrated_ , meaning that
the probability the model assigns to a decision should roughly correspond to
its likelihood of being correct. For example, if a model translates a user
instruction into a program for a robot, the confidence that the program
accurately captures the person’s intent may inform whether the robot executes
the program. Because of its importance to safety and usability, a large body
of work has addressed calibration, with most research considering single-
timestep classification models (i.e. one output decision per input). However,
probabilistic models are also commonly applied to multi-timestep tasks. In
these tasks, the model generates a _sequence_ of decisions, with each
subsequent decision dependent on the input and the previous decisions. For
example, text generation models typically predict sequences of words one at a
time, with each new word conditioned on the previous outputs. Although
calibration has been measured in some text generation tasks like machine
translation (Kumar and Sarawagi, 2019; Wang et al., 2020), determining the
accuracy of generated text sequences can be challenging, making calibration –
the relationship between accuracy and confidence – hard to estimate. As
sequence generation models, especially large language models, play a role in a
growing variety of tasks, measuring their calibration has become extremely
important.

The task of executable semantic parsing, where a model predicts an executable
program from a natural language instruction, is often modeled as a sequence
generation task. Such parsing models are used in systems that interact with
the real world, such as human-robot interfaces (Tellex et al., 2020) and
digital assistants (Gupta et al., 2018; Cheng et al., 2020; Semantic Machines
et al., 2020). Since actions in these domains – especially physical domains –
can have irreversible effects, the importance of ensuring model safety cannot
be understated. Thus, semantic parsing provides a clear motivation for having
a well-calibrated sequence generation model: At low confidence, we may prefer
for the system to defer action or request clarification, while when confidence
is high, these actions may unnecessarily annoy a user and make the system
unusable. This reasoning presupposes that the model’s confidence is well-
correlated with its probability of success.

Simultaneously, the constrained and executable nature of semantic parses makes
accuracy easier to measure. In many text-based sequence generation tasks like
machine translation, summarization, long-form question-answering (QA), and
open-ended dialogue, evaluating the quality and correctness of a generation
poses a variety of challenges, often due to the fact that there are many ways
of stating roughly the same proposition in language. Because calibration
measures the relationship between accuracy and confidence, our ability to
measure calibration will only be as good as our ability to measure accuracy.
In executable semantic parsing, the model generates a program (rather than
text) with a more restricted vocabulary and known syntactic rules. This
generally limits the number of reasonable semantically equivalent outputs and
makes quantifying accuracy easier. Furthermore, the executable nature of the
programs allows us to measure accuracy via denotation (i.e. the result of
program execution) rather than form. These factors make it an ideal domain for
benchmarking the calibration of sequence generation models.

In Section 4, we conduct what is to our knowledge the first large-scale
investigation of calibration in sequence generation models as applied to
semantic parsing tasks. We examine a variety of commonly-used models and
measure their calibration across four popular semantic parsing datasets drawn
from two different domains: task-oriented dialogue (TOD) and text-to-SQL. We
first document the model’s calibration profiles, asking how well-calibrated
modern semantic parsing systems are; this includes a large pre-trained model
queried in a few-shot setting, as well as more traditional fine-tuned models.
Using qualitative and quantitative metrics, we find that most TOD models are
already fairly well-calibrated. However, the same models are poorly-calibrated
on text-to-SQL datasets. In Section 5, we analyze various factors implicated
in calibration, attempting to shed light on the differences documented in
Section 4. We first find that dataset size does not account for the difference
in calibration between TOD and SQL models. Using measures of input and program
difficulty, we then explore the relationship between difficulty and
calibration, finding that for SQL programs, the difficulty of the input and
output are associated with poor calibration. Finally, we find that on TOD
datasets, each model’s low-confidence examples are challenging for other
models; this leads us to propose two confidence-based challenge datasets.
Because of calibration’s importance to semantic parsing specifically – and
sequence generation tasks generally – we introduce an open-source library for
computing calibration metrics and plotting confidence, compatible with the
Transformers library (Wolf et al., 2020).

##  2 Related Work

###  2.1 Calibration in NLP

Given the utility of calibrated models in decision-making, a large body of
research has focused on describing the calibration characteristics of
different architectures and models, with some work finding neural networks to
be relatively well-calibrated (Niculescu-Mizil and Caruana, 2005; Minderer et
al., 2021; Carrell et al., 2022) – including neural encoders pre-trained on
text data (Desai and Durrett, 2020) – and other research indicating they are
not (Guo et al., 2017; Wang et al., 2020; Si et al., 2022). These mixed
results preclude drawing general conclusions about neural models’ calibration,
and motivate studies like ours documenting the calibration characteristics of
standard models.

Past work has examined a variety of classification problems, often focusing on
binary or multi-class classification (Naeini et al., 2015; Guo et al., 2017;
Minderer et al., 2021; Khojah et al., 2022). Some papers have addressed
sequential NLP tasks: e.g. Jagannatha and Yu (2020) address calibration in
structured prediction tasks. More related to our sequence generation setting,
Kumar and Sarawagi (2019) and Wang et al. (2020) examine calibration in
machine translation, both finding models to be over-confident. Measuring
calibration in translation tasks is limited by the metrics used, which are
noisy proxies for accuracy and have well-documented limitations (Callison-
Burch et al., 2006; Mathur et al., 2020). In semantic parsing specifically,
past work has focused on improving confidence estimation for certain parsers.
Dong et al. (2018) propose a confidence estimation method based on model and
input features. Similarly, Chen et al. (2022) introduce an additional
confidence-estimation model for semantic parses. In our work, we extract
confidence scores from the same model used for parsing, and focus on analyzing
calibration rather than improving it.

Dataset |  Train |  Dev |  Test |  Example Input |  Example Output
---|---|---|---|---|---
SMCalFlow |  $108,753$ |  $12,271$ |  $13,496$ |  _Do I have anything going on tonight?_ |  (Yield (> (size (QueryEventResponse. results(...))) 0L))
TreeDST |  $121,652$ |  $22,910$ |  $22,841$ |  _I want to book a flight to Paris_ |  (plan (^(Flight)... (Flight.dest...)))
Spider |  $7,794$ |  $865$ |  $1,034$ |  _What are the numbers of all flights coming from Los Angeles?_ |  SELECT flno FROM flight WHERE origin = ‘‘Los Angeles’’
CoSQL |  $6,598$ |  $745$ |  $1,007$ |  _How many people are named Janessa? $|$ Do you mean the number of people whose first name is Janessa? $|$ Yes_ |  SELECT ...AS T1 JOIN ... T2.first_name = ‘‘Janessa’’
Table 1: Number of train, validation, and test examples per dataset and
example inputs and outputs. For SQL tasks, column and table names are also
included in the input (these are omitted here for readability.)

#### Calibration in LLMs

We focus on calibration in pre-trained large language models (LLMs), which has
also been addressed in other lines of work, first by Mielke et al. (2022), who
examine calibration in large pre-trained dialogue models. They focus on
“linguistic calibration,” describing a guided generation method for
introducing verbalized statements of uncertainty (e.g. “I believe”, “I’m not
sure, but…”, “I am certain…”, etc.) into QA responses. Our experiments use
logit-based numerical confidence estimates instead. Similarly, Lin et al.
(2022) analyze both logit-based and verbalized uncertainty in GPT-3. Si et al.
(2023) examine LLM calibration through the lens of reliability, focusing on QA
data and using logit-based confidence estimation. Kadavath et al. (2022)
examine whether LLMs are well-calibrated by measuring the probability an
answer is true. Finally, Zhou et al. (2023) examine how teaching models to
interpret and express certainty and uncertainty impacts calibration and
performance. With the exception of Kadavath et al. (2022), these studies focus
on single-prediction classification settings: Mielke et al. (2022) examine
TriviaQA questions (Joshi et al., 2017), Lin et al. (2022) use math questions,
and both Zhou et al. (2023) and Si et al. (2023) consider multiple QA
benchmarks including TriviaQA. While studying these settings is valuable, our
focus is on longer-form sequence generation, perhaps a more typical use-case
for LLMs.

This divergence is also pointed out by Kadavath et al. (2022), who, in
addition to several experiments on QA benchmarks like MMLU (Hendrycks et al.,
2021), include experiments on HumanEval code generation examples (Chen et al.,
2021) and their own dataset of Python code generation problems. Our
experiments differ from theirs along a few axes. Firstly, although making
claims about the calibration of “language models” broadly, Kadavath et al.
only consider a single pre-trained model (of varying sizes) and a single
program synthesis task; we consider several models across four datasets.
Furthermore, while we consider both fine-tuned and few-shot models, Kadavath
et al. measure calibration only in a few-shot setting. While we obtain
confidence estimates from the token probabilities, Kadavath et al. extract
their estimates via an additional prompt that asks the model to label a
predicted program or answer as “True” or “False”, where the confidence is
taken to be $P(True)$. This is a natural formulation for few-shot models, but
is less compatible with fine-tuned models, which are far more common in
practice. Note that Kadavath et al.’s method incurs roughly twice the cost of
program generation, as the generated program must be re-encoded to obtain a
confidence estimate.

##  3 Methods

Our first goal is to determine the typical calibration characteristics of
models applied to semantic parsing tasks. For this, we choose a range of
standard semantic parsing tasks, datasets and models.

###  3.1 Tasks and Datasets

In total, we examine four datasets across two tasks; statistics and examples
of each are provided in Table 1. All of the datasets we examine are in
English. Two are task-oriented dialogue (TOD) parsing datasets: SMCalFlow
(Semantic Machines et al., 2020) and TreeDST (Cheng et al., 2020).

In the TOD task, a user engages in a dialogue with a digital assistant in
order to achieve some goal, e.g. booking a flight or scheduling a meeting. For
each user turn, the agent predicts an executable program and provides a
response to the user based on the outcome of the program’s execution; thus,
the modeling task is to predict a program from a user input and a dialogue
history. As neither SMCalFlow nor TreeDST has an available execution suite, we
measure correctness by exact match to the reference program. For both TOD
datasets, we use the preprocessed data from Platanios et al. (2021), who
converted TreeDST into a format shared with SMCalFlow that resembles Lisp, and
use the SMCalFlow data splits given by Roy et al. (2022).

Our second task is database querying. As in TOD tasks, the modeling target is
a program; however, the program here is a SQL query that can be executed
against a database (DB) to return an answer. Crucially, this text-to-SQL task
differs from the TOD task in that the programs are highly dependent on the
structure of the execution environment. The DB’s schema influences the program
and changes depending on the DB being used. For the text-to-SQL task, we
consider two popular datasets: Spider (Yu et al., 2018) and CoSQL (Yu et al.,
2019). Both datasets contain queries over a shared set of 200 DBs; however,
while Spider queries are single-turn, CoSQL queries are multi-turn dialogues
between a user and an agent. For SQL, we do have access to an execution
engine, allowing us to measure both execution accuracy and exact-match
accuracy. Here also, we use the splits and preprocessing scripts from Roy et
al. (2022).

###  3.2 Models

Given our goal of benchmarking the calibration characteristics of different
approaches commonly in use in semantic parsing, we measure calibration via two
extant modeling paradigms: fine-tuning and in-context learning (ICL).

#### Fine-tuning

In the fine-tuning paradigm, we take a model pre-trained with a self-
supervised objective on text or code data and continue to train it (i.e. fine-
tune it) using a supervised objective on the training data for each dataset in
Table 1.222Excepting MISO, for which only part of text encoder is pre-trained
and the rest of the model is trained from scratch. We train seven models from
two commonly-used semantic parsing paradigms: transductive and sequence-to-
sequence (seq2seq). Transductive models (Zhang, 2020) treat the parsing
problem as a sequence-to-graph task. While the executable programs found in
SMCalFlow and TreeDST are expressed as Lisp-like sequences, they also have an
underlying execution graph. Rather than learning to generate the surface form,
the transductive approach seeks to directly model the underlying graph,
predicting a sequence of nodes as well as labeled, directed edges. Zhang et
al. (2019a) and Zhang et al. (2019b) introduced the MISO transductive parsing
framework for predicting directed acyclic graphs from text inputs. MISO
combines an encoder-decoder model for predicting nodes paired with a biaffine
parser (Dozat and Manning, 2017) for edge prediction. It features source and
target copy operations allowing special tokens (such as names and numbers) to
be copied from the input and previously generated tokens to be re-generated
(in the case of re-entrancy in the execution graph). This kind of model
represents an “engineering-heavy” approach where inductive bias for parsing is
encoded in the model architecture. We take MISO as an example of this class,
motivated by its application across a variety of semantic parsing tasks (Zhang
et al., 2019a, b; Stengel-Eskin et al., 2020, 2021, 2022; Li et al., 2021). We
use Stengel-Eskin et al. (2022)’s best model, which has a RoBERTa (Liu et al.,
2019) encoder and contains 127 million (M) parameters.

The seq2seq paradigm instead directly models the output sequence. While
predicting the syntactic nuances of a parse (e.g. generating the correct
number of closing parentheses) can be challenging, seq2seq models are better
able to leverage large pre-trained Transformers, often enabling them to
outperform methods with stronger inductive biases, like MISO. We use the
BenchCLAMP framework (Roy et al., 2022) to finetune seq2seq models for all TOD
and SQL datasets; specifically, we examine the T5 (Raffel et al., 2020) and
BART (Lewis et al., 2020) architectures, both of which have frequently been
applied to semantic parsing benchmarks (Shaw et al., 2021; Scholak et al.,
2021; Desai and Aly, 2021; Banerjee et al., 2022). Both are large encoder-
decoder Transformers (Vaswani et al., 2017) pre-trained on text data with
self-supervised objectives. Since SQL is used in many open-source applications
(unlike SMCalFlow and TreeDST), code scraped from the web is likely to contain
examples of it; thus, for Spider and CoSQL, we also examine Code-T5 (Wang et
al., 2021), a T5 architecture pre-trained on large amounts of web-scraped
code. As MISO is not commonly used in text-to-SQL tasks, we choose to omit it
in our SQL experiments. We examine T5-small (60M parameters), T5-base (220M),
T5-large (770M), BART-base (139M), and BART-large (406M), as well as
Code-T5-base (220M). Note that while BenchCLAMP allows for constrained
decoding according to a context-free grammar, restricting the model to
producing only valid parses, we choose to decode in an unconstrained fashion.
Because the constrained decoding process intervenes on the output logit space,
zeroing out invalid continuations and renormalizing, it could affect the
model’s calibration characteristics.

#### In-Context Learning

The ICL paradigm instead uses a model that has _only_ been trained with a
self-supervised objective. Brown et al. (2020) find that a sufficiently large
model pre-trained on text data can perform many tasks after being shown a few
examples, without any updates to the gradients. This has been extensively
explored in the semantic parsing domain, where grammar-constrained decoding
has been combined with ICL, allowing models to predict dataset-specific
programs from only a few retrieved examples (Shin et al., 2021; Shin and Van
Durme, 2022; Roy et al., 2022). Following this paradigm, for a given test
query, we retrieve a set of relevant input-output examples from the training
data and concatenate them into the model’s input, or prompt, as instructive
examples. The model then predicts a parse, constrained by the grammar of the
domain language. Past work in ICL for semantic parsing has used OpenAI’s Codex
model (Chen et al., 2021), a LLM based on the GPT architecture and trained on
web-scraped text and code. However, in addition to being costly, Codex was
recently slated for removal from the OpenAI’s public API. To ensure our
results are not affected by similar decision in the future, we instead opt to
use open-source publicly-released models for our ICL experiments. We use
Codegen (Nijkamp et al., 2022), which is available through the Transformers
library (Wolf et al., 2020). Codegen models are also auto-regressive GPT-style
models pre-trained on code, with strong performance across code generation
tasks; we explore 4 model sizes: 350M parameters, 2B, 6B, and 16B.

For each test example, we construct a prompt by retrieving 5 similar training
examples and concatenating them into the context.333The number of examples was
chosen in light of hardware memory limitations when running the largest
models. Similarly, all billion-parameter models were run at half precision,
and memory constraints precluded running constrained decoding on the 16B
model, so we do not provide sequence-level accuracies for this model. We
follow Roy et al. (2022) and use a BM25 retriever (Robertson and Zaragoza,
2009) over the full training set, indexed by the input utterance. Examples are
ordered by similarity, with the most similar example appearing last (i.e.
immediately before the test input).

#### Input representation

In TOD datasets like SMCalFlow and TreeDST, the user communicates with the
agent over the course of a dialogue; we follow previous work in using the
previous dialogue turn as input if available. Thus, each datapoint consists of
an input $X=(\mathcal{U}_{0},\mathcal{A}_{0},\mathcal{U}_{1})$ and an output
program $\mathcal{P}$, where $\mathcal{U}_{0}$ is the previous user utterance
(if it exists), $\mathcal{A}_{0}$ is an automatically-generated agent response
to the previous utterance, and $\mathcal{U}_{1}$ is the current user
utterance. Similarly, for CoSQL we include the previous dialogue turn in the
input (if available). Correctly predicting a SQL program relies on knowledge
of the DB schema; to inform the model of the schema, we follow past work in
concatenating schema information (column and table names) into the input.

#### Confidence Estimation

The models used (except for MISO) predict subword tokens, rather than the
whitespace-delimited tokens of a programming language. For example, the SQL
token SELECT is split into 3 subwords by the BART tokenizer; in other words,
the model will produce 3 probabilities for this single program token. We
estimate program token confidence by first obtaining estimates for subwords
and then aggregating. When measuring token-level accuracy, we also use program
tokens rather than subwords.

To estimate subword-level confidence, we use the baseline estimator introduced
by Hendrycks and Gimpel (2016), which is robust across many tasks (Varshney et
al., 2022). More specifically, we take the maximum probability across the
output vocabulary at each timestep. We then aggregate the probabilities across
subwords of a given token using min. Intuitively, a prediction is only as good
as its weakest link. Given a sufficiently long sequence of subwords,
aggregation methods like mean may obscure low-confidence decisions. In
practice, using mean pooling on subwords gives qualitatively similar results,
as the number of subwords per whitespace token tends to be small. However, in
Section 4.4 we see that mean leads to high ECE on sequence-level calibration,
indicating that min may be the better overall choice.

Figure 1: Token-level model confidence and mean accuracy, binned by confidence
across models (sorted by size) for TOD datasets. Point size reflects the
number of tokens in the bin. Points above the line reflect overconfidence,
while those below reflect underconfidence. Exact Match accuracy (EM, higher is
better) and Expected Calibration Error (ECE, lower is better) are given. All
models show relatively low ECE.

###  3.3 Metrics

For TOD and SQL datasets, we evaluate our models’ semantic parsing ability
using exact match accuracy (EM), where a prediction is considered correct it
it exactly matches the reference program. This can be quite strict and lead to
false negatives; for example, the snippet x > 0 AND x < 5 is logically
identical to the snippet x < 5 AND x > 0 but would result in an EM score of 0.
In the case of SQL, we could mitigate this by executing programs against the
provided DBs and comparing the result of the execution to the reference
result. This metric is perhaps too lenient, and can result in false positives,
e.g. if the gold program yields a null result, then any program yielding a
null result will be counted as correct, even if it does not correspond at all
to the user’s input. In light of this, Zhong et al. (2020) introduce test-
suite accuracy, which executes each program against a suite of test DBs
optimized for high code coverage. A program is counted as correct if it
matches the gold program’s denotations on all DBs in the suite. This reduces
the false positive rate and provides a much tighter upper-bound on
performance.

We follow past work in using expected calibration error (ECE) (Naeini et al.,
2015; Guo et al., 2017) as our calibration metric. To compute ECE, predictions
are binned by the model’s confidence; since each prediction has an accuracy of
either 0 or 1, accuracies must be averaged across examples falling in a
confidence range to obtain an expected accuracy for a given confidence score.
In its original formulation, the number of bins used is a hyperparameter.
However, Ding et al. (2020) find that a fixed binning approach used in ECE can
be suboptimal. Intuitively, because confidence scores are often distributed
non-uniformly, a fixed binning strategy may result in bins containing few
samples, leading to high-variance accuracy estimates within these bins. In
general, there is a tradeoff between introducing many small bins (estimating
accuracy with high variance) and maintaining a few large bins (estimating
accuracy with high bias). They instead introduce _adaptive binning_ , which
correlates the number of samples in a bin with the bin’s range: in regions
where confidence estimates are sparse, adaptive binning introduces more bins,
such that a smaller range is covered by each bin (reducing bias), while when
estimates are dense, adaptive binning includes fewer bins, reducing the
variance of the estimate. The number of samples for each bin is given by
$n=0.25\big{(}\frac{Z_{\alpha}/2}{\epsilon}\big{)}^{2}$, where $Z_{\alpha/2}$
is the standard normal distribution’s Z-score, $1-\alpha$ is the confidence
interval, and $\epsilon$ is a small positive value included for numerical
stability. 444Like Ding et al. (2020), we found that the metric is not
sensitive changing hyperparameters.

The expected calibration error is the difference between the average accuracy
of each bin and its confidence, weighted by the size of the bin. Let
$\hat{\mathcal{Z}}$ be the model’s distribution over the output vocabulary
$\mathcal{V}$, and let $\hat{C}=\max\hat{\mathcal{Z}}$,
$\hat{Y}=\text{argmax}\>\hat{\mathcal{Z}}$. Let $Y$ be the true class indices
and define a binary accuracy vector $A$ s.t.
$a_{i}=\delta(\hat{y}_{i},y_{i})$. After binning $\hat{Y}$ into $N$ bins
$\mathcal{B}$, $ECE(\mathcal{B})$ is defined as:

$ECE(\mathcal{B})=100*\sum\limits_{i=1}^{N}\frac{|\mathcal{B}_{i}|}{N}\Big{|}\frac{\sum\limits_{j\in\mathcal{B}_{i}}a_{j}}{|\mathcal{B}_{i}|}-\frac{\sum\limits_{j\in\mathcal{B}_{i}}c_{j}}{|\mathcal{B}_{i}|}\Big{|}$
(1)

In other words, ECE is the mean absolute error between each bin’s average
confidence and average accuracy; we scale ECE by a factor of 100 for
readability. In addition to ECE, we qualitatively analyze calibration by
plotting the average accuracy against the bin confidence. To encourage
calibration to be measured in semantic parsing, we release our metric and
plotting library as a Python package.

Figure 2: Token-level calibration curves for few-shot code models tested on
SMCalFlow.

##  4 Benchmarking Calibration

Calibration can be measured at the token-level or the sequence-level.
Sequence-level calibration is most relevant to safety; in a system using an
executable parsing model, the predicted program will be executed as a whole
(i.e. as a sequence). However, absent a (potentially expensive) external
model, sequence-level confidence scores will be composed of token-level
scores, just as sequences are composed of tokens. Moreover, token-level scores
can reveal phenomena obscured by sequence-level scores. Rather than only
measuring the model’s accuracy and confidence on a whole sequence, it may be
more informative to examine on which types of tokens the model makes mistakes
or has high calibration error. For example, token-level confidence scores can
allow us to examine which specific functions the model is struggling on.

To measure token-level accuracy against a reference program at time $t$, all
predicted tokens at timesteps $1,\ldots,t-1$ must match the reference
program’s prefix; thus, we use teacher forcing, feeding the model the gold
prefix up to the current timestep. I.e. when predicting the confidence of
token $\hat{y}_{t}$, we use tokens $y_{1},\ldots,y_{t-1}$ taken from the
_gold_ program $P$, not the predicted program $\hat{P}$. Note that this
differs from the setting used for computing EM accuracy in Fig. 1, Fig. 2, and
Fig. 4, where we compare the gold program $P=y_{1},\ldots,y_{T}$ to the
predicted program $\hat{P}=\hat{y}_{1},\ldots,\hat{y}_{T}$. For sequence-level
confidence, we use $\hat{P}$.

###  4.1 Task-oriented Dialogue Results

Fig. 1 shows the token-level calibration plots and ECE scores for all fine-
tuned models on TOD datasets. Note that in our plots, the model’s confidence
is shown on the y-axis, and the accuracy on the x-axis. While this is non-
standard, it leads to a more natural interpretation of the plot w.r.t. the
line $y=x$: points above the line are overconfident bins (confidence $>$
accuracy); those under the line are underconfident. The models are ranked by
size. The size of each point is based on the log of the number of elements in
that bin (following Mielke et al. (2022)); the largest bin for all models is
the most confident bin. This is consistent with the Exact Match (EM) accuracy
results reported in Fig. 1; for a model to achieve high accuracy, all its
output tokens must exactly match the reference tokens on most programs, i.e.
the vast majority of tokens must be predicted correctly.

Figure 3: Empirical trade-off between accuracy and calibration. Note that the
Y-axis is flipped (lower calibration error is better). Several models exist at
the Pareto front. Figure 4: Token-level model confidence and accuracy for SQL
datasets, binned by confidence across models and datasets. Unlike the models
fine-tuned on TOD data (cf. Fig. 1) models fine-tuned on SQL are poorly-
calibrated.

We note that all models are relatively well-calibrated. For the T5 series,
calibration improves with scale, while the opposite is true for BART. In Fig.
3, we plot the trade-off between accuracy and calibration error for SMCalFlow
models on the test set. Here, we observe a Pareto front, with BART models
having higher accuracy but worse ECE than comparable T5 models. Importantly,
BART’s calibration curves are non-monotonic, which is particularly troubling.
Given monotonic distortions (e.g. a sigmoid calibration curve), regression
models can be fit to the validation set to correct the calibration curve
(Zadrozny and Elkan, 2002). Such correction is much harder to do for non-
monotonic distortions.

###  4.2 Few-shot Calibration

Fig. 2 shows the calibration curves for ICL models with 5 prompt examples on
the SMCalFlow test set. First, we note that the exact-match results here are
obtained with constrained decoding, since without fine-tuning models are
generally unable to produce syntactically-correct programs. However, the
token-level confidence scores are obtained without constraints (as in Section
4.1), so the curves are comparable to those in Fig. 1. Even with constraints,
the exact match performance of these models is lower than that of the fine-
tuned models; however, they are shown only five examples per input, while the
fine-tuned models are trained on over $100,000$ examples. The ECE of the ICL
models is higher than the ECE of finetuned models. While ECE increases
initially with model scale, the ECE of the 16B model is slightly lower than
the ECE of the 6B model, suggesting that the trend could be U-shaped, a
phenomenon that has been documented in other large models (Wei et al., 2022).
Qualitatively and quantitatively, the ICL calibration curves are remarkably
similar to the BART-large curve in Fig. 1, despite the fact that BART-large is
fine-tuned on the training data. We qualitatively analyze the spikes seen in
many BPE-based models like BART and Codegen in Section 5.1, finding that they
are driven by common syntactic tokens.

###  4.3 SQL Results

Fig. 4 shows the token-level calibration characteristics of models fine-tuned
on SQL data. Adaptive binning results in fewer bins here due to the smaller
sizes of the SQL test sets. We see both qualitatively and quantitatively that
many of the same models which were well-calibrated on TOD data are poorly-
calibrated on SQL datasets. All models are substantially over-confident, and
the trends between models here are different than in the TOD setting. BART-
large is better-calibrated than BART-base, which is the worst-calibrated on
both datasets. We also see each model has higher ECE on CoSQL than Spider;
this may have to do with input complexity, since CoSQL is multi-turn. We also
note that the EM scores for SQL are generally quite low; this tracks with past
results finding that EM is an excessively strict metric for SQL (Zhong et al.,
2020). We found the execution accuracies for the models to be in-line with
those reported by Roy et al. (2022)

###  4.4 Sequence-level Calibration

The results in Fig. 1 follow past calibration work in using confidence
estimates at the level of individual classifications (i.e. tokens). However,
unlike many of the NLP tasks where calibration has been explored in the past,
the output in semantic parsing is sequential, i.e. a series of
classifications, where each timestep is dependent on the preceding decisions.
Thus, to obtain sequence-level confidence scores from token-level scores, we
need a method for aggregating token-level estimates. We explore two
aggregation functions: min and mean. These operate over the token-level
confidence scores generated during decoding; note here the decoding takes
place _without_ teacher-forcing, i.e. using the predicted program $\hat{P}$.
Table 2 shows the ECE for sequence-level confidence scores with the two
aggregation methods on the SMCalFlow dataset. Since min is typically better,
especially on BART-large and T5-large, where mean results in high over-
confidence, we adopt min moving forward. As mentioned in Section 3.2, mean may
result in high ECE because over a long sequence, a single low-confidence
prediction may be “washed out” by a sequence of high-confidence predictions.
This is especially true in program prediction tasks, where syntactic
constraints mean that many tokens can be easily predicted with high confidence
(e.g. most SQL programs begin with SELECT, etc.).

Model | ECE (Min.) | ECE (Mean)
---|---|---
MISO | 5.57 | 5.49
BART-large | 6.23 | 16.85
T5-large | 8.29 | 18.01
Table 2: Sequence-level ECE for representative models on SMCalFlow.

##  5 Discussion and Analysis

In Section 4, we found that for a fixed dataset, different models have varying
calibration characteristics, even when the models have similar architectures
and pre-training data. Similarly, we saw that the same models often differ
between datasets, and especially differ between domains (TOD vs. text-to-SQL).
Finally, we found that some models show conspicuous spikes in calibration
error in some regions of the confidence space, and that these spikes hold
across models. In light of these differences and quirks, we conduct additional
analyses on factors associated with model calibration.

###  5.1 Qualitative Analysis

#### Error spikes

In Fig. 1 (for BART-large) and in Fig. 2 for all models, we see a spike in
calibration error in the higher-confidence region. This spike generally
becomes more pronounced as the size of the model increases. Given that these
models all share a tokenizer and that the spikes are located in similar
regions of the confidence space, we hypothesize that they might be related to
the identity of the tokens in the bins. We isolate predictions in bins within
each spike (i.e. bins with high confidence and lower accuracy, where the curve
deviates from the $y=x$ line.) We find that many of the tokens are shared
between models and datasets; while each bin for each dataset does contain
“content” tokens (e.g. function names, value names), all bins for all models
contain a large number of very frequent “syntactic” tokens (e.g. (,),=,>,
etc.) shared by SMCalFlow and TreeDST. Given how common these tokens –
especially parentheses – are in the Lisp datasets, it is likely that the error
spikes are driven by overconfidence on these syntactic tokens.

#### Common errors

We find that most models are over-confident; we can see over-confidence either
as a failure of confidence estimation (i.e. the model predicts a higher
confidence value than is appropriate) or a failure of prediction (i.e. the
model makes mistakes that lead to low accuracy). Following the second
interpretation, we examine some common errors that models make on text-to-SQL
and TOD datasets. For SQL datasets, we find that higher-confidence programs
($>0.5$) sometimes are semantically correct, but fail according to exact match
because of a syntactic difference. For example, some programs use single
quotes rather than double quotes; while this does not affect the execution
result, it counts as a failure for exact-match accuracy. We discuss the
shortcomings of exact-match further in Section 5.4. In SMCalFlow, many high-
confidence errors are due to mismatches in capitalization, and others are due
to confusion between similar functions, e.g. substituting
AttendeeListHasRecipientConstraint for AttendeeListHasRecipient.

We also see some errors due to ambiguous examples, where the model predicts a
valid interpretation of the input which deviates from the reference. For
example, given the input _“List the number of different series names and
contents in the TV Channel table.”_ , the BART-large text-to-SQL model trained
on Spider predicts the program:

SELECT count ( DISTINCT series_name ) , content FROM tv_channel

which corresponds to the following valid parse of the input: _“[the number of
different series names] and [contents]…”_. The reference however is:

SELECT count ( DISTINCT series_name ) , count ( DISTINCT content ) FROM
tv_channel;

which corresponds to _“[the number of different [series names and
contents]]…"_.

###  5.2 Data size

Given the dramatic differences in calibration between Fig. 1 and Fig. 4, a
natural question to ask is _why_ SQL models are so much less calibrated. One
simple hypothesis is dataset size: Table 1 shows that the SQL datasets have
between 14x and 18x _less_ data available for fine-tuning than the TOD
datasets. Could simply increasing the size of the dataset fix the poor SQL
calibration seen in Fig. 4?

If calibration were a result primarily of the dataset size, we would expect an
SMCalFlow model trained on a small amount of data to also have high ECE. To
test this, we trained a T5-small model on the first $7,794$ examples in the
SMCalFlow dataset – the same exact number of training examples as are
available for Spider. We find that the model’s ECE increases from 1.27 to
2.40. While the ECE does increase, it is still far lower than the ECE on
Spider (7.64), indicating that low dataset size is not the primary driver of
poor calibration on text-to-SQL tasks.

###  5.3 Input and Output Difficulty

Having established that the difference in calibration between TOD and text-to-
SQL models cannot be accounted for merely by dataset size, we explore other
factors that could be associated with poor calibration. Specifically, we
examine the difficulty of the input and output. For the input, we examine out-
of-distribution (OOD) inputs, following past observations that models are
typically over-confident on OOD inputs (Bui and Liu, 2023). We measure how OOD
an input is via perplexity from an LSTM LM (Hochreiter and Schmidhuber, 1997).
To exclude the possibility of data leakage, we train this model from scratch,
without any pre-trained word embeddings. Inputs are tokenized using BPE
(Sennrich et al., 2016) and the train split is used to build a vocabulary. We
autoregressively train 2-layer unidirectional 256-dimensional LSTMs on the
training splits of SMCalFlow and Spider, using an Adam optimizer (Kingma and
Ba, 2015) with a learning rate of $0.001$.555LSTMs were chosen over
Transformers here because of the relatively small size of the data combined
with the desire to train the model from scratch. Training ends when the
validation perplexity fails to decrease for 5 consecutive epochs.

We consider sequence-level confidence bins from models shared between
SMCalFlow and Spider (this excludes MISO and Code-T5). We compute the average
perplexity of the inputs in each bin and plot the confidence and accuracy
against the mean perplexity in Fig. 5. In all cases, we expect to see accuracy
decreasing with perplexity: intuitively, as inputs become more OOD, they
become harder for the model to parse. If models are robust to OOD inputs, the
confidence should also decrease with perplexity, i.e. the model should be
“aware” of the fact that it is worse at predicting programs for OOD inputs. On
SMCalFlow, accuracy is correlated with input perplexity, decreasing on more
OOD inputs. Confidence decreases with perplexity at a similar rate, indicating
that the model has learned to identify OOD inputs and produce lower confidence
values accordingly. That the slopes of the two lines are very close aligns
with the low ECE seen on SMCalFlow across models: accuracy, confidence, and
input perplexity are all “coupled”.

Figure 5: Confidence and accuracy (y-axis) plotted by perplexity of the input
(x-axis).

For Spider, on the other hand, there is no such coupling. While accuracy is
weakly correlated with perplexity, confidence is _positively_ correlated with
perplexity. In other words, as inputs become more OOD, the model becomes
_more_ over-confident in its predictions.666Note that for Spider, the two
outlier points with high average perplexity ($>10$) contain many tokens not
seen in training, like names of countries. Removing these outliers does not
have a qualitative impact on the results. This suggests that the SQL models
may not have learned to recognize OOD inputs.

We also consider output difficulty in the SQL domain, where use the difficulty
labels provided by Yu et al. (2018), who classify SQL programs into _easy_ ,
_medium_ , _hard_ , and _extra-hard_ depending on the types of functions they
involve. We use T5-large, which is the best-calibrated (and best-performing)
model in Fig. 4, and compute the sequence-level ECE for programs separated out
by difficulty-type. We do note here that, due to the relatively small number
of programs in the SQL test set, the bins here are often sparse, introducing
variance into the estimates. To account for this, we train and evaluate models
using 3 random seeds and plot the average performance. These results are
described in Fig. 6, where ECE generally increases with difficulty. The extra-
hard programs have roughly twice the ECE of easy programs. Notably, accuracy
decreases as programs get harder; the high ECE values for harder programs
indicates that the model does not lower the confidence appropriately for these
programs.

Figure 6: T5-large ECE by target program difficulty on Spider (as defined by
Yu et al. (2018))

###  5.4 Execution Accuracy

Because SQL is executable, we can additionally examine whether models are
well-calibrated with respect to execution accuracy. This is especially
relevant in light of the results in Fig. 4, indicating that models are over-
confident. Execution accuracy is typically more lenient than exact match
accuracy, which is prone to _false negatives_ ; a program may vary
syntactically from a reference but still execute to the same result, i.e. have
the same denotation.

Another more lenient form of accuracy is accuracy@$k$ (Acc@$k$, which measures
whether the correct program is in the top $k$ programs returned by the model
after beam search. Intuitively, if there are two equally valid programs for a
given input, we would expect Acc@$2$ to capture both, while Acc@$1$ (i.e.
standard EM accuracy) would only have a $50\%$ chance of being correct. While
Acc@$k$ is more lenient, it is not a realistic metric, since in practice we
always need to choose one single program to execute.

Figure 7: Execution accuracy and exact match accuracy@$k$. All accuracies
result in over-confidence, but execution accuracy is generally the best-
calibrated.

Fig. 7 shows Acc@$k$ and the execution accuracy for T5-base on the Spider test
set. The dotted line connects Acc@$1$ to execution accuracy, showing the
range. First, we note that while all bins are over-confident, the execution
accuracy is generally better-calibrated (less over-confident) than any of the
Acc@$k$ values. Note that the model’s training signal only provides indirect
information about execution accuracy, since the loss is computed against a
single reference program (which ostensibly would execute correctly), not
against a set of equivalent programs. These results are promising: while the
T5-base model is very poorly-calibrated according to exact match, it is not
nearly as bad when considering test-suite execution accuracy, which has been
demonstrated to be the more informative metric (Zhong et al., 2020).

###  5.5 Easy and Hard TOD Subsets

Fig. 1 shows that for many of the TOD models we examine, token-level
confidence is well-correlated with accuracy; in Table 2, using min to
aggregate token-level confidence scores, we can obtain relatively low ECE at
the sequence-level as well, meaning we can predict (on average) how likely a
test example is to be correct. The ECE values in Table 2 tell us that the
lower a model’s min confidence is, the more likely it is to make a mistake.
Since different models have qualitatively different calibration curves, and
may have complementary errors, we use an ensemble of three different models
(MISO, T5-large, BART-large) to extract easy and hard splits of high- and low-
confidence programs. These splits follow in the spirit of adversarial
filtering (Zellers et al., 2018) but are based on confidence rather than
accuracy.

The hard subset contains the union of examples for which the _sequence_ -level
confidence falls below a threshold – all other examples are in easy. The
threshold for each dataset is chosen by computing the $25^{th}$ percentile
over all of the sequence-level probabilities across the three models. This
threshold is $0.86$ for SMCalFlow and $0.85$ for TreeDST. The union is taken
across the three models, i.e. if any one model assigns an example a confidence
below the threshold, the example is considered hard.

Table 3 shows the percentage of test data below the threshold for each model.
Note that, because the models do not all assign low confidence to the same
examples, the union of examples exceeds $25\%$ of the data. Similarly, because
the threshold is computed using the aggregated data from across all 3 models,
it is possible for a single model to have more than $25\%$ hard examples, as
long as the average across all 3 models is $25\%$. For SMCalFlow, MISO
contributes the most hard examples, while for TreeDST, BART-large contributes
more.

Model |  hard (SM) |  hard (Tr)
---|---|---
MISO | $33.31\%$ | $18.58\%$
BART-large | $18.54\%$ | $44.50\%$
T5-large | $23.15\%$ | $11.93\%$
Union | $40.38\%$ | $50.50\%$
Table 3: Percentage of test examples labeled as hard by each model for
SMCalFlow (SM) and TreeDST (Tr).

Table 4 shows the accuracy of each model on our subsets. We see much lower
performance across all models on the hard subset and much higher performance
on easy. MISO’s performance is lower than that of the other models; the
difference is larger than the performance difference in Fig. 1. This is partly
due to MISO contributing a large percentage of low-confidence examples to hard
(cf. Table 3) – low-confidence examples are often more likely to be
misclassified (Hendrycks and Gimpel, 2016). We release our hard and easy
subsets for both SMCalFlow and TreeDST, to act as challenge datasets for
future work.

Dataset | Model | hard | easy
---|---|---|---
SMCalFlow | MISO | 53.43 | 96.05
BART-L | 62.66 | 96.15
T5-L | 60.28 | 96.25
TreeDST | MISO | 80.32 | 94.42
BART-L | 84.97 | 98.67
T5-L | 84.10 | 98.61
Table 4: Exact match accuracy on the easy and hard subsets for all models. L
indicates “large” variant. All models perform significantly worse on hard.

###  5.6 Limitations

Our study is limited by the models, datasets, and languages we consider.
Firstly, we examine only English datasets, limiting the impact of our results;
future work may examine calibration across typologically diverse languages.
Additionally, although we consider multiple datasets and models, our datasets
are drawn from two domains and programming paradigms, and our models are
limited to Transformer-based architectures. While these choices are
representative of current standards in executable semantic parsing, we
encourage broader investigations spanning additional models and datasets.
Specifically, future work may address programs that are interleaved into text-
based interactions. As sequence-models become integrated into chat-bot
interfaces, increasing attention has been paid to augmenting these models with
“tools”, i.e. the ability to call APIs by predicting short programs (Schick et
al., 2023; Mialon et al., 2023). Examining calibration in these settings is
particularly relevant. We examine only the highest-resource settings and leave
examining how calibration profiles change with dataset size to future work.

We are also limited in how we measure calibration. ECE can be a brittle metric
(Ovadia et al., 2019; Si et al., 2022); we have tried to mitigate this by
using adaptive binning. However, there are still hyperparameters and design
choices involved in measuring ECE, and raw ECE scores can obscure a model’s
true calibration characteristics. We attempt to balance this using qualitative
assessments. Similarly, while accuracy is easier to measure in semantic
parsing than other text generation tasks, we have noted some of the
shortcomings of exact-match accuracy metrics.

##  6 Conclusion

Broadly, our results show that calibration is a complex phenomenon with a
multitude of influences. On a single dataset, different models vary in their
calibration error; moreover, the same model often varies drastically between
different datasets, indicating that calibration is a function of both the
model and the dataset. Taken together, these results point to the complexity
of measuring calibration, and suggest that care should be taken when making
claims about classes of models based on evidence from a limited number of
models or datasets. Given the utility of calibrated models and the importance
of calibration to safety, we advocate for considering calibration in the
standard evaluation suite for semantic parsing models, and release our metric
and visualization suite as a standalone package to facilitate these
comparisons.

##  7 Acknowledgements

We would like to thank Zhengping Jiang, Anthony Platanios, Chenglei Si, Subhro
Roy, Kate Sanders, Yu Su, Dan Klein, Matt Gardner, Anqi Liu, and Daniel
Khashabi for their feedback and pointers. We also thank the TACL reviewers and
the Action Editor, who provided valuable feedback. Elias Stengel-Eskin is
supported by an NSF GRFP, and this work was additionally supported by NSF
$\\#1749025$.

## References

  * Banerjee et al. (2022) Debayan Banerjee, Pranav Ajit Nair, Jivat Neet Kaur, Ricardo Usbeck, and Chris Biemann. 2022.  Modern baselines for sparql semantic parsing.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2260–2265.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bui and Liu (2023) Ha Manh Bui and Anqi Liu. 2023.  Density-softmax: Scalable and distance-aware uncertainty estimation under distribution shifts.  _arXiv preprint arXiv:2302.06495_.
  * Callison-Burch et al. (2006) Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006.  Re-evaluating the role of bleu in machine translation research.  In _11th conference of the european chapter of the association for computational linguistics_ , pages 249–256.
  * Carrell et al. (2022) Annabelle Carrell, Neil Mallinar, James Lucas, and Preetum Nakkiran. 2022.  The calibration generalization gap.  _arXiv preprint arXiv:2210.01964_.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chen et al. (2022) Shijie Chen, Ziru Chen, Huan Sun, and Yu Su. 2022.  Error detection for interactive text-to-sql semantic parsing.  In _Proceedings of the Second Workshop on Interactive Learning for Natural Language Processing_.
  * Cheng et al. (2020) Jianpeng Cheng, Devang Agrawal, Héctor Martínez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, Jason D. Williams, Hong Yu, Diarmuid Ó Séaghdha, and Anders Johannsen. 2020.  Conversational semantic parsing for dialog state tracking.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8107–8117, Online. Association for Computational Linguistics.
  * Desai and Aly (2021) Shrey Desai and Ahmed Aly. 2021.  Diagnosing transformers in task-oriented semantic parsing.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 57–62.
  * Desai and Durrett (2020) Shrey Desai and Greg Durrett. 2020.  Calibration of pre-trained transformers.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 295–302.
  * Ding et al. (2020) Yukun Ding, Jinglan Liu, Jinjun Xiong, and Yiyu Shi. 2020.  Revisiting the evaluation of uncertainty estimation and its application to explore model complexity-uncertainty trade-off.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 4–5.
  * Dong et al. (2018) Li Dong, Chris Quirk, and Mirella Lapata. 2018.  Confidence modeling for neural semantic parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 743–753.
  * Dozat and Manning (2017) Timothy Dozat and Christopher D. Manning. 2017.  Deep biaffine attention for neural dependency parsing.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR.
  * Gupta et al. (2018) Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018.  Semantic parsing for task oriented dialog using hierarchical representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2787–2792.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Jagannatha and Yu (2020) Abhyuday Jagannatha and Hong Yu. 2020.  Calibrating structured output predictors for natural language processing.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2020, page 2078. NIH Public Access.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , Vancouver, Canada. Association for Computational Linguistics.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_.
  * Khojah et al. (2022) Ranim Khojah, Alexander Berman, and Staffan Larsson. 2022.  Evaluating n-best calibration of natural language understanding for dialogue systems.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 582–594.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kumar and Sarawagi (2019) Aviral Kumar and Sunita Sarawagi. 2019.  Calibration of encoder decoder models for neural machine translation.  _arXiv preprint arXiv:1903.00802_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880.
  * Li et al. (2021) Zhuowan Li, Elias Stengel-Eskin, Yixiao Zhang, Cihang Xie, Quan Tran, Benjamin Van Durme, and Alan Yuille. 2021.  Calibrating concepts and operations: Towards symbolic reasoning on real images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Teaching models to express their uncertainty in words.  _Transactions on Machine Learning Research_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mathur et al. (2020) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020.  Tangled up in bleu: Reevaluating the evaluation of automatic machine translation evaluation metrics.  _arXiv preprint arXiv:2006.06264_.
  * Mialon et al. (2023) Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023.  Augmented language models: a survey.  _arXiv preprint arXiv:2302.07842_.
  * Mielke et al. (2022) Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.  Reducing conversational agents’ overconfidence through linguistic calibration.  _Transactions of the Association for Computational Linguistics_ , 10:857–872.
  * Minderer et al. (2021) Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. 2021.  Revisiting the calibration of modern neural networks.  _Advances in Neural Information Processing Systems_ , 34:15682–15694.
  * Naeini et al. (2015) Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015.  Obtaining well calibrated probabilities using bayesian binning.  In _Twenty-Ninth AAAI Conference on Artificial Intelligence_.
  * Niculescu-Mizil and Caruana (2005) Alexandru Niculescu-Mizil and Rich Caruana. 2005.  Predicting good probabilities with supervised learning.  In _Proceedings of the 22nd international conference on Machine learning_ , pages 625–632.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022.  Codegen: An open large language model for code with multi-turn program synthesis.  _arXiv preprint arXiv:2203.13474_.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  _Advances in neural information processing systems_ , 32.
  * Platanios et al. (2021) Emmanouil Antonios Platanios, Adam Pauls, Subhro Roy, Yuchen Zhang, Alexander Kyte, Alan Guo, Sam Thomson, Jayant Krishnamurthy, Jason Wolfe, Jacob Andreas, and Dan Klein. 2021.  Value-agnostic conversational semantic parsing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3666–3681, Online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Roy et al. (2022) Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, and Benjamin Van Durme. 2022.  Benchclamp: A benchmark for evaluating language models on semantic parsing.  _arXiv preprint arXiv:2206.10668_.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_.
  * Scholak et al. (2021) Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021.  PICARD: Parsing incrementally for constrained auto-regressive decoding from language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9895–9901. Association for Computational Linguistics.
  * Semantic Machines et al. (2020) Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020.  Task-oriented dialogue as dataflow synthesis.  _Transactions of the Association for Computational Linguistics_ , 8:556–571.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Shaw et al. (2021) Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021.  Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 922–938, Online. Association for Computational Linguistics.
  * Shin et al. (2021) Richard Shin, Christopher Lin, Sam Thomson, Charles Chen Jr, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021.  Constrained language models yield few-shot semantic parsers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7699–7715.
  * Shin and Van Durme (2022) Richard Shin and Benjamin Van Durme. 2022.  Few-shot semantic parsing with language models trained on code.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5417–5425.
  * Si et al. (2023) Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2023.  Prompting GPT-3 to be reliable.  _International Conference on Learning Representations_.
  * Si et al. (2022) Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd-Graber. 2022.  Re-examining calibration: The case of question answering.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2814–2829, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Stengel-Eskin et al. (2021) Elias Stengel-Eskin, Kenton Murray, Sheng Zhang, Aaron Steven White, and Benjamin Van Durme. 2021.  Joint universal syntactic and semantic parsing.  _Transactions of the Association for Computational Linguistics_.
  * Stengel-Eskin et al. (2022) Elias Stengel-Eskin, Emmanouil Antonios Platanios, Adam Pauls, Sam Thomson, Hao Fang, Benjamin Van Durme, Jason Eisner, and Yu Su. 2022.  When more data hurts: A troubling quirk in developing broad-coverage natural language understanding systems.  _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Stengel-Eskin et al. (2020) Elias Stengel-Eskin, Aaron Steven White, Sheng Zhang, and Benjamin Van Durme. 2020\.  Universal decompositional semantic parsing.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8427–8439.
  * Tellex et al. (2020) Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. 2020.  Robots that use language.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 3:25–55.
  * Varshney et al. (2022) Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022.  Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1995–2002.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wang et al. (2020) Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. 2020.  On the inference calibration of neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3070–3079.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8696–8708.
  * Wei et al. (2022) Jason Wei, Najoung Kim, Yi Tay, and Quoc V Le. 2022.  Inverse scaling can become u-shaped.  _arXiv preprint arXiv:2211.02011_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , pages 38–45.
  * Yu et al. (2019) Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019.  Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1962–1979.
  * Yu et al. (2018) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. 2018.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3911–3921.
  * Zadrozny and Elkan (2002) Bianca Zadrozny and Charles Elkan. 2002.  Transforming classifier scores into accurate multiclass probability estimates.  In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 694–699.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Zhang (2020) Sheng Zhang. 2020.  _Transductive Semantic Parsing_.  Ph.D. thesis, The Johns Hopkins University.
  * Zhang et al. (2019a) Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019a.  AMR parsing as sequence-to-graph transduction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 80–94, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2019b) Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. 2019b.  Broad-coverage semantic parsing as transduction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3786–3798, Hong Kong, China. Association for Computational Linguistics.
  * Zhong et al. (2020) Ruiqi Zhong, Tao Yu, and Dan Klein. 2020.  Semantic evaluation for text-to-sql with distilled test suites.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 396–411.
  * Zhou et al. (2023) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023.  Navigating the grey area: Expressions of overconfidence and uncertainty in language models.  _arXiv preprint arXiv:2302.13439_.