# Quantitative Imaging and Automated Fuel Pin Identification for Passive Gamma
Emission Tomography

Ming Fang  Department of Nuclear, Plasma, and Radiological Engineering,
University of Illinois at Urbana-Champaign, Urbana, IL 61801, United States
<EMAIL_ADDRESS>Yoann Altmann  School of Engineering and Physical
Sciences, Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS, United
Kingdom  Daniele Della Latta  Fondazione Toscana Gabriele Monasterio, Via
Giuseppe Moruzzi, 1, 56124 Pisa PI, Italy (now with Terarecon Inc., 4309
Emperor Blvd, Suite 310 Durham, NC 27703, United States)  Massimiliano
Salvatori  Fondazione Toscana Gabriele Monasterio, Via Giuseppe Moruzzi, 1,
56124 Pisa PI, Italy (now with Kymamed Srls, Viale Roma 208 54100 Massa Italy)
Angela Di Fulvio  Department of Nuclear, Plasma, and Radiological Engineering,
University of Illinois at Urbana-Champaign, Urbana, IL 61801, United States

###### Abstract

Compliance of member States to the Treaty on the Non-Proliferation of Nuclear
Weapons is monitored through nuclear safeguards. The Passive Gamma Emission
Tomography (PGET) system is a novel instrument developed within the framework
of the International Atomic Energy Agency (IAEA) project JNT 1510, which
included the European Commission, Finland, Hungary and Sweden. The PGET is
used for the verification of spent nuclear fuel stored in water pools.
Advanced image reconstruction techniques are crucial for obtaining high-
quality cross-sectional images of the spent-fuel bundle to allow inspectors of
the IAEA to monitor nuclear material and promptly identify its diversion. In
this work, we have developed a software suite to accurately reconstruct the
spent-fuel cross sectional image, automatically identify present fuel rods,
and estimate their activity. Unique image reconstruction challenges are posed
by the measurement of spent fuel, due to its high activity and the self-
attenuation.While the former is mitigated by detector physical collimation, we
implemented a linear forward model to model the detector responses to the fuel
rods inside the PGET, to account for the latter. The image reconstruction is
performed by solving a regularized linear inverse problem using the fast-
iterative shrinkage-thresholding algorithm (FISTA). We have also implemented
the traditional filtered back projection (FBP) method based on the inverse
Radon transform for comparison and applied both methods to reconstruct images
of simulated mockup fuel assemblies. Higher image resolution and fewer
reconstruction artifacts were obtained with the inverse-problem approach, with
the mean-square-error (MSE) reduced by 50%, and the structural-similarity
(SSIM) improved by 200%. We then used a convolutional neural network (CNN) to
automatically identify the bundle type and extract the pin locations from the
images; the estimated activity levels finally being compared with the ground
truth. The proposed computational methods accurately estimated the activity
levels of the present pins, with an associated uncertainty of approximately
5%.

## Introduction

Under the Treaty on the Non-Proliferation of Nuclear Weapons (NPT) [1] and
other treaties against nuclear proliferation, the International Atomic Energy
Agency (IAEA) is entrusted to verify all nuclear materials under the control
of the State with safeguards agreements in force with the Agency.
International safeguards include all the technical measures put in place by
the IAEA to verify that each State Parties complies with the aforementioned
agreements.

Most safeguards methods are based on the assay of nuclear materials through
measurements of ionizing radiation emitted by the sample. Safeguards systems
are undergoing an important modernization effort [2]. One example of such
effort is the development of the PGET systems for the inspection of spent fuel
in water pools [3, 4, 5]. Traditionally, an inspection of spent fuel in pools
was performed using the FORK detector, which encompasses, in its standard
version, one ionization chamber to measure gamma rays and two neutron-
sensitive fission chambers. The FORK detector, however, exhibits low
sensitivity to the diversion of fuel pins, being able to detect the diversion
of 50% or more of the fuel pins in the assembly under inspection [6]. A PGET
was recently developed to increase the sensitivity to a variety of fuel
diversion scenarios [5]. The software to analyze PGET images can be improved
[7] to reduce user intervention and overall perform faster and more reliable
monitoring of spent fuel and therefore implement the ultimate safeguards
objective, i.e., the prompt detection of nuclear material diverted from
peaceful uses.

The PGET of spent fuel poses some unique challenges, compared to industrial or
medical tomographic imaging, which are due to the high activity of the sources
(a single-pin activity is of the order of $10^{13}$ Bq) and the high self-
attenuation of the fuel pins. While the former can be mitigated using
collimated detectors, the latter needs to be addressed by using specific
imaging algorithms. Previous work has shown that the reconstruction quality
could be significantly improved by applying the attenuation correction [8]. In
this work, we further corrected for the gamma ray down-scattering in the
energy window of interest. We have implemented and integrated computational
methods based on proximal gradient, physics-informed Monte Carlo sampling, and
machine learning to (1) improve the PGET image quality, (2) automatically
identify missing pins, and (3) quantify the pin activities. As we will show,
the automated identification of missing pins relies heavily on the image
quality, and therefore the three tasks are inherently intertwined.

## Results

### Simulated Sinograms

We simulated the measurement of six fuel pin configurations in a mock-up
water-water energetic reactor (VVER) fuel assembly using MCNP 6.2 [9], as
shown in Fig. 1. In Cases 2 and 3, we mimicked the scenario where fuel pins
were missing. In Cases 4 and 5, we used depleted uranium to replace cobalt
because it resulted in the highest attenuation among different potential
replacement materials [10], which would allow us to examine the algorithm’s
capability of distinguishing replaced pins. In Case 6, we wanted to explore
whether the algorithm is able to render an accurate image of a space-dependent
activity distribution. Details of the simulated PGET model are included in
“Methods-Simulation Methods” section.

Figure 1: Actual pin distributions in the simulated fuel assemblies (ground
truth). Pin activity is proportional to the brightness. In Case 1, the mock-up
fuel assembly hosted 331 60Co pins of 8.879 g/cm3 density, which emitted 1.17
and 1.33 MeV gamma rays. In Cases 2 and 3, 10% 60Co pins pins were removed at
the center or uniformly. In Cases 4 and 5, 10% 60Co pins pins were replaced by
depleted uranium pins of the same size but higher density (10.4 g/cm3). In
Case 6, we decreased the activities of the middle pins by 80% and bottom pins
by 50%.

The detection system we simulated is the so-called PGET prototype device as
described by [4]. The detection unit consisted of two collimated CdTe detector
arrays on opposite sides of the fuel assembly, each encompassing 91 detectors.
The detector arrays rotated and scanned the fuel bundle in steps of
1$\degree$, which generated a $182\times 360$ sinogram, as shown in Fig. 2. We
simulated $5\times 10^{9}$ NPS (number of particle histories) in each case and
detected the photons in the 700-1500 keV energy window. Approximately 15,000
maximum counts in one pixel were achieved, comparable to the counts in an
actual PGET measurement.

Figure 2: Simulated sinogram of the fully-loaded bundle. The color scale
represents the detected photon counts.The periodicity of the singogram
resulted from the hexagonal symmetry of the simulated fuel assembly.

### Inverse Approach

We have developed a systematic approach that allows us to reconstruct high-
quality images from individual sinograms, identify the pin locations, and
quantify the pin activity levels. The image of fuel pins was reconstructed by
solving a linear inverse problem. The region of interest was divided into
$182\times 182$ pixels. We calculated the detector array response to a source
of unit activity inside each pixel, forming the system response matrix. For an
unknown fuel assembly, the measured sinogram is a linear combination of the
calculated responses, with the coefficients being the source strength of each
pixel. In this way, the image reconstruction problem was converted into a
linear inverse problem, with both the simulated sinogram and calculated
response matrix as inputs.

First, we performed a simple image reconstruction to obtain an initial
estimate of pin locations and pin radius. We calculated the system response
matrix using a deterministic ray-tracing method, assuming that the scattering
of gamma rays can be neglected [11]. The reconstructed image of Case 1 is
shown in Fig. 3a. Inner pixels are brighter than the outer ones on the image,
indicating an overestimation of the activity of the pins. This is because the
contribution of scattered photons to the system response cannot be neglected
for inner pixels, due to the heavy attenuation of unscattered photons.
Nevertheless, using this image, we can determine the center of all possible
pins as the centroids of the bright regions on the reconstructed image. In
this step, we would allow some slackness in pin identification, since a more
accurate reconstruction will be performed later. The identified pins are shown
in Fig. 3b. Horizontal and vertical line profiles passing through the center
of each pin were extracted from the image, and we fitted a Gaussian to the
average of all profiles, shown in Fig 4. The FWHM (full width at half maximum)
of the fitted Gaussian was 0.769 cm, which was taken to be the pin diameter.

Figure 3: Image reconstruction and pin identification using the simple
response matrix in Case 1. Figure 4: Gaussian fit of the average line profile.
FWHM of the fitted Gaussian is shown as the arrow. The 11 pixels around the
pin center shown as the red points were used in the fitting. The coefficient
of determination $R^{2}$ of the fit was 0.9991.

With the pin locations and pin radius known, we could create a material map of
the fuel assembly. We then implemented an accelerated Monte-Carlo algorithm to
calculate the system response matrix, accounting for both the absorption and
scattering interaction of photons in the fuel assembly. The images
reconstructed using the new response matrix are shown in Fig. 5. The images
were then fed to a convolutional neural network to perform pin identification.
The results are shown in Fig. 6. For Cases 1 to 5, we achieved 100%
classification accuracy, due to the high quality of reconstructed images. We
created a histogram of pin activities, as shown in Fig. 7. In Cases 1-5, the
standard deviation of activity estimation ranged from 3% to 6%, and the mean
of the pin activity distribution deviated from the ground truth by less than
4%. In Case 6, we successfully identified all pins, except those with the
lowest activity level (Group 1) shown in Fig. 6f and Fig. 7f. The activity of
pins in Group 2 and Group 3 was accurately reproduced with a relative error
below 1%. The inverse approach is detailed in the “Methods-Image
Reconstruction Methods” section.

Figure 5: Image reconstructed using the inverse approach. Good contrast
between the pin-present region and pin-absent region was achieved. No severe
artifacts were observed on the reconstructed images. Figure 6: Pin
identifications based on the inverse approach. Identified pins are shown as
red circles. For Cases 1 to 5, we achieved 100% classification accuracy; for
Case 6, we achieved 100% classification accuracy for the pins of medium and
high activity level.

## Discussion

The image reconstruction problem can be formulated into different linear
inverse problems based on the selected model of observation noise. As detailed
in “Inverse Problem Approach”, we formulated two inverse problems with the
Gaussian noise model and Poisson model, which can be solved using FISTA (fast-
iterative shrinkage-thresholding algorithm) [12] and PIDAL (Poisson image
deconvolution by augmented Lagrangian) algorithm [13], respectively. We
applied both FISTA and PIDAL to perform reconstruction in Case 1 and compared
them in Fig. 8. We assessed the image quality quantitatively using the mean-
square-error (MSE) and structural similarity (SSIM) [14]. MSE is the mean-
squared-difference between the reconstructed image and the ground truth, and
SSIM measures the similarity between the reconstructed image and the ground
truth. Lower MSE and higher SSIM mean better reconstruction. As shown in Table
1, the image reconstructed with FISTA resulted in a lower MSE and higher SSIM,
compared to PIDAL, and PIDAL took approximately 20% longer to run than FISTA.
Therefore, we used FISTA as the main image reconstruction algorithm.

Figure 7: Activity distribution based on the inverse approach. The pin
activity was estimated from each image by summing up the pixel values inside
the present pins. The ground truth is shown as the red line. Figure 8:
Comparison of FISTA and PIDAL reconstruction to the ground truth. The two
methods led to visually similar results, though the PIDAL reconstruction
resulted in a sparser image, compared to FISTA. Table 1: Comparison of MSE,
SSIM, and computation time using FISTA and PIDAL. The computation time was
measured on an Intel Core i9-7920X CPU.

Method | MSE | SSIM | Computation time (s)
---|---|---|---
PIDAL | 42.05 | 0.52 | 231.68
FISTA | 41.45 | 0.59 | 183.57

Figure 9: Image reconstructed using FBP. Severe image blurring occurred at the
center of Case1, Case 3 and Case 5. Figure 10: Pin identifications based on
FBP. Identified pins are shown as red circles. Mis-classification of fuel pins
appeared mostly at the center of Case 1, Case 3, and Case 5, where the
hexagonal symmetry of the pin distribution was broken. Figure 11: Activity
distribution based on FBP. The pin activity was estimated by multiplying a
normalization constant to the sum of all pixels inside the pin.The ground
truth is shown as the red line.

For comparison, we have also implemented the traditional FBP method to
reconstruct the image, which is detailed in “Methods-Image Reconstruction
Methods” section. We then input these images to the neural network for pin
identification, and estimated the pin activity levels. We compared the
performance of the inverse (FISTA) and FBP approaches in terms of image
quality, accuracy of pin identification and activity quantification. Fig. 9
shows the images reconstructed using FBP. Compared to Fig. 9, images in Fig. 5
demonstrated higher contrast and fewer reconstruction artifacts. The blurring
at the center was significantly removed using the inverse approach. We
calculated the MSE and SSIM for both sets of images, shown in Table 2. We
achieved significantly lower MSE and higher SSIM by using the inverse
approach, which resulted in lower mis-classification rates of pin
identification overall. The pin identification results based on FBP
reconstruction are shown in Fig. 10. In Fig. 10a, Fig. 10c, and Fig. 10e, the
blurring led to inaccurate pin localization around the center of FBP-
reconstructed images, and the hexagonal structure was destroyed. In contrast,
the hexagonal structure was accurately reproduced using the inverse problem
approach, as shown in Fig. 6a- 6e. Hence, given the same sinogram data, the
inverse approach is expected to better distinguish pins missing from the fuel
assembly and result in lower false alarm rates, compared to FBP. In Case 6, we
achieved the best MSE and SSIM, due to the smallest average pixel intensity in
all cases. Both FBP and inverse approach led to relatively high mis-
classification rates due to the low activity of the pins in the middle of the
assembly. However, we do not expect such a large variation of pin activities
in an actual fuel assembly [15].

Table 2: Comparison of image quality and mis-classification rates for the six
simulated cases reconstructed using traditional FBP and the inverse (INV)
approach. FP: false positive (missing pins mis-classified as present) rate;
FN: false negative (present pins mis-classified as missing) rate.

Case | MSE | SSIM | Mis-classification rate
---|---|---|---
FBP | INV | FBP | INV | FBP | INV
FP | FN | FP | FN
1 | 96.62 | 41.45 | 0.16 | 0.59 | 0 | 1.51% | 0 | 0
2 | 97.01 | 41.76 | 0.16 | 0.59 | 0.68% | 0 | 0 | 0
3 | 95.32 | 40.69 | 0.16 | 0.56 | 0.34% | 0 | 0 | 0
4 | 96.63 | 38.43 | 0.16 | 0.64 | 0 | 0 | 0 | 0
5 | 88.70 | 37.38 | 0.17 | 0.60 | 0 | 3.40% | 0 | 0
6 | 83.35 | 36.33 | 0.21 | 0.63 | 0 | 35.03% | 0 | 27.55%

We estimated the activity of each pin by summing the pixels on the
reconstructed images and created a histogram of pin activities for each case,
shown in Fig. 7 and Fig. 11. It should be noted that for the FBP
reconstruction, there is no definite relationship between the pixel sum and
the absolute activity. Appropriate normalization constant needs to be applied
to convert the pixel sum into activity, which is not always possible in an
actual inspection. In this case, we calculated the ratio between the mean of
the pixel sum distribution and the true activity for Cases 2-5. We then used
the average of these ratios as the normalization constant and applied it to
all cases to convert the pixel sum to absolute activity. In contrast,
normalization is not needed for the inverse reconstruction because the
response matrix has already been normalized per unit source particle. Compared
to Fig. 11, the pin activity distributions in Fig. 7 were narrower. We
calculated the relative error compared to the ground truth and the standard
deviation of pin activity, as shown in Table 3. We obtained smaller standard
deviations by using the inverse approach in all cases. The relative error was
negligible compared to the ground truth, except for the low activity pins in
Group 1 of Case 6.

Table 3: Comparison of activity estimations. The mean NPS per pin is the mean
of the activity distribution; relative error is the relative difference
between the mean activity and ground truth; standard deviation refers to the
standard deviation of the pin activity distribution. For FBP in Group 1 of
Case 6, no data is shown because no present pin is identified.

Case | Mean NPS per pin ($\times 10^{7}$) | Relative Error (%) | Standard deviation (%)
---|---|---|---
Truth | FBP | INV | FBP | INV | FBP | INV
1 | 1.51 | 1.57 | 1.57 | 3.91 | 3.74 | 6.16 | 4.36
2 | 1.70 | 1.72 | 1.74 | 1.39 | 2.21 | 7.57 | 3.44
3 | 1.70 | 1.71 | 1.73 | -0.68 | 2.02 | 8.54 | 5.09
4 | 1.70 | 1.72 | 1.73 | 1.12 | 1.46 | 7.71 | 3.33
5 | 1.70 | 1.65 | 1.66 | -2.91 | -2.25 | 15.75 | 6.85
6 | Group 1 | 0.29 | N.A. | 0.39 | N.A. | 35.73 | N.A. | 14.22
Group 2 | 0.72 | 0.75 | 0.72 | 4.69 | 0.54 | 8.25 | 4.64
Group 3 | 1.44 | 1.42 | 1.43 | -1.25 | -0.59 | 10.68 | 3.70

Compared to $\mathrm{UO}_{2}$ fuel rod, the cobalt rod simulated in this work
resulted in less attenuation because of its lower density and atomic number.
Nevertheless, this work describes a general approach for correcting the gamma-
ray attenuation and down-scattering in the energy window of interest and it
can be easily adapted to $\mathrm{UO}_{2}$ fuel assemblies by updating the
atomic composition and gamma-ray source term in the simulation. In this work,
we assumed that the detector performance is the same in the simulation of the
sinogram and response matrix, which is not true for the real PGET device. The
correction for varying detector performances across the detectors, either at
the hardware or software level, is crucial to obtain an accurate estimation of
pin activity. When identifying drifting detectors in post-processing, e.g.,
whose response is anomalously different compared to neighbour detectors, a
simple approach consists in neglecting their response. In a preliminary
analysis on simulated data, we replaced the response of an increasing number
of detectors in the sinogram with null arrays. We found that if the number of
“neglected” detectors is below ten in the PGET setup, the reconstructed image
is negligibly affected by not including their response. Future work will be
needed to study the robustness of the software suite as a function of a
systematic bias in the detector performance.

## Conclusion

In this work, we have implemented a full set of software, which is able to
reconstruct cross-sectional images of mockup fuel assemblies acquired by a
simulated PGET system, identify missing fuel pins, and estimate fuel pin
activities based on the reconstructed image. We have developed a linear
forward model that accounts for the scattering of gamma rays in the assembly
to accurately characterize the response matrix of the PGET system. The image
reconstruction was formulated into a linear inverse problem by modeling the
observation noise as Gaussian, which was solved using FISTA. The reconstructed
image was fed to a convolutional neural network to automatically identify the
present pins and determine their centroids. Compared to the FBP approach, the
inverse problem approach resulted in over 50% lower MSE and 200% higher SSIM,
and consequently lower mis-classification rates in pin identification in all
cases. Based on the pin identification results, we estimated the pin activity
by summing up the pixel values around the centroid inside the pin radius on
the image. Compared to the FBP, the inverse approach resulted in smaller
standard deviations of pin activity in all cases, with negligible bias with
respect to the ground truth. The proposed inverse approach to reconstruct a
fuel pin cross section, identify fuel pins and calculate their activity took
approximately 8 minutes to run on an Intel Core i9-7920X CPU without
parallelization. We are currently improving the algorithm to allow automatic
classification of different activity groups in a real fuel assembly.

## Methods

### Monte Carlo Simulation of the PGET based on MCNP

Fig. 12 shows the cross-sectional view of the MCNP model of Case 1. In Case 4
and 5, depleted uranium pins were used, where Co was replaced by 0.20%
enrichment UO2. We used the F4 tally as the detector response model in the
MCNP simulation and the response matrix calculation. The F4 tally estimates
the average photon flux in the detector cell by summing the track lengths of
all particles in the 700-1500 keV energy window [16]. The absolute activity
measurement can be obtained using the proposed method as long as the simulated
quantity in the response matrix is consistent with the simulated response to
an unknown inspected fuel bundle. When applying the proposed technique to
experimental data, one would want to validate the simulated detector response
with the measured one, to properly incorporate specific detector’s properties,
such as its energy resolution and time response, in the simulated model.

Figure 12: The cross-sectional view of the MCNP model in Case 1. The hexagonal
fuel assembly contains 331 60Co rods and is submerged in a water cylinder of
33 cm diameter. The fuel pin is simulated as a 60Co cylinder of 7 mm diameter,
coated with 1 mm thick aluminum cladding. Gamma rays emitted by the 60Co rods
are detected by two collimated CdTe detector arrays located on two sides of
the fuel assembly, each encompassing 91 detectors. The tungsten collimator
between the detector array and fuel assembly is 10 cm long and has an aperture
of 1.5 mm. Each CdTe detector is closely attached to the collimator and is of
size $0.175\times 0.35\times 0.35{\text{ cm}}^{3}$. The distance between two
neighbouring detectors is 4 mm. The bottom array is shifted to left by 2 mm,
with respect to the top one [3, 4]. The detector arrays rotate clockwise and
scan the fuel bundle in steps of 1$\degree$, which generates a $182\times 360$
sinogram.

### Image Reconstruction Methods

In this section, we describe in detail the two image reconstruction methods
implemented in this work: the FBP and linear inverse problem method. We
applied both methods to reconstruct images from the sinograms and compared
their performance.

#### Filtered Back-Projection

Fig. 13 shows the tomography measurement of the fuel assembly at angle
$\theta$. The FBP method relies on two approximations: first, we neglect the
attenuation and scattering of gamma rays in the system; second, for pins at
different locations, we assume that the geometric efficiencies are the same.
Under these approximations, the sinogram $\boldsymbol{\phi}(\rho,\theta)$ can
be simplified as the integral of the source distribution $\mathbf{s}(x,y)$
along the red line passing through the detector center, i.e.,

$\displaystyle\boldsymbol{\phi}(\rho,\theta)=\iint_{\mathbb{R}^{2}}\mathbf{s}(x,y)\delta(x\cos\theta+y\sin\theta-\rho)dxdy.$
(1)

Eq. (1) is the standard forward model in X-ray tomography imaging. The
classical inverse operation from the sinogram to the source is the filtered
back-projection, which is also a standard imaging algorithm in X-ray imaging
[17] and will not be detailed here.

Figure 13: The tomography measurement of the fuel assembly at an angle
$\theta$. The counts of detector $(\rho,\theta)$ can be approximated by the
integral of the source distribution $s(x,y)$ along the red line that passes
through the detector center.

#### Inverse Problem Approach

The inverse reconstruction approach is based on the conversion of the image
reconstruction problem into a linear inverse problem. Let the vectorized image
of the fuel assembly be $\mathbf{s}$, which contains $N$ unknown pixel values,
and the vectorized simulated sinogram of size $M=65,520$ ($182\times 360$) be
$\boldsymbol{\phi}$. The inverse approach relies on the assumption that
$\boldsymbol{\phi}$ is a linear function of $\mathbf{s}$:

$\displaystyle\boldsymbol{\phi}=\mathbf{A}\mathbf{s}+\mathbf{n},$ (2)

where $\mathbf{A}$ is the system response matrix of size $M\times N$, and
$\mathbf{n}$ models random observation noise, assumed to be isotropic Gaussian
distributed. Physically, the $i$-th column of the response matrix $\mathbf{A}$
is the vectorized sinogram corresponding to a pin distribution with unit
activity in pixel $i$ but zero elsewhere. Accurate determination of the system
response matrix is crucial to obtain a high-quality image and avoid systematic
bias in pin identification and activity estimation. The analytical derivation
of the response matrix is discussed in the next section.

Given the simulated sinogram and system response matrix $\mathbf{A}$, we can
reconstruct the image by solving the following equation:

$\hat{\mathbf{s}}=\operatorname*{arg\,min}_{{s_{i}\geq 0,\forall
i}}\quad\left[\frac{1}{2}\|\boldsymbol{\phi}-\mathbf{A}\mathbf{s}\|_{2}^{2}+\lambda\|\mathbf{s}\|_{1}\right]=\operatorname*{arg\,min}_{{s_{i}\geq
0,\forall
i}}\quad\left[\frac{1}{2}(\boldsymbol{\phi}-\mathbf{A}\mathbf{s})^{T}(\boldsymbol{\phi}-\mathbf{A}\mathbf{s})+\lambda\sum_{i=1}^{N}|\mathbf{s}_{i}|\right]$
(3)

where the first term is the data-fidelity term assuming Gaussian noise, and
the second term is a regularization term acknowledging that the fuel pin is
sparsely distributed. The regularization parameter $\lambda$ is chosen based
on the noise level. To solve Eq. (3), we have implemented the fast iterative
shrinkage-thresholding algorithm (FISTA) [12].

As an alternative, we can model the observation noise by Poisson noise and the
data-fidelity term is changed accordingly [13, 18] as follows

$\hat{\mathbf{s}}=\operatorname*{arg\,min}_{{s_{i}\geq 0,\forall
i}}\quad\left[\sum_{j=1}^{M}([\mathbf{A}\mathbf{s}]_{j}-\boldsymbol{\phi}_{j}\log[\mathbf{A}\mathbf{s}]_{j})+\lambda\|\mathbf{s}\|_{1}\right].$
(4)

To solve Eq. (4), we used the PIDAL algorithmic structure described in [13].

#### Computation of the Response Matrix

The calculation of the system response matrix $\mathbf{A}$ involves the
calculation of detector response to each pixel, with each response being one
column in the matrix. For inner pixels, the contribution from scattered
photons is non-negligible due to the high attenuation. To account for this, we
have implemented an algorithm that combines stochastic photon transport with
deterministic collimator-detector modeling to accurately calculate the system
response matrix.

For real fuel assemblies, no prior information of pin distribution and pin
radius will be given. It is therefore necessary to first estimate these
quantities before running the Monte Carlo simulation. As discussed in the
“Results-Inverse approach” section, this is done by performing a rough image
reconstruction using our previously developed model, where uniform attenuation
map is used and no scattering effect is considered [11]. Based on the
reconstructed image, one could determine the pin locations and pin radius, and
create a material map, where the material is assumed to be cobalt inside the
pin, and water outside.

Next, we simulate the travel of photons inside the fuel assembly using Monte
Carlo. Two sets of grids are generated. The fine grid has a pixel size of
$0.5\times 0.5$ mm2, while the coarse grid has a pixel size of $2\times 2$
mm2. We first discretize the region of interest on the fine grid and calculate
the response to the pixels whose center points are inside the pin. Then we
calculate the response to a pixel on the coarse grid by summing the responses
to the corresponding $4\times 4$ pixels on the fine grid. Based on the pin
identification result in the previous step, the potential pin-present region
are pixelated into 5,329 coarse pixels. We iterate over all these pixels and
form the response matrix. In this way, we are able to reduce the pixelization
error introduced during the discretization of the pins, without increasing the
final response matrix size.

The response to each fine pixel whose center point is inside the pin is
calculated in the following way. A photon
$(\vec{r_{0}},\vec{\Omega_{0}},E_{0},W_{0})$ is created, with the initial
position $\vec{r_{0}}$ uniformly sampled inside the pixel, the initial moving
direction $\vec{\Omega_{0}}$ uniformly sampled in $4\pi$ space, the initial
energy $E_{0}$ sampled from the source energy spectrum, and the initial weight
$W_{0}$ assigned to 1. We track the photon inside the PGET using the delta-
tracking algorithm [19] and determine the position where the photon interacts
with medium. We force the interaction to be Compton scattering by multiplying
its weight with the probability that the interaction is Compton scattering,

$W_{i}=W_{i-1}\frac{\mu_{\text{sc}}(\vec{r}_{i-1},E_{i-1})}{\mu(\vec{r}_{i-1},E_{i-1})}$
(5)

where $\mu_{\text{sc}}(\vec{r}_{i-1},E_{i-1})$ and
${\mu(\vec{r}_{i-1},E_{i-1})}$ are the Compton scattering attenuation
coefficient and total attenuation coefficient at the $i$-th interaction site,
respectively. A scattering angle is sample based on Kahn’s rejection algorithm
[20], and the photon energy $E_{i}$ and moving direction $\vec{\Omega_{i}}$
are updated accordingly. This process is repeated until the photon escapes the
fuel assembly or the photon energy is below the detection threshold. A copy of
the photon is saved at its creation site and each interaction site to a sub-
projection map.

Once the sub-projection map is obtained, we apply the convolutional forced
detection (CFD) technique to calculate the detector response. CFD is a widely-
used variance reduction technique for X-ray downscatter simulation in single
photon emission computed tomography (SPECT), which has been shown to be 50-100
times faster than conventional forced detection technique [21] and several
thousand times faster than full-3D Monte Carlo simulation [22]. Fig. 14
illustrates its principle. In CFD, the photon is forced to travel
perpendicularly to the detector plane upon its creation or interaction with
medium to effectively simulate all the possible spatial distributions of
particle interactions and consequent detection events. For an unscattered
photon, the contribution to the F4 tally in the detector cell is given by

$W\times\frac{\Delta\Omega}{4\pi}\times
e^{-\int\mu(\vec{r},E)ds}\times\frac{T_{l}}{V}$ (6)

and for a scattered photon, the contribution is

$W\times\frac{\sigma(\theta)\Delta\Omega}{2\pi\int_{0}^{\pi}\sigma(\theta^{\prime})\sin(\theta^{\prime})d\theta^{\prime}}\times
e^{-\int\mu(\vec{r},E(\theta))ds}\times\frac{T_{l}}{V}$ (7)

In Eq. (6) and (7), $W$ is the photon weight, $\Delta\Omega$ is the solid
angle subtended by the detector in steradian, $\sigma(\theta)$ is the
differential cross-section for Compton scattering, $\mu(\vec{r},E)$ is the
total attenuation coefficient along the CFD path, and the last term stands for
the contribution to F4 tally, i.e., average track length in detector cell
normalized by detector volume.

Figure 14: Illustration of the CFD technique. A photon shown as the gray disk
is created on the left and travels in the medium along the solid arrow. The
disk radius represents the photon energy. Upon it creation and each
interaction with the medium, the photon is forced to be scattered along the
dash arrow and to be detected by the detector with a certain probability.

Due to the non-ideal collimation, a photon will contribute counts not only to
the detector in the same column, but also to the neighbouring ones. This
phenomenon is characterized by the depth-dependent point spread function (PSF)
of the collimator, i.e., the photon count distribution on the detection plane
when a point source is imaged. PSF depends on the vertical distance $z$
between the photon and the detector array [23]:

$\begin{gathered}{PSF_{z}(x)=\frac{2\sqrt{\ln(2)}}{\sqrt{\pi}\mathrm{FWHM}(z)}\exp(\frac{-4\ln{2}\times
x^{2}}{\mathrm{FWHM}(z)^{2}}),\mathrm{FWHM}(z)=b+\frac{b}{l_{\text{eff}}}z,{l_{\text{eff}}}=l-\frac{2}{\mu}}\end{gathered}$
(8)

where $b$ is the collimator aperture, $l$ is the collimator length, $\mu$ is
the total attenuation coefficient in the collimator, and $\mathrm{FWHM}$ is
the width of the collimator PSF.

To calculate the response to a pin at an angle $\theta$, the photons saved in
the sub-projection map are first rotated by an angle $-\theta$. We then sum
the weights of the photons in the same pixel on the sub-projection map and
calculate their contribution to the detector response based on Eq. (6) or (7),
which creates a projection map. We then convolve each row in the projection
map with the PSF function and sum all rows up to get the response at this
angle. Next, we increment the angle $\theta$ and calculate the response at the
next angle. In this way, photon tracking in the fuel assembly is done only
once, which saves computation time.

We simulated the travel of 6,400 photons to create the sub-projection map for
one pixel. It took approximately 4 minutes to calculate the response matrix
containing 5,329 pixels using CFD on an Intel Core i9-7920X CPU.

### Pin Localization

Images reconstructed using both approaches are input to a convolution neural
network, referred to as U-net [24], to perform pin identification. The
experimental data provided by IAEA within the framework of a technology open
challenge [7] were used for the training of the U-net. The training data
consisted of the FBP images reconstructed from the measured sinograms of mock-
up assemblies of 12 different geometries with a variable number of 60Co pins,
and examples of the reconstructed images of these assemblies can be found in
[7]. The U-net was trained to minimize a binary crossentropy loss between the
predicted pin masks and ground truth. For each image, we provided four
randomly rotated ones as input to the network, for data augmentation. Data
augmentation allowed us to generate new training instances from existing ones,
artificially boosting the size of the training set. The network was trained
for 10,000 epochs, i.e., complete iteration over the whole dataset, using a
learning rate of $10^{-5}$ and an Adam optimizer [25]. We found that $10^{-5}$
is the learning rate for which the slope of the crossentropy loss is
maximized, allowing to reach the minimum. The training was early stopped when
the loss reached its minimum value. The trained U-net can extract visible
structures from the input image and output a mask based on the extracted
features. Bright regions on the mask with areas above a certain threshold are
identified as present pins, and their centroids are calculated as the
coordinates of the center of each pin.

## References

  * [1] United Nations Office for Disarmament Affairs.  Treaty on the non-proliferation of nuclear weapons (1970).  URL http://disarmament.un.org/treaties/t/npt/text.
  * [2] ElBaradei, M.  Addressing verification challenges.  In _Proceedings of an International Safeguards Symposium_ , 21 (Vienna, Austria, 2006).
  * [3] Honkamaa, T. _et al._ A prototype for passive gamma emission tomography.  In _Proc. Symp. Int. Safeguards_ (2014).
  * [4] White, T. _et al._ Application of passive gamma emission tomography (pget) for the verification of spent nuclear fuel.  In _INMM 59th Annual Meeting, Baltimore, Maryland, USA_ (2018).
  * [5] Mayorov, M. _et al._ Gamma emission tomography for the inspection of spent nuclear fuel.  In _2017 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC)_ , 1–2 (2017).
  * [6] SCK CEN.  Use of the fork detector in safeguards inspections (2007).  URL https://inis.iaea.org/collection/NCLCollectionStore/_Public/40/100/40100153.pdf.
  * [7] IAEA.  Iaea tomography reconstruction and analysis challenge.  https://ideas.unite.un.org/iaea-tomography/Page/Home (2018).  [Online; accessed 12-June-2020].
  * [8] Backholm, R. _et al._ Simultaneous reconstruction of emission and attenuation in passive gamma emission tomography of spent nuclear fuel.  _Inverse Problems & Imaging_ 14, 317 (2020).  URL http://aimsciences.org//article/id/4227366b-428b-4ed1-bda6-31696cf5f9d2.  DOI 10.3934/ipi.2020014.
  * [9] Goorley, J. T. _et al._ Initial mcnp6 release overview-mcnp6 version 1.0.  Tech. Rep., Los Alamos National Lab.(LANL), Los Alamos, NM (United States) (2013).
  * [10] Di Fulvio, A. _et al._ Neutron rodeo phase ii final report.  Tech. Rep., Argonne National Lab.(ANL), Argonne, IL (United States) (2019).
  * [11] Fang, M., Latta, D. D., Altmann, Y., Salvatori, M. & Fulvio, A. D.  Comparison of Image Reconstruction Methods for Simulated Passive Gamma Emission Tomography.  In _INMM 61st Annual Meeting_ (Baltimore, Maryland, USA, 2020).
  * [12] Beck, A. & Teboulle, M.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  _SIAM journal on imaging sciences_ 2, 183–202 (2009).
  * [13] Figueiredo, M. A. T. & Bioucas-Dias, J.  Restoration of poissonian images using alternating direction optimization.  _IEEE Transactions on Image Processing_ 19, 3133–3145 (2010).  DOI 10.1109/TIP.2010.2053941.
  * [14] Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 600–612 (2004).
  * [15] Jardine, L.  Radiochemical assays of irradiated vver-440 fuel for use in spent fuel burnup credit activities.  Tech. Rep., Lawrence Livermore National Lab. (2005).
  * [16] MCNP, A.  General monte carlo n-particle transport code, version 5, volume i: Overview and theory.  _X-5 Monte Carlo Team, Los Alamos national laboratory_ (2003).
  * [17] Prince, J. L. & Links, J. M.  _Medical imaging signals and systems_ (Pearson Prentice Hall Upper Saddle River, 2006).
  * [18] Lefkimmiatis, S. & Unser, M.  Poisson image reconstruction with hessian schatten-norm regularization.  _IEEE transactions on image processing_ 22, 4314–4327 (2013).
  * [19] Woodcock, E., Murphy, T., Hemmings, P. & Longworth, S.  Techniques used in the gem code for monte carlo neutronics calculations in reactors and other systems of complex geometry.  In _Proc. Conf. Applications of Computing Methods to Reactor Problems_ , vol. 557 (1965).
  * [20] Kahn, H.  Applications of monte carlo.  Tech. Rep., RAND Corp., Santa Monica, Calif. (1954).
  * [21] Hutton, B. F., Buvat, I. & Beekman, F. J.  Review and current status of spect scatter correction.  _Physics in Medicine & Biology_ 56, R85 (2011).
  * [22] Beekman, F. J., de Jong, H. W. & van Geloven, S.  Efficient monte carlo based reconstruction for general quantitative spect.  In _2001 IEEE Nuclear Science Symposium Conference Record (Cat. No. 01CH37310)_ , vol. 4, 1864–1868 (IEEE, 2001).
  * [23] de Jong, H. W., Slijpen, E. T. & Beekman, F. J.  Acceleration of monte carlo spect simulation using convolution-based forced detection.  _IEEE Transactions on Nuclear Science_ 48, 58–64 (2001).
  * [24] Ronneberger, O., Fischer, P. & Brox, T.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , 234–241 (Springer, 2015).
  * [25] Kingma, D. P. & Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).

## Acknowledgements

This work was funded in part by the Nuclear Regulatory Commission Faculty
Development Grant number 31310019M0011 and the Consortium for Verification
Technology under Department of Energy National Nuclear Security Administration
award number DE-NA0002534. This work was also supported by the Royal Academy
of Engineering under the Research Fellowship scheme RF201617/16/31.

## Author contributions statement

M.F. developed the forward model and analyzed the data, Y.A. implemented the
FISTA and PIDAL algorithms, D.D.L. and M.S. developed and trained the neural
network, A.D.F. implemented the FBP and the MC model, conceived and oversaw
the project. M.F., Y.A., A.D.F., and M.S. wrote the manuscript. All authors
reviewed the manuscript.

## Additional information

The authors declare no competing interests. Software not already available
online will be made publicly and freely available on the authors’ website once
this manuscript is accepted for publication.


# Backtracing: Retrieving the Cause of the Query

Rose E. Wang Pawan Wirawarn Omar Khattab
Noah Goodman Dorottya Demszky
Stanford University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Many online content portals allow users to ask questions to supplement their
understanding (e.g., of lectures). While information retrieval (IR) systems
may provide answers for such user queries, they do not directly assist content
creators—such as lecturers who want to improve their content—identify segments
that caused a user to ask those questions. We introduce the task of
_backtracing_ , in which systems retrieve the text segment that most likely
caused a user query. We formalize three real-world domains for which
backtracing is important in improving content delivery and communication:
understanding the cause of (a) student confusion in the Lecture domain, (b)
reader curiosity in the News Article domain, and (c) user emotion in the
Conversation domain. We evaluate the zero-shot performance of popular
information retrieval methods and language modeling methods, including bi-
encoder, re-ranking and likelihood-based methods and ChatGPT. While
traditional IR systems retrieve semantically relevant information (e.g.,
details on “projection matrices” for a query “does projecting multiple times
still lead to the same point?”), they often miss the causally relevant context
(e.g., the lecturer states “projecting twice gets me the same answer as one
projection”). Our results show that there is room for improvement on
backtracing and it requires new retrieval approaches. We hope our benchmark
serves to improve future retrieval systems for backtracing, spawning systems
that refine content generation and identify linguistic triggers influencing
user queries.111 Our code and data are opensourced:
https://github.com/rosewang2008/backtracing.

Backtracing: Retrieving the Cause of the Query

Rose E. Wang Pawan Wirawarn Omar Khattab Noah Goodman Dorottya Demszky
Stanford University<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Content creators and communicators, such as lecturers, greatly value feedback
on their content to address confusion and enhance its quality (Evans and
Guymon, 1978; Hativa, 1998). For example, when a student is confused by a
lecture content, they post questions on the course forum seeking
clarification. Lecturers want to determine where in the lecture the
misunderstanding stems from in order to improve their teaching materials
(McKone, 1999; Harvey, 2003; Gormally et al., 2014). The needs of these
content creators are different than the needs of information seekers like
students, who may directly rely on information retrieval (IR) systems such as
Q&A methods to satisfy their information needs (Schütze et al., 2008; Yang et
al., 2015; Rajpurkar et al., 2016; Joshi et al., 2017; Yang et al., 2018).

Figure 1:  The task of backtracing takes a query and identifies the context
that triggers this query. Identifying the cause of a query can be challenging
because of the lack of explicit labeling, large corpus size, and domain
expertise to understand both the query and corpus.  Figure 2:  Retrieving the
correct triggering context can provide insight into how to better satisfy the
user’s needs and improve content delivery. We formalize three real-world
domains for which backtracing is important in providing context on a user’s
query: (a) The Lecture domain where the objective is to retrieve the cause of
student confusion; (b) The News Article domain where the objective is to
retrieve the cause of reader curiosity; (c) The Conversation domain where the
objective is to retrieve the cause of user emotion (e.g., anger). The user’s
query is shown in the gray box and the triggering context is the green-
highlighted sentence. Popular retrieval systems such as dense retriever-based
and re-ranker based systems retrieve incorrect contexts shown in red.

Identifying the cause of a query can be challenging because of the lack of
explicit labeling, implicit nature of additional information need, large size
of corpus, and required domain expertise to understand both the query and
corpus. Consider the example shown in Figure 1. First, the student does not
explicitly flag what part of the lecture causes their question, yet they
express a latent need for additional information outside of the lecture
content. Second, texts like lecture transcripts are long documents; a lecturer
would have a difficult time pinpointing the precise source of confusion for
every student question they receive. Finally, some queries require domain
expertise for understanding the topic and reason behind the student’s
confusion; not every student question reflects the lecture content verbatim,
which is what makes backtracing interesting and challenging.

To formalize this task, we introduce a novel retrieval task called
backtracing. Given a query (e.g., a student question) and a corpus (e.g., a
lecture transcript), the system must identify the sentence that most likely
provoked the query. We formalize three real-world domains for which
backtracing is important for improving content delivery and communication.
First is the Lecture domain where the goal is to retrieve the cause of student
confusion; the query is a student’s question and the corpus is the lecturer’s
transcript. Second is the News Article domain where the goal is to retrieve
the cause of a user’s curiosity in the news article domain; the query is a
user’s question and the corpus is the news article. Third is the Conversation
domain where the goal is to retrieve the cause of a user’s emotion (e.g.,
anger); the query is the user’s conversation turn expressing that emotion and
the corpus is the complete conversation. Figure 2 illustrates an example for
each of these domains. These diverse domains showcase the applicability and
common challenges of backtracing for improving content generation, similar to
heterogeneous IR datasets like BEIR (Thakur et al., 2021).

We evaluate a suite of popular retrieval systems, like dense retriever-based
(Reimers and Gurevych, 2019a; Guo et al., 2020; Karpukhin et al., 2020) or re-
ranker-based systems (Nogueira and Cho, 2019; Craswell et al., 2020; Ren et
al., 2021). Additionally, we evaluate likelihood-based retrieval methods which
use pre-trained language models (PLMs) to estimate the probability of the
query conditioned on variations of the corpus (Sachan et al., 2022), such as
measuring the query likelihood conditioned on the corpus with and without the
candidate segment. Finally, we also evaluate the long context window
gpt-3.5-turbo-16k ChatGPT model because of its ability to process long texts
and perform instruction following. We find that there is room for improvement
on backtracing across all methods. For example, the bi-encoder systems
(Reimers and Gurevych, 2019a) struggle when the query is not semantically
similar to the text segment that causes it; this often happens in the
Conversation and Lecture domain, where the query may be phrased differently
than the original content. Overall, our results indicate that backtracing is a
challenging task which requires new retrieval approaches to take in causal
relevance into account; for instance, the top-3 accuracy of the best model is
only $44\%$ on the Lecture domain.

In summary, we make the following contributions in this paper:

  * •

We propose a new task called backtracing where the goal is to retrieve the
cause of the query from a corpus. This task targets the information need of
content creators who wish to improve their content in light of questions from
information seekers.

  * •

We formalize a benchmark consisting of three domains for which backtracing
plays an important role in identifying the context triggering a user’s query:
retrieving the cause of student confusion in the Lecture setting, reader
curiosity in the News Article setting, and user emotion in the Conversation
setting.

  * •

We evaluate a suite of popular retrieval systems, including bi-encoder and re-
ranking architectures, as well as likelihood-based methods that use pretrained
language models to estimate the probability of the query conditioned on
variations of the corpus.

  * •

We show that there is room for improvement and limitations in current
retrieval methods for performing backtracing, suggesting that the task is not
only challenging but also requires new retrieval approaches.

Figure 3:  Illustration of backtracing. The goal of backtracing is to identify
the most likely sentence from the ordered corpus $X$ that caused the query
$q$. One example is the Lecture domain where the corpus is a lecture
transcript and the query is a student question. The lecturer only discusses
about projecting twice and the student further extends that idea to something
not raised in the lecture, namely into projecting a matrix an arbitrary $n$
times.

##  2 Related works

The task of information retrieval (IR) aims to retrieve relevant documents or
passages that satisfy the information need of a user (Schütze et al., 2008;
Thakur et al., 2021). Prior IR techniques involve neural retrieval methods
like ranking models (Guo et al., 2016; Xiong et al., 2017; Khattab and
Zaharia, 2020) and representation-focused language models (Peters et al.,
2018; Devlin et al., 2018; Reimers and Gurevych, 2019a). Recent works also use
PLMs for ranking texts in performing retrieval (Zhuang and Zuccon, 2021;
Zhuang et al., 2021; Sachan et al., 2022); an advantage of using PLMs is not
requiring any domain- or task-specific training, which is useful for settings
where there is not enough data for training new models. These approaches have
made significant advancements in assisting information seekers in accessing
information on a range of tasks. Examples of these tasks include recommending
news articles to read for a user in the context of the current article they’re
reading (Voorhees, 2005; Soboroff et al., 2018), retrieving relevant bio-
medical articles to satisfy health-related concerns (Tsatsaronis et al., 2015;
Boteva et al., 2016; Roberts et al., 2021; Soboroff, 2021), finding relevant
academic articles to accelerate a researcher’s literature search (Voorhees et
al., 2021), or extracting answers from texts to address questions (Yang et
al., 2015; Rajpurkar et al., 2016; Joshi et al., 2017; Yang et al., 2018).

However, the converse needs of content creators have received less
exploration. For instance, understanding what aspects of a lecture cause
students to be confused remains under-explored and marks areas for improvement
for content creators. Backtracing is related to work on predicting search
intents from previous user browsing behavior for understanding why users issue
queries in the first place and what trigger their information needs (Cheng et
al., 2010; Kong et al., 2015; Koskela et al., 2018). The key difference
between our approach and prior works is the nature of the input data and
prediction task. While previous methods rely on observable user browsing
patterns (e.g., visited URLs and click behaviors) for ranking future search
results, our backtracing framework leverages the language in the content
itself as the context for the user query and the output space for prediction.
This shift in perspective allows content creators to get granular insights
into specific contextual, linguistic triggers that influence user queries, as
opposed to behavioral patterns.

Another related task is question generation, which also has applications to
education (Heilman and Smith, 2010; Duan et al., 2017; Pan et al., 2019).
While question generation settings assume the answer can be identified in the
source document, backtracing is interested in the triggers for the questions
rather than the answers themselves. In many cases, including our domains, the
answer to the question may exist outside of the provided source document.

##  3 Backtracing

Formally, we define backtracing as: Given corpus of $N$ sentences
$X=\\{x_{1},\dots,x_{N}\\}$ and query $q$, backtracing selects

$\hat{t}=\arg\max_{t\in 1\dots N}p(t|x_{1},\dots,x_{N},q)$ (1)

where $x_{t}$ is the $t^{th}$ sentence in corpus $X$ and $p$ is a probability
distribution over the corpus indices, given the corpus and the query. Figure 3
illustrates this definition and grounds it in our previous lecture domain
example. This task intuitively translates to: Given a lecture transcript and
student question, retrieve the lecture sentence(s) that most likely caused the
student to ask that question.

Ideal methods for backtracing are ones that can provide a continuous scoring
metric over the corpus and can handle long texts. This allows for
distinguishable contributions from multiple sentences in the corpus, as there
can be more than one sentence that could cause the query. In the case where
there is more than one target sentence, our acceptance criterion is whether
there’s overlap between the target sentences and the predicted sentence.
Additionally, some text domains such as lectures are longer than the context
window lengths of existing language models. Effective methods must be able to
circumvent this constraint algorithmically (e.g., by repeated invocation of a
language model).

Our work explores the backtracing task in a “zero-shot” manner across a
variety of domains, similar to Thakur et al. (2021). We focus on a restricted
definition of zero-shot in which validation on a small development set is
permitted, but not updating model weights. This mirrors many emerging real-
world scenarios in which some data-driven interventions can be applied but not
enough data is present for training new models. Completely blind zero-shot
testing is notoriously hard to conduct within a reusable benchmark (Fuhr,
2018; Perez et al., 2021) and is much less conducive to developing different
methods, and thus lies outside our scope.

##  4 Backtracing Benchmark Domains

We use a diverse set of domains to establish a benchmark for backtracing,
highlighting both its broad applicability and the shared challenges inherent
to the task. This section first describes the domain datasets and then
describes the dataset statistics with respect to the backtracing task.

|  | Lec | News | Conv
---|---|---|---|---
Query | Total | $210$ | $1382$ | $671$
| Avg. words | $30.9$ | $7.1$ | $11.6$
| Max words | $233$ | $27$ | $62$
| Min words | $4$ | $1$ | $1$
Corpus | Total | $11042$ | $2125$ | $8263$
| Avg. size | $525.8$ | $19.0$ | $12.3$
| Max size | $948$ | $45$ | $6110$
| Min size | $273$ | $7$ | $6$
Table 1:  Dataset statistics on the query and corpus sizes for backtracing.
Lec is the Lecture domain, News is the News Article domain, and Conv is the
Conversation domain. The corpus size is measured on the level of sentences for
Lecture and News Article, and of conversation turns for Conversation.

###  4.1 Domains

Figure 2 illustrates examples of the corpus and query in each domain. Table 1
contains statistics on the dataset. The datasets are protected under the CC-BY
license.

#### Lecture

We use real-world university lecture transcripts and student comments to
construct the Lecture domain. Lectures are a natural setting for students to
ask questions to express confusion about novel concepts. Lecturers can benefit
from knowing what parts of their lecture cause confusion. We adapt the paired
comment-lecture dataset from Sight (Wang et al., 2023), which contains lecture
transcripts from MIT OpenCourseWare math videos and real user comments from
YouTube expressing confusion. While these comments naturally act as queries in
the backtracing framework, the comments do not have ground-truth target
annotations on what caused the comment in the first place. Our work
contributes these annotations. Two annotators (co-authors of this paper)
familiar with the task of backtracing and fluent in the math topics at a
university-level annotate the queries222The annotators must be fluent in the
math topics to understand both the lecture and query, and backtrace
accordingly.. They select up to 5 sentences and are allowed to use the
corresponding video to perform the task. $20$ queries are annotated by both
annotators and these annotations share high agreement: the annotators
identified the same target sentences for $70\%$ of the queries, and picked
target sentences close to each other. These annotation results indicate that
performing backtracing with consensus is possible. Appendix B includes more
detail on the annotation interface and agreement. The final dataset contains
210 annotated examples, comparable to other IR datasets (Craswell et al.,
2020, 2021; Soboroff, 2021).333After conducting 2-means 2-sided equality power
analysis, we additionally concluded that the dataset size is sufficiently
large—the analysis indicated a need for 120 samples to establish statistically
significant results, with power $1-\beta=0.8$ and $\alpha=0.05$. In the case
where a query has more than one target sentence, the accuracy criterion is
whether there’s overlap between the target sentences and predicted sentence
(see task definition in Section 3).

#### News Article

We use real-world news articles and questions written by crowdworkers as they
read through the articles to construct the News Article domain. News articles
are a natural setting for readers to ask curiosity questions, expressing a
need for more information. We adapt the dataset from Ko et al. (2020) which
contains news articles and questions indexed by the article sentences that
provoked curiosity in the reader. We modify the dataset by filtering out
articles that cannot fit within the smallest context window of models used in
the likelihood-based retrieval methods (i.e., $1024$ tokens). This adapted
dataset allows us to assess the ability of methods to incorporate more
contextual information and handling more distractor sentences, while
maintaining a manageable length of text. The final dataset contains 1382
examples.

#### Conversation

We use two-person conversations which have been annotated with emotions, such
as anger and fear, and cause of emotion on the level of conversation turns.
Conversations are natural settings for human interaction where a speaker may
accidentally say something that evokes strong emotions like anger. These
emotions may arise from cumulative or non-adjacent interactions, such as the
example in Figure 2. While identifying content that evokes the emotion
expressed via a query differs from content that causes confusion, the ability
to handle both is key to general and effective backtracing systems that
retrieve information based on causal relevance. Identifying utterances that
elicit certain emotions can pave the way for better emotional intelligence in
systems and refined conflict resolution tools. We adapt the conversation
dataset from Poria et al. (2021) which contain turn-level annotations for the
emotion and its cause, and is designed for recognizing the cause of emotions.
The query is one of the speaker’s conversation turn annotated with an emotion
and the corpus is all of the conversation turns. To ensure there are enough
distractor sentences, we use conversations with at least 5 sentences and use
the last annotated utterance in the conversation. The final dataset contains
671 examples.

###  4.2 Domain Analysis

To contextualize the experimental findings in Section 6, we first analyze the
structural attributes of our datasets in relation to backtracing.

#### How similar is the query to the cause?

To answer this question, we plot the semantic similarity of the query to the
ground-truth cause sentence (GT) in Figure 4. We additionally plot the maximal
similarity of the query to any corpus sentence (Max) and the difference
between the ground-truth and maximal similarity (Diff). This compares the
distractor sentences to the ground-truth sentences; the larger the difference
is, the less likely semantic relevance can be used as a proxy for causal
relevance needed to perform backtracing. This would also indicate that poor
performance of similarity-based methods because the distractor sentences
exhibit higher similarity. We use the all-MiniLM-L12-v2 S-BERT model to
measure semantic similarity (Reimers and Gurevych, 2019a).

Notably, the queries and their ground-truth cause sentences exhibit low
semantic similarity across domains, indicated by the low blue bars.
Additionally, indicated by the green bars, Conversation and Lecture have the
largest differences between the ground-truth and maximal similarity sentences,
whereas News Article has the smallest. This suggests that there may be
multiple passages in a given document that share a surface-level resemblance
with the query, but a majority do not cause the query in the Conversation and
Lecture domains. In the News Article domain, the query and cause sentence
exhibit higher semantic similarity because the queries are typically short and
mention the event or noun of interest. Altogether, this analysis brings forth
a key insight: Semantic relevance doesn’t always equate causal relevance.

Figure 4:  Each dataset plot shows the query similarity to the ground truth
cause sentence (GT), to the corpus sentence with maximal similarity (Max), and
the difference between the maximal and ground-truth similarity sentences
(Diff).

#### Where are the causes located in the corpus?

Figure 5:  Each row plot is a per-domain histogram of where the ground-truth
cause sentence lies in the corpus document. The x-axis reports the location of
the cause sentence; $0$ means the cause sentence is the first sentence and $1$
the last sentence. The y-axis reports the count of cause sentences at that
location.

Understanding the location of the cause provides insight into how much context
is needed in identifying the cause to the query. Figure 5 visualizes the
distribution of cause sentence locations within the corpus documents. These
plots show that while some domains have causes concentrated in specific
sections, others exhibit a more spread-out pattern. For the News Article
domain, there is a noticeable peak at the beginning of the documents which
suggests little context is needed to identify the cause. This aligns with the
typical structure of news articles where crucial information is introduced
early to capture the reader’s interest. As a result, readers may have
immediate questions from the onset. Conversely, in the Conversation domain,
the distribution peaks at the end, suggesting that more context from the
conversation is needed to identify the cause. Finally, in the Lecture domain,
the distribution is relatively uniform which suggests a broader contextual
dependence. The causes of confusion arise from any section, emphasizing the
importance of consistent clarity throughout an educational delivery.

An interesting qualitative observation is that there are shared cause
locations for different queries. An example from the Lecture domain is shown
in Figure 6 where different student questions are mapped to the same cause
sentence. This shows the potential for models to effectively perform
backtracing and automatically identify common locations of confusion for
lecturers to revise for future course offerings.

Lecture: […] So it’s 1 by 2x0 times 2y0, which is 2x0y0, which is, lo and
behold, 2. […] Student A’s question: why is 2xo(yo) = 2? Student B’s question:
When he solves for the area of the triangle, why does he say it doesn’t matter
what X0 and Y0 are? Does he just mean that all values of f(x) = 1/x will
result in the area of the triangle of the tangent line to be 2? Student C’s
question: Why always 2?? is there a prove?

Figure 6:  An example of a common confusion point where several students posed
questions concerning a particular part of the lecture.

##  5 Methods

We evaluate a suite of existing, state-of-the-art retrieval methods and report
their top-1 and top-3 accuracies: Do the top 1 and 3 candidate sentences
include the ground-truth sentences? Reporting top-k accuracy is a standard
metric in the retrieval setting. We also report their minimum distance within
the top-1 and top-3 candidates: What is the minimum distance between the
method’s candidates and the ground-truth sentences? The methods can be broadly
categorized into similarity-based (i.e., using sentence similarity) and
likelihood-based retrieval methods. Similar to Sachan et al. (2022), the
likelihood-based retrieval methods use PLMs to measure the probability of the
query conditioned on variations of the corpus and can be more expressive than
the similarity-based retrieval methods; we describe these variations in detail
below. We use GPT-2 (Radford et al., 2019), GPT-J (Wang and Komatsuzaki,
2021), and OPT-6.7B (Zhang et al., 2022) as the PLMs. We additionally evaluate
with gpt-3.5-turbo-16k, a new model that has a long context window ideal for
long text settings like Sight. However, because this model does not output
probability scores, we cast only report its top-1 results.

#### Random.

This method randomly retrieves a sentence from the corpus.

#### Edit distance.

This method retrieves the sentence with the smallest edit distance from the
query.

#### Bi-encoders.

This method retrieves the sentence with the highest semantic similarity using
the best performing S-BERT models (Reimers and Gurevych, 2019b). We use multi-
qa-MiniLM-L6-cos-v1 trained on a large set of question-answer pairs and all-
MiniLM-L12-v2 trained on a diversity of text pairs from sentence-transformers
as the encoders.

#### Cross-encoder.

This method picks the sentence with the highest predicted similarity score by
the cross-encoder. We use ms-marco-MiniLM-L-6-v2 (Thakur et al., 2021).

#### Re-ranker.

This method uses a bi-encoder to retrieve the top $k$ candidate sentences from
the corpus, then uses a cross-encoder to re-rank the $k$ sentences. We use
all-MiniLM-L12-v2 as the bi-encoder and ms-marco-MiniLM-L-6-v2 as the cross-
encoder. Since the smallest dataset—Daily Dialog—has a minimum of 5 sentences,
we use $k=5$ for all datasets.

####  gpt-3.5-turbo-16k.

This method is provided a line-numbered corpus and the query, and generates
the line number that most likely caused the query. The prompt used for
gpt-3.5-turbo-16k is in Appendix C.

|  | Lecture | News Article | Conversation
---|---|---|---|---
|  | @1 | @3 | @1 | @3 | @1 | @3
| Random | $0$ | $3$ | $6$ | $21$ | $11$ | $31$
| Edit | $4$ | $8$ | $7$ | $18$ | $1$ | $16$
| BM25 | $8$ | $15$ | $43$ | $65$ | $1$ | $35$
| Bi-Encoder (Q&A) | $23$ | $37$ | $48$ | $71$ | $1$ | $32$
| Bi-Encoder (all-MiniLM) | $26$ | $40$ | $49$ | $75$ | $1$ | $37$
| Cross-Encoder | $22$ | $39$ | $66$ | $\bf 85$ | $1$ | $15$
| Re-ranker | $\bf 30$ | $\bf 44$ | $66$ | $\bf 85$ | $1$ | $21$
| gpt-3.5-turbo-16k | $15$ | N/A | $\bf 67$ | N/A | $\bf 47$ | N/A
Single-sentence | GPT2 | $21$ | $34$ | $43$ | $64$ | $3$ | $46$
$p(q|s_{t})$ | GPTJ | $23$ | $42$ | $\bf 67$ | $\bf 85$ | $5$ | $\bf 65$
| OPT 6B | $\bf 30$ | $43$ | $66$ | $82$ | $2$ | $56$
Autoregressive | GPT2 | $11$ | $16$ | $9$ | $18$ | $5$ | $54$
$p(q|s_{\leq t})$ | GPTJ | $14$ | $24$ | $55$ | $76$ | $8$ | $60$
| OPT 6B | $16$ | $26$ | $52$ | $73$ | $18$ | $\bf 65$
ATE | GPT2 | $13$ | $21$ | $51$ | $68$ | $2$ | $24$
$p(q|S)-p(q|S/\ \\{s_{t}\\}\ )$ | GPTJ | $8$ | $18$ | $\bf 67$ | $79$ | $3$ | $18$
| OPT 6B | $2$ | $6$ | $64$ | $76$ | $3$ | $22$

Table 2:  Accuracy ($\uparrow$ % betterd). The best models in each column are
bolded. For each dataset, we report the top-$1$ and $3$ accuracies.
gpt-3.5-turbo-16k reports N/A for top-3 accuracy because it does not output
deterministic continuous scores for ranking sentences.

|  | Lecture | News Article | Conversation
---|---|---|---|---
| @1 | @3 | @1 | @3 | @1 | @3
| Random | $167.5$ | $67.8$ | $7.6$ | $3.0$ | $3.7$ | $1.7$
| Edit | $157.9$ | $70.7$ | $7.7$ | $3.4$ | $1.3$ | $0.9$
| BM25 | $122.7$ | $50.7$ | $4.6$ | $1.4$ | $1.3$ | $0.7$
| Bi-Encoder (Q&A) | $91.9$ | $35.2$ | $4.1$ | $1.2$ | $1.3$ | $0.8$
| Bi-Encoder (all-MiniLM) | $84.7$ | $38.6$ | $3.7$ | $1.0$ | $1.3$ | $0.7$
| Cross-Encoder | $96.6$ | $33.8$ | $2.5$ | $\bf 0.6$ | $1.3$ | $0.9$
| Re-ranker | $92.2$ | $41.4$ | $2.7$ | $\bf 0.6$ | $1.3$ | $0.9$
| gpt-3.5-turbo-16k | $73.9$ | N/A | $\bf 1.5$ | N/A | $\bf 1.0$ | N/A
Single-sentence | GPT2 | $5.4^{*}$ | $2.1^{*}$ | $4.6$ | $1.5$ | $1.5$ | $0.6$
$p(q|s_{t})$ | GPTJ | $\bf 5.0^{*}$ | $\bf 1.9^{*}$ | $2.5$ | $0.7$ | $1.4$ | $\bf 0.4$
| OPT 6B | $5.2^{*}$ | $2.3^{*}$ | $2.7$ | $0.8$ | $1.3$ | $0.5$
Autoregressive | GPT2 | $5.6^{*}$ | $3.4^{*}$ | $7.2$ | $4.8$ | $2.0$ | $0.8$
$p(q|s_{\leq t})$ | GPTJ | $5.5^{*}$ | $3.4^{*}$ | $1.8$ | $0.8$ | $2.0$ | $0.8$
| OPT 6B | $5.1^{*}$ | $3.5^{*}$ | $1.9$ | $1.0$ | $1.9$ | $0.7$
ATE | GPT2 | $7.4^{*}$ | $2.8^{*}$ | $4.7$ | $1.3$ | $1.5$ | $0.9$
$p(q|S)-p(q|S/\ \\{s_{t}\\}\ )$ | GPTJ | $7.2^{*}$ | $3.2^{*}$ | $2.9$ | $0.9$ | $1.6$ | $1.0$
| OPT 6B | $7.1^{*}$ | $\bf 1.9^{*}$ | $3.2$ | $1.1$ | $2.4$ | $1.0$

Table 3:  Minimum Sentence Distance from Ground Truth ($\downarrow$ better)
The best models in each column are bolded. For each dataset, we report the
minimum sentence distance from the ground truth cause sentence of the method’s
top-$1$ and $3$ candidates; 0 meaning that the method always predicts the
ground truth candidate sentence. Note for the likelihood-based methods on the
Lecture domain were evaluated on 20-sentence chunks of the original text due
to the context window limitation. If the top sentence is not in the top-chunk,
it is excluded in distance metric. We’ve marked the affected metrics with an
asterisk ∗.

#### Single-sentence likelihood-based retrieval $p(q|x_{t})$.

This method retrieves the sentence $x_{t}\in X$ that maximizes $p(q|x_{t})$.
To contextualize the corpus and query, we add domain-specific prefixes to the
corpus and query. For example, in Sight, we prepend “Teacher says: ” to the
corpus sentence and “Student asks: ” to the query. Due to space constraints,
Appendix C contains all the prefixes used.

#### Auto-regressive likelihood-based retrieval $p(q|x_{\leq t})$.

This method retrieves the sentence $x_{t}$ which maximizes $p(q|x_{\leq t})$.
This method evaluates the importance of preceding context in performing
backtracing. Lecture is the only domain where the entire corpus cannot fit
into the context window. This means that we cannot always evaluate
$p(q|x_{\leq t})$ for $x_{t}$ when $|x_{\leq t}|$ is longer than the context
window limit. For this reason, we split the corpus $X$ into chunks of $k$
sentences, (i.e., $X_{0:k-1},X_{k:2k-1},\dots$) and evaluate each $x_{t}$
within their respective chunk. For example, if $x_{t}\in X_{k:2k-1}$, the
auto-regressive likelihood score for $x_{t}$ is $p(q|X_{k:t})$. We evaluate
with $k=20$ because it is the maximum number of sentences (in addition to the
query) that can fit in the smallest model context window.

#### Average Treatment Effect (ATE) likelihood-based retrieval
$p(q|X)-p(q|X\setminus x_{t})$.

This method takes inspiration from treatment effects in causal inference
(Holland, 1986). We describe how ATE can be used as a retrieval criterion. In
our setting, the treatment is whether the sentence $x_{t}$ is included in the
corpus. We’re interested in the effect the treatment has on the query
likelihood:

$\displaystyle\texttt{ATE{}}(x_{t})$
$\displaystyle=p_{\theta}(q|X)-p_{\theta}(q|X\setminus\\{x_{t}\\}).$ (2)

ATE likelihood methods retrieve the sentence that maximizes
$\texttt{ATE{}}(x_{t})$. These are the sentences that have the largest effect
on the query’s likelihood. We directly select the sentences that maximize
Equation 2 for News Article and Conversation. We perform the same text
chunking for Lecture as in the auto-regressive retrieval method: If $x_{t}\in
X_{k:2k-1}$, the ATE likelihood score for $x_{t}$ is measured as
$p(q|X_{k:2k-1})-p(q|X_{k:2k-1}\setminus\\{x_{t}\\})$.

##  6 Results

The accuracy results are summarized in Table 2, and distance results in Table
3.

#### The best-performing models achieve modest accuracies.

For example, on the Lecture domain with many distractor sentences, the best-
performing model only achieves top-3 $44\%$ accuracy. On the Conversation
domain with few distractor sentences, the best-performing model only achieves
top-3 $65\%$ accuracy. This underscores that measuring causal relevance is
challenging and markedly different from existing retrieval tasks.

#### No model performs consistently across domains.

For instance, while a similarity-based method like the Bi-Encoder (all-MiniLM)
performs well on the News Article domain with top-3 $75\%$ accuracy, it only
manages top-3 $37\%$ accuracy on the Conversation domain. These results
complement the takeaway from the domain analysis in Section 4 that semantic
relevance is not a reliable proxy for causal relevance. Interestingly, on the
long document domain Lecture, the long-context model gpt-3.5-turbo-16k
performs worse than non-contextual methods like single-sentence likelihood
methods. This suggests that accounting for context is challenging for current
models.

#### Single-sentence methods generally outperform their autoregressive
counterparts except on Conversation.

This result complements the observations made in Section 4’s domain analysis
where the location of the causes concentrates at the start for News Article
and uniformly for Lecture, suggesting that little context is needed to
identify the cause. Conversely, conversations require more context to
distinguish the triggering contexts, which suggests why the autoregressive
methods perform generally better than the single-sentence methods.

#### ATE likelihood methods does not signicantly improve upon other methods.

Even though the ATE likelihood method is designed the calculate the effect of
the cause sentence, it competes with noncontextual methods such as the single-
sentence likelihood methods. This suggest challenges in using likelihood
methods to measure the counterfactual effect of a sentence on a query.

##  7 Conclusion

In this paper, we introduce the novel task of backtracing, which aims to
retrieve the text segment that most likely provokes a query. This task
addresses the information need of content creators who want to improve their
content, in light of queries from information seekers. We introduce a
benchmark that covers a variety of domains, such as the news article and
lecture setting. We evaluate a series of methods including popular IR methods,
likelihood-based retrieval methods and gpt-3.5-turbo-16k. Our results indicate
that there is room for improvement across existing retrieval methods. These
results suggest that backtracing is a challenging task that requires new
retrieval approaches with better contextual understanding and reasoning about
causal relevance. We hope our benchmark serves as a foundation for improving
future retrieval systems for backtracing, and ultimately, spawns systems that
empower content creators to understand user queries, refine their content and
provide users with better experiences.

## Limitations

#### Single-sentence focus.

Our approach primarily focuses on identifying the most likely single sentence
that caused a given query. However, in certain scenarios, the query might
depend on groups or combinations of sentences. Ignoring such dependencies can
limit the accuracy of the methods.

#### Content creators in other domains.

Our evaluation primarily focuses on the dialog, new article and lecture
settings. While these domains offer valuable insights, the performance of
backtracing methods may vary in other contexts, such as scientific articles
and queries from reviewers. Future work should explore the generalizability of
backtracing methods across a broader range of domains and data sources.

#### Long text settings.

Due to the length of the lecture transcripts, the transcripts had to be
divided and passed into the likelihood-based retrieval methods. This approach
may result in the omission of crucial context present in the full transcript,
potentially affecting the accuracy of the likelihood-based retrieval methods.
Exploring techniques to effectively handle larger texts and overcome model
capacity constraints would be beneficial for improving backtracing performance
in long text settings, where we would imagine backtracing to be useful in
providing feedback for.

#### Multimodal sources.

Our approach identifies the most likely text segment in a corpus that caused a
given query. However, in multimodal settings, a query may also be caused by
other data types, e.g., visual cues that are not captured in the transcripts.
Ignoring such non-textual data can limit the accuracy of the methods.

## Ethics Statement

Empowering content creators to refine their content based on user feedback
contributes to the production of more informative materials. Therefore, our
research has the potential to enhance the educational experiences of a user,
by assisting content creators through backtracing. Nonetheless, we are mindful
of potential biases or unintended consequences that may arise through our work
and future work. For example, the current benchmark analyzes the accuracy of
backtracing on English datasets and uses PLMs trained predominantly on English
texts. As a result, the inferences drawn from the current backtracing results
or benchmark may not accurately capture the causes of multilingual queries,
and should be interpreted with caution. Another example is that finding the
cause for a user emotion can be exploited by content creators. We consider
this as an unacceptable use case of our work, in addition to attempting to
identify users in the dataset or using the data for commercial gain.

## Acknowledgements

We’d like thank Michael Zhang and Dilip Arumugam for the fruitful
conversations at the start of this project. We’d also like to thank Gabriel
Poesia for helpful feedback on the paper.

## References

  * Boteva et al. (2016) Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. 2016.  A full-text learning to rank dataset for medical information retrieval.  In _Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20–23, 2016. Proceedings 38_ , pages 716–722. Springer.
  * Cheng et al. (2010) Zhicong Cheng, Bin Gao, and Tie-Yan Liu. 2010.  Actively predicting diverse search intent from user browsing behaviors.  In _Proceedings of the 19th international conference on World wide web_ , pages 221–230.
  * Craswell et al. (2021) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021.  Overview of the trec 2020 deep learning track.
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. 2020.  Overview of the trec 2019 deep learning track.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Duan et al. (2017) Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017.  Question generation for question answering.  In _Proceedings of the 2017 conference on empirical methods in natural language processing_ , pages 866–874.
  * Evans and Guymon (1978) Warren E Evans and Ronald E Guymon. 1978.  Clarity of explanation: A powerful indicator of teacher effectiveness.
  * Fuhr (2018) Norbert Fuhr. 2018.  Some common mistakes in ir evaluation, and how they can be avoided.  In _Acm sigir forum_ , volume 51, pages 32–41. ACM New York, NY, USA.
  * Gormally et al. (2014) Cara Gormally, Mara Evans, and Peggy Brickman. 2014.  Feedback about teaching in higher ed: Neglected opportunities to promote change.  _CBE—Life Sciences Education_ , 13(2):187–199.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016.  A deep relevance matching model for ad-hoc retrieval.  In _Proceedings of the 25th ACM international on conference on information and knowledge management_ , pages 55–64.
  * Guo et al. (2020) Mandy Guo, Yinfei Yang, Daniel Cer, Qinlan Shen, and Noah Constant. 2020.  Multireqa: A cross-domain evaluation for retrieval question answering models.
  * Harvey (2003) Lee Harvey. 2003.  Student feedback [1].  _Quality in higher education_ , 9(1):3–20.
  * Hativa (1998) Nira Hativa. 1998.  Lack of clarity in university teaching: A case study.  _Higher Education_ , pages 353–381.
  * Heilman and Smith (2010) Michael Heilman and Noah A Smith. 2010.  Good question! statistical ranking for question generation.  In _Human language technologies: The 2010 annual conference of the North American Chapter of the Association for Computational Linguistics_ , pages 609–617.
  * Holland (1986) Paul W Holland. 1986.  Statistics and causal inference.  _Journal of the American statistical Association_ , 81(396):945–960.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over bert.  In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_ , pages 39–48.
  * Ko et al. (2020) Wei-Jen Ko, Te-Yuan Chen, Yiyan Huang, Greg Durrett, and Junyi Jessy Li. 2020.  Inquisitive question generation for high level text comprehension.  _arXiv preprint arXiv:2010.01657_.
  * Kong et al. (2015) Weize Kong, Rui Li, Jie Luo, Aston Zhang, Yi Chang, and James Allan. 2015.  Predicting search intent based on pre-search context.  In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 503–512.
  * Koskela et al. (2018) Markus Koskela, Petri Luukkonen, Tuukka Ruotsalo, Mats Sjöberg, and Patrik Floréen. 2018.  Proactive information retrieval by capturing search intent from primary task context.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ , 8(3):1–25.
  * Library (2023) MiniChain Library. 2023.  MiniChain Library.  https://github.com/srush/minichain#typed-prompts.  [Online; accessed 4-June-2024].
  * McKenzie (2023) Ian McKenzie. 2023.  Inverse Scaling Prize: First Round Winners.  https://irmckenzie.co.uk/round1#:~:text=model%20should%20answer.-,Using%20newlines,-We%20saw%20many.  [Online; accessed 4-June-2024].
  * McKone (1999) Kathleen E McKone. 1999.  Analysis of student feedback improves instructor effectiveness.  _Journal of Management Education_ , 23(4):396–415.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with bert.  _arXiv preprint arXiv:1901.04085_.
  * Pan et al. (2019) Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019.  Recent advances in neural question generation.  _arXiv preprint arXiv:1905.08949_.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  _Advances in neural information processing systems_ , 34:11054–11070.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Poria et al. (2021) Soujanya Poria, Navonil Majumder, Devamanyu Hazarika, Deepanway Ghosal, Rishabh Bhardwaj, Samson Yu Bai Jian, Pengfei Hong, Romila Ghosh, Abhinaba Roy, Niyati Chhaya, et al. 2021.  Recognizing emotion cause in conversations.  _Cognitive Computation_ , 13:1317–1332.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392.
  * Reimers and Gurevych (2019a) Nils Reimers and Iryna Gurevych. 2019a.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Reimers and Gurevych (2019b) Nils Reimers and Iryna Gurevych. 2019b.  Sentence-bert: Sentence embeddings using siamese bert-networks.
  * Ren et al. (2021) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021.  Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking.  _arXiv preprint arXiv:2110.07367_.
  * Roberts et al. (2021) Kirk Roberts, Dina Demner-Fushman, Ellen M Voorhees, Steven Bedrick, and Willian R Hersh. 2021.  Overview of the trec 2021 clinical trials track.  In _Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021)_.
  * Sachan et al. (2022) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022.  Improving passage retrieval with zero-shot question generation.  _arXiv preprint arXiv:2204.07496_.
  * Schütze et al. (2008) Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008.  _Introduction to information retrieval_ , volume 39.  Cambridge University Press Cambridge.
  * Soboroff (2021) Ian Soboroff. 2021.  Overview of trec 2021.  In _30th Text REtrieval Conference. Gaithersburg, Maryland_.
  * Soboroff et al. (2018) Ian Soboroff, Shudong Huang, and Donna Harman. 2018.  Trec 2018 news track overview.  In _TREC_ , volume 409, page 410.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  _arXiv preprint arXiv:2104.08663_.
  * Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015.  An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.  _BMC bioinformatics_ , 16(1):1–28.
  * Voorhees et al. (2021) Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021.  Trec-covid: constructing a pandemic information retrieval test collection.  In _ACM SIGIR Forum_ , volume 54, pages 1–12. ACM New York, NY, USA.
  * Voorhees (2005) Ellen M Voorhees. 2005.  The trec robust retrieval track.  In _ACM SIGIR Forum_ , volume 39, pages 11–20. ACM New York, NY, USA.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  Gpt-j-6b: A 6 billion parameter autoregressive language model.
  * Wang et al. (2023) Rose Wang, Pawan Wirawarn, Noah Goodman, and Dorottya Demszky. 2023.  Sight: A large annotated dataset on student insights gathered from higher education transcripts.  In _Proceedings of Innovative Use of NLP for Building Educational Applications_.
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-end neural ad-hoc ranking with kernel pooling.  In _Proceedings of the 40th International ACM SIGIR conference on research and development in information retrieval_ , pages 55–64.
  * Yang et al. (2015) Yi Yang, Wen-tau Yih, and Christopher Meek. 2015.  Wikiqa: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 2013–2018.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhuang et al. (2021) Shengyao Zhuang, Hang Li, and Guido Zuccon. 2021.  Deep query likelihood model for information retrieval.  In _Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part II 43_ , pages 463–470. Springer.
  * Zhuang and Zuccon (2021) Shengyao Zhuang and Guido Zuccon. 2021.  Tilde: Term independent likelihood model for passage re-ranking.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1483–1492.
  * Ziems et al. (2023) Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023.  Can large language models transform computational social science?  _arXiv preprint arXiv:2305.03514_.

##  Appendix A Computational Setup

We ran our experiments on a Slurm-based university compute cluster on A6000
machines. The experiments varied in length in time—some took less than an hour
to run (e.g., the random baselines), while others took a few days to run
(e.g., the ATE likelihood-based methods on Lecture).

##  Appendix B Lecture annotation interface

Figure 7 shows the interface used for annotating the Lecture dataset.

Figure 7: Annotation interface

##  Appendix C Contextualized prefixes for scoring

This section describes the prompts used for the likelihood-based retrieval
methods and gpt-3.5-turbo-16k.

The prompts used for gpt-3.5-turbo-16k follow the practices in works from NLP,
education and social sciences (McKenzie, 2023; Library, 2023; Ziems et al.,
2023; Wang et al., 2023). Specifically, we enumerate the sentences in the
corpus as multiple-choice options and each option is separated by a newline.
We add context for the task at the start of the prompt, and the constraints of
outputting a JSON-formatted text for the task at the end of the prompt. We
found the model to be reliable in outputting the text in the desirable format.

###  C.1 Lecture

For the likelihood-based retrieval methods, the sentences are concatenated by
spaces and “A teacher is teaching a class, and a student asks a
question.$\backslash$nTeacher: ” is prepended to the corpus. Because the text
comes from transcribed audio which is not used in training dataset of the PLMs
we use in our work, we found it important for additional context to be added
in order for the probabilities to be slightly better calibrated. For the
query, “Student: ” is prepended to the text. For example, $X=$ “A teacher is
teaching a class, and a student asks a question.$\backslash$n Teacher:
[sentence 1] [sentence 2] …”, and $q=$ “Student: [query]”.

The prompt used for gpt-3.5-turbo-16k is in Figure 8.

gpt-3.5-turbo-16k prompt for Lecture Consider the following lecture
transcript: {line-numbered transcript} Now consider the following question:
{query} Which of the transcript lines most likely provoked this question? If
there are multiple possible answers, list them out. Format your answer as:
["line number": integer, "reason": "reason for why this line most likely
caused this query", …] Figure 8:  gpt-3.5-turbo-16k prompt for Lecture. For
the line-numbered transcript, “Teacher: ” is prepended to each sentence, the
sentences are separated by line breaks, and each line begins with its line
number. For the query, “Student: ” is prepended to the text. For example, a
line-numbered article looks like “0. Teacher: [sentence 1]$\backslash$n1.
Teacher: [sentence 2]$\backslash$n2. Teacher: [sentence 3] …”, and the query
looks like “Student: [query]”.  gpt-3.5-turbo-16k prompt for News Article
Consider the following article: {line-numbered article} Now consider the
following question: {query} Which of the article lines most likely provoked
this question? If there are multiple possible answers, list them out. Format
your answer as: ["line number": integer, "reason": "reason for why this line
most likely caused this query", …] Figure 9:  gpt-3.5-turbo-16k prompt for
News Article. For the line-numbered article, “Text: ” is prepended to each
sentence, the sentences are separated by line breaks, and each line begins
with its line number. For the query, “Question: ” is prepended to the text.
For example, a line-numbered article looks like “0. Text: [sentence
1]$\backslash$n1. Text: [sentence 2]$\backslash$n2. Text: [sentence 3] …”, and
the query looks like “Question: [question]”.  gpt-3.5-turbo-16k prompt for
Conversation Consider the following conversation: {line-numbered conversation}
Now consider the following line: {query} The speaker felt {emotion} in this
line. Which of the conversation turns (lines) most likely caused this emotion?
If there are multiple possible answers, list them out. Format your answer as:
["line number": integer, "reason": "reason for why this line most likely
caused this emotion", …] Figure 10:  gpt-3.5-turbo-16k prompt for
Conversation. For the line-numbered conversation, the speaker is added to each
turn, the turns are separated by line breaks, and each line begins with its
line number. For the query, the speaker is also added. For example, a line-
numbered conversation may look like “0. Speaker A: [utterance]$\backslash$n1.
Speaker B: [utterance]$\backslash$n2. Speaker A: [utterance] …”, and the query
may look like “Speaker A: [query]”.

###  C.2 News Article

For the likelihood-based retrieval methods, the sentences are concatenated by
spaces and “Text: ” is prepended to the corpus. For the query, “Question: ” is
prepended to the text. For example, $X=$ “Text: [sentence 1] [sentence 2] …”,
and $q=$ “Question: [question]”.

The prompt used for gpt-3.5-turbo-16k is in Figure 9.

###  C.3 Conversation

For the likelihood-based retrieval methods, the speaker identity is added to
the text, and the turns are separated by line breaks. For the query, the same
format is used. For example, $X=$ “Speaker A: [utterance]$\backslash$nSpeaker
B: [utterance]”, and $q=$ “Speaker A: [query]”.

The prompt used for gpt-3.5-turbo-16k is in Figure 10.