# Determination of Thermal Conductivity of phase pure 10H-SiC Thin Films by
non-destructive Raman Thermometry

Madhusmita Sahoo<EMAIL_ADDRESS><EMAIL_ADDRESS>Surface and
Sensors Studies Division, Materials Science Group, A CI of HBNI, IGCAR,
Kalpakkam 603102, India  Kalyan Ghosh  School of Physical Sciences, National
Institute of Science Education and Research (NISER) Bhubaneswar, An OCC of
Homi Bhabha National Institute, Jatni-752050, Odisha, India  Swayamprakash
Sahoo  School of Physical Sciences, National Institute of Science Education
and Research (NISER) Bhubaneswar, An OCC of Homi Bhabha National Institute,
Jatni-752050, Odisha, India  Pratap K. Sahoo<EMAIL_ADDRESS>School
of Physical Sciences, National Institute of Science Education and Research
(NISER) Bhubaneswar, An OCC of Homi Bhabha National Institute, Jatni-752050,
Odisha, India  Center for Interdisciplinary Sciences (CIS), NISER Bhubaneswar,
HBNI, Jatni-752050, Odisha, India  Tom Mathews  Surface and Sensors Studies
Division, Materials Science Group, A CI of HBNI, IGCAR, Kalpakkam 603102,
India  Sandip Dhara  Surface and Sensors Studies Division, Materials Science
Group, A CI of HBNI, IGCAR, Kalpakkam 603102, India

(August 31, 2024)

###### Abstract

10 H SiC thin films are potential candidates for devices that can be used in
high temperature and high radiation environment. Measurement of thermal
conductivity of thin films by a non-invasive method is very useful for such
device fabrication. Micro-Raman method serves as an important tool in this
aspect and is known as Raman Thermometry. It utilises a steady-state heat
transfer model in a semi-infinite half space and provides for an effective
technique to measure thermal conductivity of films as a function of film
thickness and laser spot size. This method has two limiting conditions i.e.
thick film limit and thin film limit. The limiting conditions of this model
was explored by simulating the model for different film thicknesses at
constant laser spot size. 10H SiC films of three different thicknesses i.e.
104, 135 and 156 nm were chosen to validate the thin film limiting condition.
It was found that the ideal thickness at which this method can be utilised for
calculating thermal conductivity is 156 nm. Thermal conductivity of 156 nm
film is found to be 102.385 $(Wm^{-1}K^{-1})$.

††preprint: AIP

##  I INTRODUCTION

Energy harvesting utilising ionising and non ionising radiation sources have
generated a lot of interest in the scientific community in recent years.
Amongst various methods and materials, it is particularly challenging to find
a suitable material for utilising ionising radiation. SiC has been considered
as a suitable material in this regard due to its ability to perform in high
temperature and high radiation conditions where conventional semiconductor
devices cannot perform adequately.Nava  _et al._ (2008) It is important that
the thermal conductivity is measured by a non destructive method after the
whole device is assembled. It becomes all the more important for an
radioactive environment, where one would need necessary provisions and
methodology to measure thermal conductivity intermittently to know the health
of the device. While many studies have been done for more common polytype like
3C, 4H, and 6H, higher hexagonal polytypes of SiC remain unexplored.Cheng Z.
(2022); Slack (1964); Protik  _et al._ (2017) Hence, it is desirable that a
phase pure SiC is synthesized so that defect-phonon interaction can be
minimised leading to higher thermal conductivity.Cheng Z. (2022) In this
study, we have focused on determining the thermal conductivity of phase pure
10H SiC. Several methods have been reported so far to determine thermal
conductivity of dielectric materials. These methods are steady state
method,Zhang and Grigoropoulos (1995) 3 $\omega$ method,Cahill, Katiyar, and
Abelson (1994) photo acoustic method,Swimm (1983) thermal microscopy method
Callard  _et al._ (1999) and thermo-reflectance method. Burzo, Komarov, and
Raad (2002); Komarov  _et al._ (2003) These methods are invasive in nature
wherein the original sample is either damaged or extensive sample preparation
and data analysis is required. Hence, Perichon et al. demonstrated a non
invasive and non destructive micro-Raman method to determine thermal
conductivity of thick films. Périchon  _et al._ (2000) However, necessary
condition for this method is the sample thickness must be one order higher
than the laser diameter. This implied that only the films having thickness in
the order of microns were suitable for the method proposed by Perichon et
al.Périchon  _et al._ (2000) Huang et al. modified the methodology to evaluate
thermal conductivity of thin films having thickness in submicrometer to
nanometer range.Huang S. (2009) Prior to this study, only few researchers have
used the method proposed by Huang et al. for determining thermal conductivity
of Si based devices, 2D graphene and biological samples.Gan, Samvedi, and
Tomar (2015); Freedman  _et al._ (2017); Zhang, Gan, and Tomar (2014) In this
report, we have used the method proposed by Huang et al. for determining
thermal conductivity of 10H SiC thin films in the range of 104-156 nm, which
is being reported for the first time to the best of our knowledge.

##  II EXPERIMENTAL DETAILS

SiC thin films were deposited on cleaned Si substrates using RF magnetron
sputtering. The deposition parameters such as RF power, deposition time, gas
flow, target to substrate distance are as follows. RF power = 90 Watt,
reflectant power = 0 Watt, deposition time = 30 mins, gas flow(Ar) =15 sccm
and target to substrate distance (3.5inch). Thickness of films were varied by
changing only chamber pressure during deposition. Three different thickness of
104 nm, 153 nm and 156 nm were obtained at a chamber pressure of 1*10-2,
2*10-2 and 5*10-2 mbar, respectively. The thickness of the films were measured
by cross sectional field emission scanning electron microscopy (FESEM) (SIGMA
model, Carl Zeiss). Crystallographic phase of the films were determined from
GIXRD by using Rigaku Smart Lab X-ray diffractometer with monochromatic Cu
K-$\alpha$ radiation ($\lambda$ = 1.5418 Å). Raman spectra of the samples were
acquired using a 532 nm laser as the excitation source (Jobin-Yvon LabRam HR
Evolution, Horiba). The spectra were collected using a spectrometer coupled
with a CCD based detector in back-scattered geometry with 50x lens.
Temperature dependent Raman spectra were recorded using a high temperature
Linkam stage.

##  III Crystallographic and Morphological Studies

The crystallographic phase of the RF magnetron sputtered films were determined
by matching peaks of X-ray diffraction pattern with peaks from ICDD card no
89-2214. A representational grazing incidence X-ray diffraction pattern of 156
nm thick 10H SiC is shown in figure 1. The peak at 44.5∘ and 28.3∘ corresponds
to (018) and (008) planes of 10 H SiC.

Figure 1: Grazing incidence X-ray diffraction spectra of 156 nm thick 10H SiC
film

Surface morphology of 156 nm 10H SiC film is shown in fig 2, which indicated
smooth and continuous film formation. All other films were also found to
possess similar morphological features. Cross sectional FESEM is shown in
inset of fig 2. It reveals uniform thickness across the substrate. The
thicknesses measured from cross-sectional FESEM is used in calculation of
thermal conductivity of the film (kf), which discussed in detail in Section
IV.

Figure 2: Surface morphology of SiC samples using FESEM (Inset) Cross-
sectional thickness of a deposited SiC sample.

##  IV Raman Thermometry

###  IV.1 The model

It has been a general practice for researchers to combine temperature
dependent Raman peak shift within the ambit of thermal scanning probe
microscopy and calculate thermal conductivity. However, heating source in both
the methods are different. Therefore in this work we have followed the
methodology developed by Huang et al. that involves a separate mathematical
model to account for both the local heat induced due to a Gaussian laser beam
and the thickness of the sample.Huang S. (2009)

In this method, a Gaussian laser beam is considered to be incident on a
relatively thick substrate. It is assumed that heat transfer is complete and
heat loss is minimum. This reduces the problem to a steady state heat transfer
problem from a circular region in a semi-infinite space.

The temperature across the film sample then satisfies the Laplacian

$\triangledown^{2}t(r,z)=0$ (1)

Applying necessary boundary conditions Huang et al. deduced the formula for
thermal conductivity of the thick film to be

$k=\frac{\left[I_{0}\left(1\right)+I_{1}\left(1\right)\right]\cdot
P}{\sqrt{2\pi}er_{0}\triangle T}$ (2)

where $I_{0}$ and $I_{1}$ are modified Bessel’s equations of zeroth and first
order respectively, which are derived by Dryden P represents power of laser,
$r_{0}$ is radius of laser beam and $\triangle T$ is induced temperature rise
in sample due to laser irradiation. Dryden (1983) The presence of laser beam
as a heating source is quintessential to the model proposed. Hence, it becomes
necessary to know exact spot size of the laser beam on the sample. The spot
size for a Gaussian laser beam is calculated by equation 3, which is given
below.

$r_{o}=\frac{\lambda}{\pi\cdot N.A.}$ (3)

where $\lambda$ represents the wavelength of the laser beam and N.A. is the
numerical aperture of the lens, $r_{o}$ is required radius of the laser spot.

However, above basic equation focuses on bulk films with thickness at least
one order of magnitude higher than the laser beam diameter. Extending this to
a thin film model, thermal conductivity of the film in eq. 2 became apparent
thermal conductivity ($k_{app}$) of the entire sample as given below.

$k_{app}=\frac{\left[I_{0}\left(1\right)+I_{1}\left(1\right)\right]\cdot
P}{\sqrt{2\pi}er_{0}\triangle T}$ (4)

Heat flow across the sample must still satisfy equation 1. Hence, a Laplacian
was applied to substrate and film separately. Subsequently, necessary boundary
conditions were applied to find a relation between thermal conductivity of the
substrate($k_{s}$), thin film($k_{f}$) and the entire sample($k_{app}$) giving
rise to the following equation.

$\frac{k_{s}}{k_{app}}=1+\sqrt{\frac{2}{\pi}}\frac{e-e^{-1}}{\left[I_{0}\left(1\right)+I_{1}\left(1\right)\right]}\frac{\delta}{r_{0}}\frac{k_{s}}{k_{f}}\left[1-\left(\frac{k_{f}}{k_{s}}\right)^{2}\right]$
(5)

where $\delta$ is the thickness of the film.

Equation 5 shows a dependency on $\delta/r_{o}$ ratio of the sample. Huang et
al. have further simplified equation 4 by assuming $k_{f}$ is less than
$k_{s}$. This holds true for SiO2, which was the material of interest to them.
However, there is no literature value available for bulk 10H SiC. Hence, a
simulation was carried out using C programme to explore variation of kf and
kapp with a change in $\delta/r_{o}$ ratio to circumvent the limiting
condition used by Huang et al. The substrate conductivity(ks) is used as a
normalizing factor for both of them. The results of this simulation is given
in figure 3.

Figure 3:  Relationship pattern between $k_{app}$ and $k_{f}$ for various
values of $\frac{\delta}{r_{0}}$.

It is observed from figure 3 that for $\delta/r_{o}$ ratio of 0.01 (very thin
film region), curve shows a highly concave nature and the effect of $k_{app}$
is predominant over $k_{f}$. The concavity of the curve decreases for thicker
films as $\delta/r_{o}$ value increases to 0.4. A reversal of nature is
observed at $\delta/r_{o}$ =0.4. This marks transition of film from thin film
to bulk and can be considered as the critical value for the associated
material beyond which assumptions pertaining to thin film limit will not work.
This can be explained by the fact that $\delta/r_{o}$ ratio increases with
increase in film thickness. As a result, heat flow across the film increases
thereby increasing significance of $k_{f}$ parameter. In very thin films, heat
flow occurs beyond the film in entire sample making $k_{app}$ value
significant. Conversely, for bulk films, heat transfer is limited mostly to
the film making $k_{f}$ value significant. Simulation of equation 5 provides
us a generous plot to observe this transition as we move from thin to bulk
region as presented in figure 3.

###  IV.2 Calculation of thermal conductivity of 10H SiC films

Thermal conductivity of 10H SiC films were calculated by utilising equation 4
and 5 of section IV A. It is known that thermal conductivity is a function of
temperature of the film when it is subjected to a heat gradient. In the
current study, shift in Raman peak position is indicative of thermal gradient
of the film. The thermal gradient was achieved by irradiating the film with a
532 nm laser at two different laser power i.e 1.25 mW and 5 mW. Raman spectra
of films were first measured by keeping laser power at 1.25 mW, wherein laser
induced sample heating is presumed to be negligible. Sample heating was then
carried out by increasing the temperature from 298K to 773K using a Linkam
stage and measuring the corresponding Raman spectrum. Raman spectra thus
obtained had a Raman peak at 960cm-1, which is a characteristic of hexagonal
SiC polytype.Sasinková  _et al._ (2015) The shift in this 960 cm-1 peak for
156 nm 10H SiC film with respect to change in temperature is shown in fig 4.
It is observed that with increase in temperature, peak shifted towards lower
wave number(red shift). The red shift is due to anharmonicity induced in the
film with increase in temperature. Similar red shift was also observed for
other two films. Subsequently, a calibration plot was generated by plotting
Raman peak shift against temperature and is shown in fig 5. From the linear
plot one can observe that with increase in substrate temperature the Raman
peak shift reduced. The slope became shallower with higher thickness,
indicating a smaller temperature difference between two consecutive points.
Subsequently, with higher thickness the value of $\Delta$T reduces and value
of kapp increases. A separate set of Raman spectra were recorded at room
temperature by keeping laser power at 5mW for deducing apparent thermal
conductivity (kapp). The sample heating due to change in laser power was
determined by observing the change in peak position and subsequently utilising
the calibration plot to find the corresponding temperature change in the film,
which is $\Delta$T. This temperature difference is then used in equation 4 of
section IV A to calculate the apparent thermal conductivity of the sample
(kapp), which includes thermal conductivity of film and substrate. Since,
thickness($\delta$) for each film is known from cross-sectional FESEM images,
we calculated thermal conductivity of the film (Kf) by using equation 5 of
section IV A. Two roots were obtained for the quadratic equation in each case.
The negative root was ignored on account of physical impossibility and only
the positive root for Kf is reported in Table 1.

Figure 4:  SiC Raman band shift with temperature of 156 nm thick 10H-SiC film
measured irradiated with 1.25 mW laser power

As presented in Table 1 $\delta/r_{o}$ values for films of thickness 104, 135
and 156 nm are 0.18, 0.23 and 0.26, respectively. These values lie in thin
film limit as indicated in fig 3. It can be observed in fig 3 that when
$\delta/r_{o}$ varies from 0.2 to 0.3, the relationship between Kapp and Kf
tends to become linear implying Kapp becomes equal to Kf. This is corroborated
in our calculation. It can be observed in table 1 that with increase in
$\delta/r_{o}$ value from 0.18 to 0.26, Kapp becomes comparable to Kf for 156
nm film.

Figure 5: Raman Shift Vs Temperature calibration plots of 104, 135, and 156 nm 10H-SiC films Table 1: ${\delta}$/$r_{o}$, calculated Kapp and Kf values for 10H SiC films of different thicknesses | Thickness
---
(nm)
$\delta/r_{0}$ |  $\Delta$ T |  | $K_{app}$
---
(using eq. 4)
| $K_{f}$
---
$(Wm^{-1}K^{-1})$
[using eq. 5]
104 | 0.18 | 36.79 | 33.81 | 11.87
135 | 0.23 | 32.69 | 38.05 | 17.86
156 | 0.26 | 10.97 | 113.32 | 102.36

This also indicates that within thin film limit, thermal conductivity value
can vary within range of 11- 103 $(Wm^{-1}K^{-1})$. This gives us an idea
about the range of thermal conductivity values for 10H SiC films below 200 nm
thickness. There have been various reports on thermal conductivity of SiC.
Slack et al. observed that thermal conductivity of SiC monocrystal at room
temperature is 490 $(Wm^{-1}K^{-1})$. Slack (1964)

Composite materials in SiC was found to have reduced thermal conductivity
value of 252–270 $(Wm^{-1}K^{-1})$ as compared to monocrystals studied by
slack et al.Nakano  _et al._ (2004) However there is no report of thermal
conductivity of phase pure SiC thin film in any polytype. So, there is no
reference to compare thermal conductivity of thin film (Kf) and we are hereby
reporting and effective way to determine thermal conductivity of phase pure
SiC thin films by a nondestructive method for the first time. As per our
observation, thermal conductivity of 10H SiC is 102.36 $(Wm^{-1}K^{-1})$.

##  V CONCLUSION

In this work, we have calculated thermal conductivity of 10H SiC films of
three thicknesses by utilising Raman Thermometry. A novel thin film limit was
worked out for films in nanometer range by extrapolating the model given by
Huang et al. and the same was applied to calculate thermal conductivity of 10H
SiC film. Our observation confirms thickness dependence of thermal
conductivity which can vary from 11 -102 $(Wm^{-1}K^{-1})$. It was found that
the 156 nm film is the ideal thickness for 10H SiC films to calculate thermal
conductivity by this method as its Kapp is comparable to Kf.

###### Acknowledgements.

Madhusmita Sahoo acknowledges Dr Shaju Albert for his support in carrying out
the project. The authors from NISER acknowledge the Department of Atomic
Energy (DAE), India for supporting this work through the project RIN-4001.

## Statements and Declarations

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## References

  * Nava  _et al._ (2008) F. Nava, G. Bertuccio, A. Cavallini, and E. Vittone, “Silicon carbide and its use as a radiation detector material,” Measurement Science and Technology 19, 102001 (2008).
  * Cheng Z. (2022) K. K. e. a. Cheng Z., Liang J., “High thermal conductivity in wafer-scale cubic silicon carbide crystals.” Nature Communications 13, 7201 (2022).
  * Slack (1964) G. A. Slack, “Thermal conductivity of pure and impure silicon, silicon carbide, and diamond,” Journal of Applied physics 35, 3460–3466 (1964).
  * Protik  _et al._ (2017) N. H. Protik, A. Katre, L. Lindsay, J. Carrete, N. Mingo, and D. Broido, “Phonon thermal transport in 2h, 4h and 6h silicon carbide from first principles,” Materials Today Physics 1, 31–38 (2017).
  * Zhang and Grigoropoulos (1995) X. Zhang and C. P. Grigoropoulos, “Thermal conductivity and diffusivity of free-standing silicon nitride thin films,” Review of scientific instruments 66, 1115–1120 (1995).
  * Cahill, Katiyar, and Abelson (1994) D. G. Cahill, M. Katiyar, and J. Abelson, “Thermal conductivity of a-si: H thin films,” Physical review B 50, 6077 (1994).
  * Swimm (1983) R. T. Swimm, “Photoacoustic determination of thin-film thermal properties,” Applied Physics Letters 42, 955–957 (1983).
  * Callard  _et al._ (1999) S. Callard, G. Tallarida, A. Borghesi, and L. Zanotti, “Thermal conductivity of sio2 films by scanning thermal microscopy,” Journal of non-crystalline solids 245, 203–209 (1999).
  * Burzo, Komarov, and Raad (2002) M. G. Burzo, P. L. Komarov, and P. E. Raad, “Influence of the metallic absorption layer on the quality of thermal conductivity measurements by the transient thermo-reflectance method,” Microelectronics journal 33, 697–703 (2002).
  * Komarov  _et al._ (2003) P. L. Komarov, M. G. Burzo, G. Kaytaz, and P. E. Raad, “Transient thermo-reflectance measurements of the thermal conductivity and interface resistance of metallized natural and isotopically-pure silicon,” Microelectronics journal 34, 1115–1118 (2003).
  * Périchon  _et al._ (2000) S. Périchon, V. Lysenko, P. Roussel, B. Remaki, B. Champagnon, D. Barbier, and P. Pinard, “Technology and micro-raman characterization of thick meso-porous silicon layers for thermal effect microsystems,” Sensors and Actuators A: Physical 85, 335–339 (2000).
  * Huang S. (2009) F. X. e. a. Huang S., Ruan Xd., “Measurement of the thermal transport properties of dielectric thin films using the micro-raman method.” J. Zhejiang Univ. Sci. A 10 (2009).
  * Gan, Samvedi, and Tomar (2015) M. Gan, V. Samvedi, and V. Tomar, “Raman spectroscopy-based investigation of thermal conductivity of stressed silicon microcantilevers,” Journal of Thermophysics and Heat Transfer 29, 845–857 (2015).
  * Freedman  _et al._ (2017) K. J. Freedman, G. Goyal, C. W. Ahn, and M. J. Kim, “Substrate dependent ad-atom migration on graphene and the impact on electron-beam sculpting functional nanopores,” Sensors 17, 1091 (2017).
  * Zhang, Gan, and Tomar (2014) Y. Zhang, M. Gan, and V. Tomar, “Raman thermometry based thermal conductivity measurement of bovine cortical bone as a function of compressive stress,” Journal of Nanotechnology in Engineering and Medicine 5 (2014).
  * Dryden (1983) J. R. Dryden, “The Effect of a Surface Coating on the Constriction Resistance of a Spot on an Infinite Half-Plane,” Journal of Heat Transfer 105, 408–410 (1983), https://asmedigitalcollection.asme.org/heattransfer/article-pdf/105/2/408/5573361/408_1.pdf .
  * Sasinková  _et al._ (2015) V. Sasinková, J. Huran, A. Kleinová, P. Boháček, J. Arbet, and M. Sekáčova, “Raman spectroscopy study of SiC thin films prepared by PECVD for solar cell working in hard environment,” in  _Reliability of Photovoltaic Cells, Modules, Components, and Systems VIII_, Vol. 9563, edited by N. G. Dhere, J. H. Wohlgemuth, and R. Jones-Albertus, International Society for Optics and Photonics (SPIE, 2015) p. 95630V.
  * Nakano  _et al._ (2004) H. Nakano, K. Watari, Y. Kinemuchi, K. Ishizaki, and K. Urabe, “Microstructural characterization of high-thermal-conductivity SiC ceramics,” Journal of the European Ceramic Society 24, 3685–3690 (2004).


# On the Thermodynamics of Gravitational Radiation

S. C. Ulhoa<EMAIL_ADDRESS>Instituto de Física, Universidade de Brasília,
70910-900, Brasília, DF, Brazil  Canadian Quantum Research Center,
204-3002 32 Ave Vernon, BC V1T 2L7 Canada  F. L. Carneiro
<EMAIL_ADDRESS>Universidade Federal do Norte do Tocantins,
Araguaína, TO, Brazil  J. W. Maluf<EMAIL_ADDRESS>Instituto de Física,
Universidade de Brasília, 70910-900, Brasília, DF, Brazil

###### Abstract

This article deals with the thermodynamics of gravitational radiation arising
from the Bondi-Sachs space-time. The equation of state found allows us to
conclude that the dependence of the energy density on the temperature is a
quadratic power of the latter. Such a conclusion is possible once the
consequences of the first law of thermodynamics are analyzed. Then, in analogy
to electromagnetic radiation, the same approach as used by Planck to obtain
the quantum of energy of the gravitational radiation is proposed. An energy
for the graviton proportional to the cubic frequency is found. The graviton is
here understood as the quantum of gravitational energy.

##  I Introduction

In 1884 Boltzmann derived Stefan’s law, in what would come to be known as the
Stefan-Boltzmann law [1]. This law stated that the energy density of
electromagnetic radiation was proportional to the fourth power of the
temperature. The Maxwell’s equations alone do not allow one to conclude that
electromagnetic radiation is thermal, since such equations do not depend on
temperature. They however establish an equation of state for the field, that
is, the pressure is equal to one third of the energy. Boltzmann only
considered the thermodynamic consequences of the electromagnetic field
equation of state to demonstrate the dependence of energy density on
temperature. In his seminal paper published in 1901 [2], Planck both showed
how to obtain the constant of proportionality of the Stefan-Boltzmann law and
introduced the Quanta theory. He considered that blackbody radiation could be
described by a collection of harmonic oscillators whose energy was discrete.
Each quantum of energy was proportional to the frequency of the radiation. The
quantum of energy of electromagnetic radiation became known as the photon and
the constant of proportionality as Planck’s constant, $\displaystyle h$. That
constant has proved fundamental to every quantum process since then.

The gravitational field has several similarities with the electromagnetic
field, for instance the Einstein equations also do not explicitly depend on
the temperature. So there would be no reason to expect any thermal radiation
in the framework of such a description of the gravitational field. On the
other hand, the very definition of gravitational energy, in the metric
formulation of the field, is quite controversial. In this context, no
definition achieves the properties expected of such a quantity. Namely, the
energy must be independent of the coordinate system and it must be sensitive
to the choice of the reference frame. There is, on the other hand, a
dynamically equivalent description of general relativity, known as
Teleparallelism Equivalent to General Relativity (TEGR), in which quantities
such as energy, momentum, and angular momentum are well defined [3]. Armed
with such a description of the gravitational field, one can consider the
thermodynamic consequences of a specific equation of state. In particular, the
gravitational radiation is of special interest because, like the
electromagnetic field, it can reveal quantum aspects of the physical system.

In the same way that Maxwell’s theory allows us to establish an explicit
relationship between the energy density $\displaystyle\epsilon$ and the
electromagnetic radiation pressure $\displaystyle p$, and consequently, derive
a relation between energy density and temperature based on fundamental
thermodynamics, our aim is to establish a thermodynamic description of
gravitational radiation. In order to describe the gravitational radiation, we
consider the Bondi-Sachs spacetime, which represents a source emitting
gravitational waves. The energy of this spacetime consists of a rapidly
decaying mass aspect and another term that decays more slowly. We focus on the
metric in the limit of being far from the source, but not at infinity. In this
limit, the contribution from the source (or mass aspect) vanishes, while the
other term remains finite. Consequently, we identify this remaining term as
the energy of the gravitational radiation. By evaluating the gravitational
pressure in the same limit, we can compare the two expressions and obtain an
equation of state for gravitational radiation. From this point, we pursue
Planck’s idea and, through the application of the first law of thermodynamics,
establish a relationship between the energy density of gravitational radiation
and its temperature. Thus, we obtain a purely macroscopic and classical
equation of state for gravitational radiation. Based on this equation of
state, we calculate the Gibbs energy, similar to what is observed for
electromagnetic radiation. Consequently, we adopt a statistical model for the
radiation and arrive at an intriguing result: a cubic dependence between a
hypothetical gravitational quantum (graviton) and the frequency of the
radiation. The article is divided as follows. In section II, the Bondi-Sachs
spacetime is described and the results of the gravitational energy and
pressure of the radiation are given. In section III, Boltzmann and Planck
procedures are used for gravitational radiation. With this, the energy of the
graviton is established. Finally, in the last section, the final
considerations are presented.

##  II Bondi-Sachs Space-Time

The purpose of this section is to briefly present the Bondi-Sachs space-time,
in addition to the gravitational energy-momentum calculated within the
framework of TEGR in ref. [4]. It describes the gravitational radiation at
spacelike and null infinities. The line-element, in the natural unities
system, has the form

$\displaystyle\displaystyle ds^{2}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle
g_{00}\,du^{2}+g_{22}\,d\theta^{2}+g_{33}\,d\phi^{2}+2g_{01}\,du\,dr+2g_{02}\,du\,d\theta+2g_{03}\,du\,d\phi+2g_{23}d\theta\,d\phi\,,$
(1)

where $\displaystyle u=t-r$ is the retarded time and the asymptotic behavior
at $\displaystyle r\rightarrow\infty$ of the metric tensor is given by

$\displaystyle\displaystyle g_{00}$ $\displaystyle\displaystyle\simeq$
$\displaystyle\displaystyle-1+{{2M}\over r}\,,$ $\displaystyle\displaystyle
g_{01}$ $\displaystyle\displaystyle\simeq$
$\displaystyle\displaystyle-1+{{c^{2}+d^{2}}\over{2r^{2}}}\,,$
$\displaystyle\displaystyle g_{02}$ $\displaystyle\displaystyle\simeq$
$\displaystyle\displaystyle l+{1\over r}(2cl+2d\bar{l}-p)\,,$
$\displaystyle\displaystyle g_{03}$ $\displaystyle\displaystyle\simeq$
$\displaystyle\displaystyle\bar{l}\sin\theta+{1\over
r}(-2c\bar{l}+2dl-\bar{p})\sin\theta\,,$ $\displaystyle\displaystyle g_{22}$
$\displaystyle\displaystyle\simeq$ $\displaystyle\displaystyle
r^{2}+2cr+2(c^{2}+d^{2})\,,$ $\displaystyle\displaystyle g_{33}$
$\displaystyle\displaystyle\simeq$
$\displaystyle\displaystyle[r^{2}-2cr+2(c^{2}+d^{2})]\sin^{2}\theta\,,$
$\displaystyle\displaystyle g_{23}$ $\displaystyle\displaystyle\simeq$
$\displaystyle\displaystyle 2dr\sin\theta+{{4d^{3}}\over{3r}}\sin\theta\,,$
(2)

with

$\displaystyle l=\partial_{2}c+2c\,\cot\theta+\partial_{3}d\,\csc\theta\,,$

and

$\displaystyle\bar{l}=\partial_{2}d+2d\,\cot\theta-\partial_{3}c\csc\theta\,.$

Here $\displaystyle M(u,\theta)$,
$\displaystyle\partial_{0}c(u,\theta)=\partial c/\partial u$ and
$\displaystyle\partial_{0}d(u,\theta)$ are the mass aspect, the first and
second news functions respectively. The functions $\displaystyle p$ and
$\displaystyle\bar{p}$ are defined in references of [4]. They do not
contribute to the energy-momentum and therefore will not be detailed here. The
time derivative of the mass aspect yields the loss of mass, while the news
functions are interpreted as degrees of freedom of gravitational radiation. In
the TEGR the energy-momentum depends on the choice of the tetrad field, which
represents the choice of the observer. For that, a set of tetrads was chosen
that satisfies the condition $\displaystyle e_{(0)}\,^{i}=0$. With the help of
the tetrad field, the Weitzenböck connection can be defined, i.e.

$\displaystyle\Gamma_{\mu\lambda\nu}=e^{a}\,_{\mu}\partial_{\lambda}e_{a\nu}\,,$

whose antisymmetric part determines the following torsion tensor

$T^{a}\,_{\lambda\nu}=\partial_{\lambda}e^{a}\,_{\nu}-\partial_{\nu}e^{a}\,_{\lambda}\,.$
(3)

It is worth noting that the Weitzenböck connection relates to the Christoffel
symbols by the following mathematical identity

$\Gamma_{\mu\lambda\nu}={}^{0}\Gamma_{\mu\lambda\nu}+K_{\mu\lambda\nu}\,,$ (4)

where

$\displaystyle\displaystyle K_{\mu\lambda\nu}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\frac{1}{2}(T_{\lambda\mu\nu}+T_{\nu\lambda\mu}+T_{\mu\lambda\nu})\,,$
(5)

is the contorsion tensor. Such an identity allows writing the curvature scalar
in terms of torsions, i.e.

$eR(e)\equiv-e(\frac{1}{4}T^{abc}T_{abc}+\frac{1}{2}T^{abc}T_{bac}-T^{a}T_{a})+2\partial_{\mu}(eT^{\mu})\,,$
(6)

where $\displaystyle e=det(e^{a}\,_{\mu})$. It should be noted that the left-
hand side of the above expression is the Hilbert-Einstein Lagrangian density,
thus the TEGR Lagrangian density takes the following form

$\displaystyle\displaystyle\mathfrak{L}(e_{a\mu})$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle-\kappa\,e\,(\frac{1}{4}T^{abc}T_{abc}+\frac{1}{2}T^{abc}T_{bac}-T^{a}T_{a})-\mathfrak{L}_{M}$
(7) $\displaystyle\displaystyle\equiv$
$\displaystyle\displaystyle-\kappa\,e\Sigma^{abc}T_{abc}-\mathfrak{L}_{M}\;,$

where $\displaystyle\mathfrak{L}_{M}$ is the Lagrangian density of matter
fields, $\displaystyle\kappa=\frac{1}{16\,\pi}$ is the coupling constant in
natural units and $\displaystyle\Sigma^{abc}$ is defined by

$\Sigma^{abc}=\frac{1}{4}(T^{abc}+T^{bac}-T^{cab})+\frac{1}{2}(\eta^{ac}T^{b}-\eta^{ab}T^{c})\;.$
(8)

The derivative of this Lagrangian density with respect to the tetrad leads to
the following field equation

$\partial_{\nu}\left(e\Sigma^{a\lambda\nu}\right)=\frac{1}{4\kappa}e\,e^{a}\,_{\mu}(t^{\lambda\mu}+T^{\lambda\mu})\;,$
(9)

where

$t^{\lambda\mu}=\kappa\left[4\,\Sigma^{bc\lambda}T_{bc}\,^{\mu}-g^{\lambda\mu}\,\Sigma^{abc}T_{abc}\right]\,,$
(10)

which is interpreted as the energy-momentum of the gravitational field. Such a
definition has been shown to be quite consistent over the years. Hence the
energy-momentum vector is defined by

$P^{a}=\int_{V}d^{3}x\,e\,e^{a}\,_{\mu}(t^{0\mu}+T^{0\mu})\,.$ (11)

It is important to realize that the energy-momentum vector is invariant under
coordinate transformations and is covariant under Lorentz transformations.
This means that the zero component, identified with the energy, has all the
qualities expected for a consistent definition of gravitational energy.

Thus, for the Bondi-Sachs metric, the energy reads [4]

$P^{(0)}=4\kappa\int_{0}^{2\pi}d\phi\int_{0}^{\pi}d\theta\sin\theta\biggl{[}M+\partial_{0}F\biggr{]}\,,$
(12)

where

$F=-{1\over 4}\biggl{(}l^{2}+\bar{l}^{2}\biggr{)}+{1\over 2}c^{2}+d^{2}$ (13)

It is worth noting that the term $\displaystyle\partial_{0}F$ generalises the
standard Bondi-Sachs energy. The momentum is given by

$\displaystyle\displaystyle P^{(i)}$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle
4\kappa\int_{0}^{2\pi}d\phi\int_{0}^{\pi}d\theta\sin\theta\biggl{[}(M+\partial_{0}F)\hat{r}^{i}$
(14) $\displaystyle\displaystyle+{1\over
4}(l\partial_{0}M)\hat{\theta}^{i}+{1\over
4}(\bar{l}\partial_{0}M)\hat{\phi}^{i}\biggr{]}\,.$

It should be noted that the energy-momentum is presented in terms of its
components. The energy can also be expressed as

$P^{(0)}=\int d^{3}x\,\epsilon\,,$ (15)

where
$\displaystyle\epsilon=4\kappa\,\partial_{r}\biggl{[}M+\partial_{0}F\biggr{]}$,
assuming that a realistic physical system, like a radiating star, does not
have a singularity. That means $\displaystyle\epsilon$ is the volumetric
energy density. The derivative of the momentum with respect to time is
precisely the force, thus

$\frac{dP^{(i)}}{dt}=-\int dS_{j}\,\phi^{(i)j}\,,$ (16)

where $\displaystyle\phi^{(1)1}=p_{r}=-4\kappa\,\frac{\partial}{\partial
t}\biggl{[}M+\partial_{0}F\biggr{]}$ is the radial pressure [3]. There is,
therefore, a well-defined equation of state for this gravitational radiation

$\epsilon=p_{r}\,,$ (17)

that is, the energy density $\displaystyle\epsilon$ is equal to the radial
pressure $\displaystyle p_{r}$. Next, the consequences of this equation of
state are analyzed.

##  III Thermodynamics of Gravitational Radiation

The first law of thermodynamics is essentially a conservation law in which the
heat produced by a heat engine minus the work done by the engine equals the
internal energy of the system. Although historically established in the
context of heat engines, thermodynamics has a more fundamental character. It
was this fundamental feature of thermodynamics that led Boltzmann to propose
an ingenious interpretation of the entropy thermodynamic potential. Until
then, entropy accounted for the reversibility of thermodynamic phenomena.
Boltzmann proposed that it should be proportional to the logarithm of the
number of possible states of the system. The first law formulated in terms of
entropy is then

$TdS=dU+pdV\,,$ (18)

which in terms of the free energy $\displaystyle F=U-TS$ is

$dF=-SdT-pdV\,.$ (19)

It should be noted that the free energy is a thermodynamic potential which
depends on the variables temperature and volume $\displaystyle F\equiv
F(T,V)$, thus $\displaystyle S=-\left(\frac{\partial F}{\partial
T}\right)_{V}$ and $\displaystyle p=-\left(\frac{\partial F}{\partial
V}\right)_{T}$. Hence $\displaystyle\left(\frac{\partial S}{\partial
V}\right)_{T}=\left(\frac{\partial p}{\partial T}\right)_{V}$. Then the first
law of thermodynamics reads

$T\frac{dp}{dT}=\epsilon+p\,,$ (20)

where $\displaystyle\epsilon=\frac{\partial U}{\partial V}$ and $\displaystyle
p$ is the pressure. One should not confuse it with the quantity in the Bondi-
Sachs space-time. It is worth noting that such an equation yields the Stefan-
Boltzmann law for the electromagnetic radiation equation of state
$\displaystyle p=\epsilon/3$.

Let’s consider the equation of state for gravitational radiation obtained in
the previous section $\displaystyle\epsilon=p_{r}$, we identify the
thermodynamic energy $\displaystyle U$ with the gravitational energy
$\displaystyle P^{(0)}$, therefore, equation (20) reads

$T\frac{dp_{r}}{dT}=2p_{r}\,,$ (21)

which yields the following dependence of pressure on the temperature

$p_{r}=\sigma\,T^{2}$ (22)

where $\displaystyle\sigma$ is the integration constant. The energy has the
same dependence, $\displaystyle\epsilon=\sigma\,T^{2}$, while the entropy is
$\displaystyle S=2\sigma TV$. It should be noted that entropy is calculated
from the relation $\displaystyle\left(\frac{\partial S}{\partial
V}\right)_{T}=\left(\frac{\partial p}{\partial T}\right)_{V}$. As an immediate
consequence one can calculate the Gibbs free energy

$\displaystyle\displaystyle G$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle U-TS+PV$ (23) $\displaystyle\displaystyle=$
$\displaystyle\displaystyle 0\,.$

This result is very interesting, as it also occurs for the electromagnetic
radiation. In the latter case, the vanishing of $\displaystyle G$ implies the
vanishing of the chemical potential $\displaystyle\mu$, since $\displaystyle
G=\mu N$, where $\displaystyle N$ is the number of particles. Hence the number
of particles must be non-zero (or there would be no physical system), which
gave rise to the idea of photons for the electromagnetic field. This suggests
that the gravitational radiation may have a quantum of energy $\displaystyle
U_{0}$. Planck used Boltzmann’s concept of entropy to derive the photon
energy. Here we’re going to take a slightly different route to obtain the
“graviton” energy. Thus, it is considered that gravitational radiation can be
understood as a collection of discrete oscillators, each one with the energy
$\displaystyle U_{n}=nU_{0}$. Then the average value of the energy is

$\bar{U}=\frac{\sum_{n}U_{n}\exp{\left(\frac{U_{n}}{kT}\right)}}{\sum_{n}\exp{\left(\frac{U_{n}}{kT}\right)}}\,,$
(24)

where $\displaystyle k$ is the Boltzmann constant. That yields

$\bar{U}=\frac{U_{0}}{\exp{\left(\frac{U_{0}}{kT}\right)}-1}\,.$ (25)

The number of oscillators that exist between frequencies $\displaystyle\nu$
and $\displaystyle\nu+d\nu$ is $\displaystyle dN=8\pi V\nu^{2}d\nu$. That
leads to the following energy density

$\epsilon=\frac{8\pi U_{0}\nu^{2}}{\exp{\left(\frac{U_{0}}{kT}\right)}-1}\,,$
(26)

which can be used to define the intensity of the radiation $\displaystyle
I=\frac{\epsilon}{4\pi}$. Such a radiation intensity should reflect the
squared temperature behavior when integrated over all frequencies, i.e.,

$\int I(\nu,T)d\nu=\sigma\,T^{2}\,.$ (27)

If we assume that the quantum of energy of the gravitational radiation depends
on the frequency of the radiation as a power law, then in view of the previous
equation the energy is

$U_{0}=\alpha\,\nu^{3}\,,$ (28)

which is the energy of the graviton. The equation (27) may be rewritten as

$\displaystyle\sigma
T^{2}=\frac{2k^{2}T^{2}}{3c^{2}\alpha}\int_{0}^{\infty}\frac{x}{e^{x}-1}\,dx\,,$

once the variable $\displaystyle x=\frac{\alpha\nu^{3}}{kT}$ is settled. Here
SI units are used, in this system of units $\displaystyle c$ is the speed of
light, which cannot be confused with the Bondi-Sachs space-time function.
Since the integral above has a well-defined value
$\displaystyle\int_{0}^{\infty}\frac{x}{e^{x}-1}\,dx=\frac{\pi^{2}}{6}$, then
the $\displaystyle\sigma$ constant in terms of the Boltzmann constant and the
$\displaystyle\alpha$ constant is given by

$\displaystyle\sigma=\frac{\pi^{2}k^{2}}{9c^{2}\alpha}\,.$

In order to determine the value of such a constant, it should be noted that
the alpha constant has a dimension of $\displaystyle J\,s^{3}$, the meaning of
this result is possibly that the constant, which appears in the energy of the
graviton, is identified with Planck’s constant itself times the squared Planck
time, i.e., $\displaystyle\alpha\sim 10^{-122}$. Thus the order of magnitude
of this constant is $\displaystyle\sigma\sim 10^{60}$, in SI units. This may
indicate that the $\displaystyle\alpha$ constant is completely independent of
Planck’s constant. Perhaps only experimental measurement can establish the
value of this constant.

##  IV Conclusion

In this article we show that the definition of gravitational energy and
pressure, in the context of Teleparallelism Equivalent to General Relativity,
leads to a well-defined equation of state for the Bondi-Sachs space. Such an
equation of state for gravitational radiation induces a specific dependence on
temperature, that is, $\displaystyle p_{r}=\sigma\,T^{2}$. It is interesting
to note that such an expression is a direct consequence of the first law of
thermodynamics along with the existence of the equation of state. Since the
Gibbs free energy is zero, we assume that the gravitational radiation also
obeys the Planck’s hypothesis. Therefore we adjust the quantum of energy to
reproduce the correct dependence on the temperature. We obtain the energy of
the graviton as $\displaystyle U_{0}=\alpha\,\nu^{3}$. It is worth noting that
if an experiment were to confirm the dependence of gravitational energy
density on temperature, this would render the metric approach to gravitation
unfeasible. This is due to the fact that any dependence on temperature implies
the existence of an equation of state involving energy and pressure. In this
way the gravitational energy, whose existence is at least controversial in the
context of general relativity, would have a physical existence. In Addition,
the first law of thermodynamics, together with an equation of state, predicts
a dependence of the pressure (or the energy) on the temperature. On the other
hand, the non-existence of the concept of gravitational energy would
contradict the first law of thermodynamics, if any dependence on the
temperature of a gravitational radiation is verified. Thus measuring the
temperature of a gravitational wave is a viable way to rule out the non-
localizability of the gravitational energy. In the last 15 years, a series of
measurements involving pulsars indicate the existence of background
gravitational waves, which were not necessarily produced by black holes [5,
6]. They can be legitimate gravitational radiation and it is necessary to
investigate what relationship there is between such waves and thermal energy.
Thus an analogous experiment can be carried out in order to correlate the
frequency with the temperature of these waves.

## References

  * [1] L. Boltzmann, Ableitung des Stefanschen Gesetzes, bettrefend die Abhängigkeit der Wärmestrahlung von der Temperatur aus der elektromagnetischen Lichttheorie, Annalen der Physik 22, 29 (1884).
  * [2] M. Planck, Über das Gesetz der Energieverteilung im Normalspektrum, Annalen der Physik 309, 3, p. 553-563 (1901).
  * [3] J. W. Maluf, The teleparallel equivalent of general relativity, Annalen der Physik 525, 5 p. 339-357 (2013).
  * [4] J. W. Maluf, J. F. da Rocha-Neto and S. C. Ulhoa, Bondi-Sachs energy-momentum and the energy of gravitational radiation, Gen. Relativ. Gravit. 47, 82 (2015).
  * [5] X. Heng et al, Searching for the Nano-Hertz Stochastic Gravitational Wave Background with the Chinese Pulsar Timing Array Data Release I, Res. Astron. Astrophys. 23, 075024 (2023).
  * [6] G. Agazie et al, The NANOGrav 15 yr Data Set: Evidence for a Gravitational-wave Background, ApJL 951, L8 (2023).


.tifpng.pngconvert #1 .tif

# Deep Generative Graph Distribution Learning
for Synthetic Power Grids

Mahdi Khodayar, Jianhui Wang, and Zhaoyu Wang Mahdi Khodayar and Jianhui Wang
are with the Department of Electrical and Computer Engineering, Southern
Methodist University, Dallas, TX, USA. e-mail<EMAIL_ADDRESS>and
<EMAIL_ADDRESS>Zhaoyu Wang is with the Department of Electrical and Computer
Engineering, Iowa State University, IA, USA. e-mail<EMAIL_ADDRESS>
###### Abstract

Power system studies require the topological structures of real-world power
networks; however, such data is confidential due to important security
concerns. Thus, power grid synthesis (PGS), i.e., creating realistic power
grids that imitate actual power networks, has gained significant attention. In
this letter, we cast PGS into a graph distribution learning (GDL) problem
where the probability distribution functions (PDFs) of the nodes (buses) and
edges (lines) are captured. A novel deep GDL (DeepGDL) model is proposed to
learn the topological patterns of buses/lines with their physical features
(e.g., power injection and line impedance). Having a deep nonlinear recurrent
structure, DeepGDL understands complex nonlinear topological properties and
captures the graph PDF. Sampling from the obtained PDF, we are able to create
a large set of realistic networks that all resemble the original power grid.
Simulation results show the significant accuracy of our created synthetic
power grids in terms of various topological metrics and power flow
measurements.

###### Index Terms:

Power Grid Synthesis, Graph Distribution Learning, Deep Learning, Generative
Modeling

##  I  INTRODUCTION

MANY power system studies require the actual topologies of real-world power
networks with real bus and line coordinates as well as real physical
charactristics such as power injections and line impedance values. However,
due to the confidentiality concerns, there are very few datasets of real power
grids such as the IEEE test cases [1] and the Polish grid [2] that are
publicly available for such studies. Even these datasets do not contain
important information regarding the geographical locations of buses, and lack
the diverse characteristics of real topologies. In recent studies, several
works are focused on the spatial models for power networks that are realistic
but synthesized without revealing any confidential information. [3, 4] create
power networks based on the locations of cities and power plants in Texas;
however, no geographical or performance comparison with the actual power grid
is provided to justify the approach. Although [3, 4] contain useful
engineering details, they do not devise a general algorithm that can work with
any realistic power network; thus, in practice, these approaches are limited.
In this line of research, [5] is the only learning-based power grid synthesis
(PGS) model presented in recent literature that applies a Gaussian Mixture
Model (GMM) to compute the distributions of bus locations in the Western
Interconnect network. However, due to lack of generalization, GMM is
theoretically incapable of learning the physical properties of power grid
components. Moreover, the data required by [5] exceeds the power grid data as
this model represents the demand based on the average residential power usage
and city populations. Also, similar to [3] and [4], the GMM approach [5] puts
strict assumptions on the node degree and line distributions which do not
necessarily hold.

This letter presents a novel deep graph distribution learning (DeepGDL)
algorithm to create synthetic power grids. We present a recurrent model that
efficiently captures node/edge distributions of arbitrary complex networks.
Leveraging a deep architecture, DeepGDL captures complex nonlinear manifolds
of nodes/edges in the actual power network, and generates synthetic power
grids that imitate the original network. To the best of our knowledge, DeepGDL
is the only deep learning-based PGS solution. Also, in contrast to all
previous works, the presented algorithm needs no additional information other
than the network topology and the bus/line physical properties; hence, it can
be applied as a general framework to any PGS problem with minimum amounts of
data.

##  II DeepGDL for Power Grid Synthesis

Let us consider the actual power grid obtained from the Columbia University
Synthetic Power Grid (CUSPG) dataset [6] that contains 14430 buses and 18884
lines located in Western US. We represent this network as an undirected
weighted graph $G_{P}=(V,E)$ where $n_{i}\in V$ denotes the $i$-th node while
an edge $l_{i,j}\in E$ is a transmission line connecting two nodes $n_{i}$ and
$n_{j}$. For each node $n_{i}$, $G_{P}$ contains four features including the
latitude $x(n_{i})$, longitude $y(n_{i})$, power supply $pw(n_{i})$ (i.e.,
power generated at the bus $i$), and power demand $pd(n_{i})$. A weight
$W_{i,j}=X(l_{i,j})\in\mathbb{R}$ is defined to represent the reactance of the
line corresponding to each $l_{i,j}$. Our objective is to learn the PDFs of
nodes $n_{i}\in V$ as well as edges $l_{i,j}\in E$, and sample from them to
generate synthetic power grids.

Algorithm 1 is the pseudocode of our presented model. First, using the
modularity optimization in [7], $G_{P}$ is decomposed into $K$ communities
(i.e., subgraphs) $\Psi=\\{C^{1},C^{2},...,C^{K}\\}$ where the nodes inside
each $C^{k}~{}(1\leq k\leq K)$ are densely connected while the nodes in
distinct communities have sparse connections. For each $C^{k}=(V^{k},E^{k})$,
the adjacency matrix $A^{k}$ is modeled based on the node ordering $\tau$ that
maps any $n_{i}\in V^{k}$ to a row/column index $\tau(i)$ in $A^{k}$. Having
the set of all $|V^{k}|!$ node permutations denoted by $\mathrm{T}$, one can
write a new adjacency matrix $A^{\tau}\in
A^{\mathrm{T}}=\\{A^{\tau}|\tau\in\mathrm{T}\\}$ defined by
$A^{\tau}_{i,j}=A^{k}_{\tau(i),\tau(t)}=X(e_{\tau(i),\tau(t)})$; hence, we
model $C^{k}$ as an arbitrarily-sized sequence of node orderings each with a
unique nodal and edge feature sequence. Considering the ordering $\tau$, a
nonlinear function $f_{n}$ maps $(C^{k},\tau)$ to a 2-dimensional node feature
tensor $F_{\tau}^{k}$ using the following formulation:

$F_{\tau}^{k}=f_{n}(C^{k},\tau)=\big{(}F_{\tau}^{k}(1),F_{\tau}^{k}(2),...,F_{\tau}^{k}(|V^{k}|)\big{)}$
(1)

where
$F_{\tau}^{k}(i)=<x(n_{\tau(i)}),y(n_{\tau(i)}),pw(n_{\tau(i)}),pd(n_{\tau(i)})>$
is the nodal feature of the $i$-th node w.r.t the ordering $\tau$. Similarly,
we define $f_{\Omega}$ that maps $(G,\tau)$ to a 2-dimensional edge feature
tensor $\Omega_{\tau}^{k}$ computed by:

$\Omega_{\tau}^{k}=f_{\Omega}(G,\tau)=\big{(}\Omega_{\tau}^{k}(1),\Omega_{\tau}^{k}(2),..,\Omega_{\tau}^{k}(|V^{k}|)\big{)}$
(2)

where
$\Omega_{\tau}^{k}(i)=<A^{\tau}_{1,i},A^{\tau}_{2,i},...,A^{\tau}_{i-1,i}>^{T}$
is the weighted adjacency vector of node $\tau(i)$; thus, for an undirected
weighted $C^{k}$, the tensor $<F_{\tau}^{k},\Omega_{\tau}^{k}>$ uniquely
defines the graph $C^{k}$. As a result, one can recover $C^{k}$ by the mapping
$F_{r}(<F_{\tau}^{k},\Omega_{\tau}^{k}>)=C^{k}$; hence, in order to learn the
distribution of the nodes and edges in communities $C^{k}~{}(k\leq K)$ denoted
by $P(C^{k})$, we maximize the likelihood of their observation in actual
network $G_{P}$ by:

$\displaystyle\small{Max}~{}\sum\limits_{k=1}^{K}[P(C^{k})=\sum\limits_{\mathclap{<F_{\tau},\Omega_{\tau}>}}P(F_{\tau}^{k},\Omega_{\tau}^{k})\mathbb{I}(f_{r}(F_{\tau}^{k},\Omega_{\tau}^{k})=C^{k})]$
(3)

where $\mathbb{I}(x)=1$ if $x$ is true and zero otherwise. Using this
optimization, we maximize the observation of $G_{P}$’s communities, hence
finding a model that can imitate $G_{P}$.

Input: Actual Power Network $G_{P}=(V,E)$

1 Define GRU’s starting token $START$ and ending token $END$. Community
counter $k=1$ while  _$k\leq K$_ do

2 Initialize: $[\Omega_{\tau}^{k}(0),F_{\tau}^{k}(0)]=START$, $i=1$

3 while  _$[\Omega_{\tau}^{k}(i),F_{\tau}^{k}(i)]\neq END$_ do

4 Compute graph state ${S_{i}}$ using (5)

5 Create a new node (i-th node) using (6)

6 $i=i+1$

7 Generate Synthetic Community $C^{k}_{Syn}$ using (6) Train GRU,
$f_{F}^{nn}$, and $f_{\Omega}^{nn}$ using gradient descent

8$P_{Syn}=(V_{Syn}=\cup_{k=1}^{K}V^{k}_{Syn},E_{Syn}\cup_{k=1}^{K}E^{k}_{Syn})$
while  _$P_{Syn}$ has multiple components_ do

9 compute ${\varphi(n_{i})}$ as the mean Euclidean distance between $n_{i}$
and its $10$ closest neighbor nodes in $V_{Syn}$.

10 Select $n_{i}$ whose degree is at most 2 from all nodes in $V_{Syn}$ using
the probability ${\varphi(n_{i})}^{-1}$.

11 Add an edge between two nodes $n_{i}$ and $n_{j}$ with probability
$d(n_{j})(Euc_{i,j})^{-1}$. Here, $d(n_{j})$ denotes the $n_{j}$’s degree
while $Euc_{i,j}$ is the Euclidean distance of $n_{i}$ and $n_{j}$.

Output: Synthetic Network $P_{Syn}=(V_{Syn},E_{Syn})$

Algorithm 1 Deep Graph Distribution Learning

Let us write a decomposition for $P({F_{\tau}^{k}},{\Omega_{\tau}^{k}})$ in
(3) using conditional probabilities of the observed sequence of nodal and edge
features implied by $\tau$ for any community $C^{k}$:

$\displaystyle\small{\prod_{i=1}^{|V^{k}|}{P(}(F_{\tau}^{k}(i),\Omega_{\tau}^{k}(i))|(F_{\tau}^{k}(1),\Omega_{\tau}^{k}(1))\\\
,...,(F_{\tau}^{k}(i-1),\Omega_{\tau}^{k}(i-1)))}$ (4)

The recursive characteristics of the graph features in (4) motivates us to
model nodal and edge features by a recurrent neural network. Therefore, we
model $P({F_{\tau}^{k}},{\Omega_{\tau}^{k}})$ using a deep Gated Recurrent
Unit (GRU) network. For each community $C^{k}$, at each iteration $i$ of our
deep recurrent model, the GRU computes a graph state $S_{i}$ using the
previous node/edge observations $I(t)$ $(j=1,2,...,i)$ by the following
recurrent structure:

$\begin{array}[]{l}\small{{I(t)}=[\alpha^{k}(t)=F_{\tau}^{k}(t),\beta^{k}(t)=\Omega_{\tau}^{k}(i,j)]}\\\
\small{z(t)}=Sig({W_{1}}\cdot[{I(t)},{M(t)}])\\\
{r(t)}=Sig{({W_{2}}\cdot[{I(t)},{M(t)}])}\\\
\small{{{\tilde{M}}(t)}=tnh({W_{M}}\cdot[{I(t)},{r(t)}*{M(t-1)}])}\\\
\small{{M(t)}=(1-{z(t)})*M(t-1)\,+\,{z(t)}\,*\,{{\tilde{M}}}(t)}\end{array}$
(5)

Here, $\alpha^{k}(t)$ and $\beta^{k}(t)$ in $I(t)$ are the intermediate nodal
and edge features of $C^{k}$ observed at the $j$-th round of the recursive
formulation. At each round $j$, the temporal features, $z(t)$ and $r(t)$, are
computed for the graph observation $I(t)$ using a sigmoid function $Sig$ with
weights $W_{1}$ and $W_{2}$, respectively. Then, the GRU’s updating vector
${{\tilde{M}}(t)}$ is computed using the intermediate graph features, $r(t)$
and $I(t)$, as well as the GRU state at the previous round $M(t-1)$. Here,
$tnh$ denotes a tangent hyperbolic function with weight $W_{M}$ applied to
compute ${{\tilde{M}}(t)}$. The GRU state $M(t)$ is finally computed as a
linear regression of $M(t-1)$ and ${{\tilde{M}}(t)}$ weighted by the temporal
feature $z(t)$. After running the recursion (5) for $i$ rounds, the final GRU
state $C(j=i)$ is our extracted graph state $S_{i}$.

At each iteration $i$, we model the node/edge distribution of each community
$C^{k}$, and generate a new synthetic community
$C^{k}_{Syn}=(V^{k}_{Syn},E^{k}_{Syn})$ corresponding to $C^{k}$ using the
following formulations:

$\begin{array}[]{l}\xi_{i}^{F}=f_{F}^{nn}({S_{i}}),\xi_{i}^{\Omega}=f_{\Omega}^{nn}({S_{i}})\\\
F_{\tau}^{k}(i)\sim{P_{{\xi_{i}^{F}}}}\,\,\,\,\,\,\\{Sampling~{}the~{}features~{}of~{}i-th~{}node\\}\\\
\Omega_{\tau}^{k}(i)\sim{P_{{\xi_{i}^{\Omega}}}}\,\,\,\,\,\,\\{Sampling~{}the~{}weight~{}of~{}i-th~{}edge\\}\end{array}$
(6)

Figure 1: Visualization of the actual power grid $G_{P}$ and the synthetic network $G_{Syn}$ TABLE I: Topological properties and power flow analysis of the actual and synthetic networks  | Topological Properties | Power Flow Statistics (MW)
---|---|---
Networks | #Nodes | #Edges | $d_{avg}$ | $\omega$ | $D$ | $\rho*10^{-4}$ | $Q$ | $C$ |  Average |  Median |  STDV |  Max
$G_{P}$ | 14430 | 18554 | 2.572 | 15.061 | 37 | 1.782 | 0.953 | 0.072 | 168.93 | 33.32 | 320.93 | 4,777.07
$G_{Syn}$ | 14408 | 18302 | 2.510 | 15.253 | 37 | 1.820 | 0.961 | 0.072 | 173.27 | 29.31 | 302.21 | 4,687.83
GMM | 14269 | 19881 | 2.269 | 16.229 | 35 | 1.953 | 0.804 | 0.063 | 110.25 | 42.78 | 479.08 | 5,521,37
DeepDGL Mean(STDV) |  | 14301
---
($\bm{3.1*10^{3}}$)
| 17869
---
($\bm{4.3*10^{3}}$)
| 2.561
---
(0.08)
| 15.226
---
(0.29)
| 36.71
---
(0.13)
| 1.726
---
(0.15)
| 0.952
---
(0.02)
| 0.071
---
(0.004)
171.18 (11.34) |  30.42 (1.20) |  307.18 (15.59) |  4,590.25 (217.01)

where $f_{F}^{nn}$ and $f_{\Omega}^{nn}$ are two sigmoidal neural networks
(SNNs) with the input $S_{i}$ and output vectors $\xi_{i}^{F}$ and
$\xi_{i}^{\Omega}$, respectively. The vector $\xi_{i}^{F}$ encodes the nodal
feature distribution of the $i$-th node in $C^{k}_{Syn}$. Similarly,
$\xi_{i}^{\Omega}$ encodes the edge weight distribution corresponding to this
node. Sampling from these two distributions in (6), we create a new node
(i.e., the i-th synthesized node) of $C^{k}_{Syn}$. We run (6) to create new
nodes and edges until the GRU reaches a predefined $END$ state, which
terminates the graph generation process and outputs $C^{k}_{Syn}$ for every
community $C^{k}$. Each of the generated communities $C^{k}_{Syn}$ is a
component of our final synthetic power network $P_{Syn}$.

As shown in [5], in dense regions of real-world power grids, the low-degree
buses are generally connected to high-degree buses. Thus, in order to merge
the generated components and form a single-component robust network $P_{Syn}$,
we connect the low-degree nodes $n_{i}\in\cup_{k=1}^{K}V^{k}_{Syn}$ in dense
regions, to high-degree nodes $n_{j}\in\cup_{k=1}^{K}V^{k}_{Syn}$ until the
$P_{Syn}$ becomes single-component. The resulting $P_{Syn}$ is finally
reported as the output.

##  III Simulation Results

First, we set several hyperparameters defined in the DeepGDL algorithm. The
GRU state variables $S$, $z$ and $r$, are considered to have $120$ dimensions.
Both $f_{f}^{nn}$ and $f_{\Lambda}^{nn}$ are modeled as a SNN with $45$
neurons in its first layer and $35$ neurons in the second layer. The gradient
descent is considered to have a learning rate of $10^{-3}$ while its batch
size is $30$. In this study, $K=72$ graph communities are considered for
$G_{P}$. We generate $10^{2}$ synthetic power grids via DeepGDL. Fig. 1(a) and
(b) depict the actual network $G_{P}$ and one of our synthetic networks
$G_{Syn}$, respectively. Each community is shown by a distinct color.
Moreover, Fig. 1(c) and (d) show the amount of power supply at each bus (node)
in the two networks. To compare the performance of the generated grids with
the actual grid, we conduct cascading failure experiments using DC power flow.
Fig. 1(e) and (f) demonstrate the Yield (i.e. ratio of demand supplied after
cascades to the actual value of demand) in the two power grids for cascades
initiatd in $10^{3}$ regions of radius $100$km. Each point shows the computed
Yield at the center of its corresponding region. As shown in Fig. 1, $G_{Syn}$
accurately resembles the actual grid regarding the topological characteristics
and power flow measurements. Table I reports detailed topological measurements
and power flow statistics obtained by $G_{P}$, DeepGDL algorithm, and the
state-of-the-art GMM [5] model. It is shown that DeepGDL has significant
improvements over GMM in terms of average node degree $d_{avg}$, average path
length $\omega$, network Diameter $D$, Density $\rho$, Modularity $Q$, and
Average Clustering Coefficient $C$. Also, Table I shows the substantial
accuracy improvements of DeepGDL over the state-of-the-art in terms of power
flow measurements.

##  IV Conclusions

This letter presents a deep graph distribution learning algorithm for the
problem of power grid synthesis. A novel recurrent model is proposed to
capture deep nodal and edge features from a realistic power network. To the
best of our knowledge, DeepGDL is the only deep learning model that can
synthesize large-scale networks. Simulation results show significant accuracy
of DeepGDL in terms of topological measurements and power flow analysis
metrics.

## References

  * [1] “IEEE benchmark systems,” www.labs.ece.uw.edu/pstca/, accessed: 2019-01-20.
  * [2] “Polish grid,” www.pserc.cornell.edu/matpower/, accessed: 2019-01-21.
  * [3] K. M. Gegner, A. B. Birchfield, T. Xu, K. S. Shetye, and T. J. Overbye, “A methodology for the creation of geographically realistic synthetic power flow models,” in _Power and Energy Conference at Illinois (PECI), 2016 IEEE_ , 2016, pp. 1–6.
  * [4] A. B. Birchfield, T. Xu, K. M. Gegner, K. S. Shetye, and T. J. Overbye, “Grid structural characteristics as validation criteria for synthetic networks,” _IEEE Transactions on power systems_ , vol. 32, no. 4, pp. 3258–3265, 2017\.
  * [5] S. Soltan, A. Loh, and G. Zussman, “A learning-based method for generating synthetic power grids,” _IEEE Systems Journal_ , no. 99, pp. 1–10, 2018\.
  * [6] S. Soltan, A. Loh, and g. Zussman, “Columbia university synthetic power grid with geographical coordinates,” Tech. Rep., 1 2018.
  * [7] X. Lu, K. Kuzmin, M. Chen, and B. K. Szymanski, “Adaptive modularity maximization via edge weighting scheme,” _Information Sciences_ , vol. 424, pp. 55–68, 2018.


# It’s About Time: Incorporating Temporality in Retrieval Augmented Language
Models

Anoushka Gade<EMAIL_ADDRESS>San Jose State UniversitySan JoseCAUSA
and  Jorjeta Jetcheva<EMAIL_ADDRESS>San Jose State UniversitySan
JoseCAUSA

###### Abstract.

The web serves as a global repository of knowledge, used by billions of people
to search for information. Ensuring that users receive the most relevant and
up-to-date information, especially in the presence of multiple versions of web
content from different time points remains a critical challenge for
information retrieval. This challenge has recently been compounded by the
increased use of question answering tools trained on Wikipedia or web content
and powered by large language models (LLMs) (Liu et al., 2023) which have been
found to make up information (or hallucinate), and in addition have been shown
to struggle with the temporal dimensions of information. Even Retriever
Augmented Language Models (RALMs) which incorporate a document database to
reduce LLM hallucination are unable to handle temporal queries correctly. This
leads to instances where RALMs respond to queries such as ”Who won the
Wimbledon Championship?”, by retrieving document passages related to Wimbledon
but without the ability to differentiate between them based on how recent they
are.

In this paper, we propose and evaluate, TempRALM, a temporally-aware Retriever
Augmented Language Model (RALM) with few-shot learning extensions, which takes
into account both semantically and temporally relevant documents relative to a
given query, rather than relying on semantic similarity alone. We show that
our approach results in up to 74% improvement in performance over the baseline
RALM model, without requiring model pre-training, recalculating or replacing
the RALM document index, or adding other computationally intensive elements.

Information Retrieval, Natural Language Processing, Large Language Models

##  1\. Introduction

The web serves as an ever-expanding reservoir of real-world knowledge, with
textual documents constituting a significant fraction of its content.
Moreover, information changes over time, leading to updates to existing
documents, or the addition of new documents. This leads to multiple versions
of information from various time frames to co-exist and grow over time. A
major challenge in information retrieval is ensuring that users get access to
the most relevant and up-to-date knowledge at any time. This challenge has
recently been compounded by the increased use of question answering tools
powered by large language models (LLMs), which have gained popularity as a
result of the release of chatGPT (Liu et al., 2023). LLMs have been shown to
absorb and serve immense quantities of information from textual data (Petroni
et al., 2019). This information is typically derived from a static snapshot of
a large number of documents scraped from the web at a specific point in time.
However, real-world information changes continuously, frequently on a daily,
hourly or even real-time basis. To address the challenge of frequently
changing data, language models must expand or update their memory on a regular
basis, which involves significant computational, financial and environmental
costs.

Figure 1. Overview of TempRALM: In this figure, we show the difference between
Atlas and TempRALM. In TempRALM, the retriever fetches documents on semantic
relevance with respect to the query as well as temporal relevance relative to
the query-time

In recent years, there has been an increasing interest in using Retrieval
Augmented Language Models (RALMs) as a way of addressing the problem of
frequently evolving information, and also reducing the tendency of LLMs to
make up information known as hallucinations. RALMs use an external document
corpus as a knowledge source rather than relying solely on parametric memory
like standard LLMs. This corpus, stored in the form of a document index (a
format which makes document retrieval efficient), can be augmented and updated
to reflect current versions of documents stored in it, for e.g., Wikipedia
documents, and web pages. For a given text-based query, RALM uses its
retriever component to obtain a ranked list of the most relevant documents
from the document index and gives them to its language model component (also
known as a reader), which uses the ranked documents as context to generates a
response to the query. RALMS have been shown to generate more specific and
factual responses to queries in knowledge intensive tasks such as question
answering and fact checking than stand-alone LLMs (Lewis et al., 2020). These
models have also shown to perform well in few-shot training scenarios (Izacard
et al., 2022) using as few as 64 examples to achieve a good performance.

Although RALMs perform well in factual question answering, they typically use
a document index, which contains a single version of each document. However,
in many real-world applications, new information continues to be generated,
without invalidating or replacing existing information, which results in
multiple versions of documents to be present in the document index. For
example, scientific and medical journals continuously publish new academic
papers which are additions and enhancements to the body of knowledge that has
been published previously, for e.g., new Natural Language Processing (NLP)
research papers are posted to the arxiv platform (Arxiv, 1991) almost daily.
This challenge is further compounded when trying to build a question answering
system that supports time-sensitive queries such as ”Which language model is
the state-of-the-art on the latest summarization benchmark?”, because standard
RALM models would retrieve numerous papers, each stating it has established a
new state of the art in document summarization. This is a direct result of the
fact that standard LLMs and the LLM components of RALMs do not take into
account temporal metadata (Dhingra et al., 2021).

In this paper, we show that RALMs face challenges in handling temporality in
even simpler and more structured settings. For example, in the context of
frequently changing information such as the identities of the latest winners
of the Wimbledon tennis championship, we show that Atlas (Izacard et al.,
2022), a representative state-of-the-art RALM, is generally unable to provide
an answer that is meaningful relative to the time at which the question is
asked. For example, if the question ”Who won the Wimbledon Men’s
championship?” is asked on December 31, 2019, current state-of-the-art RALM
architectures would retrieve relevant documents by matching the year 2019 in
the query timestamp with the Wimbledon match date in the passages. However, if
the same question is asked a day later, i.e. on January 1, 2020, the current
RALM architecture can no longer retrieve the relevant documents as the year no
longer matches, and since they do not have the ability to understand temporal
relationships.

Prior attempts to address this problem have focused on pre-training the
language model by inducing a temporal context as plain text (Dhingra et al.,
2021), where the model is trained with a snapshot of timestamp-prefixed data
to enable the model to learn to handle temporal data. However, the snapshot it
is trained on is static, and pre-training of the models is an expensive
operation, which cannot be performed as frequently as real-world data tends to
be updated.

In this paper, we introduce a novel, simple, yet effective method to retrieve
temporally correct documents with respect to a given query and use it to
augment Atlas (Izacard et al., 2022), a standard RALM model with few-shot
learning extensions. Our model, TempRALM, includes an extension of the RALM
retriever’s document retrieval and ranking algorithm, to take into account
both semantically and temporally relevant documents relative to each query,
rather than relying on semantic similarity alone. We show that our approach
results in up to 74% improvement in performance over the baseline Atlas model,
while being computationally efficient. TempRALM does not require pre-training,
recalculating or replacing the document index, or the addition of other
computationally intensive elements.

The rest of this paper is organized as follows. Section 2 overviews related
work. We introduce our methodology and experimental setup in Sections 3 and 4.
Section 5 discusses our experimental results. We conclude the paper with a
summary and next steps in Section 6.

Figure 2. Retrieval Augmented Language Model with Fusion-in-Decoder
Architecture

##  2\. Related Work

Large Language Models (LLMs) (Brown et al., 2020), (Google et al., 2023) and
conversational interfaces to these models like ChatGPT (Liu et al., 2023),
have the potential to revolutionize our daily lives by helping people discover
and understand information in real-time. However, these systems are prone to
hallucinations and highly confident sounding answers that are in fact wrong,
limiting their use for knowledge-intensive tasks such as Question-Answering,
Fact Checking, etc. Retrieval Augmented Language Models (RALMs)(Lewis et al.,
2020) offer a promising approach to mitigating these problems, by adding a
document database (also known as document index) as a knowledge source which
is used to prompt or contextualize LLM-generated answers. Typically, a
Wikipedia snapshot is used to construct the document index of general-purpose
RALMs. When a query is made, a set of documents are retrieved (by a retriever
model within the RALM) and a response is generated using the query and the
retrieved documents as input to an LLM. This approach has been shown to
significantly improve the performance of LLMs on knowledge intensive tasks
shown by(Lewis et al., 2020). Moreover, training the retriever does not
require a labeled mapping between queries and documents, and thus does not
introduce data labeling complexity to the training process.

To alleviate the need of large datasets for training or fine-tuning RALMs,
(Izacard et al., 2022) propose Atlas, a pre-trained model with few-shot
learning capabilities. It consists of a pre-trained retriever, (Izacard et
al., 2021), which is a dense retriever based on the BERT (Devlin et al., 2018)
architecture, trained using contrastive loss, as well as a language model, T5,
(Raffel et al., 2019), trained to perform a range of sequence-to-sequence
tasks, e.g., summarization, translation, and question answering. Additionally
both models are fine-tuned jointly as a Fusion-in-Decoder (de Jong et al.,
2023) on common-crawl data using Wikipedia as a document index. The authors
showed that Atlas performs well out-of-box and can be adapted to different
tasks such as question-answering, fact-checking, etc with as few as 64 few-
shot examples.

An aspect of question answering that has emerged as a challenge for both RALMs
and LLMs are questions whose answers evolve over over time as seen in (Zhao et
al., 2022). For example, the answer to ”Who is the President of the United
States?” will have a different answer every 4 years or so.

In (Dhingra et al., 2021), the authors explore the challenges such questions
pose to LLMs, and show how pre-trained models on a single snapshot of the
Internet at a given point in time are inherently incapable of answering them
correctly. In their work in (Izacard et al., 2022), the authors investigate
this problem in the context of RALMs and show that RALMs can handle these
questions better than pure LLMs, by replacing the document index with an up-
to-date document index. However, this modifications is not sufficient to
address question answering in domains such as scientific, medical and legal
publications, where it is not easy to replace one fact with another. In
addition, updates to knowledge in these domains result in adding new
documents, rather than replacing existing documents with updated versions.

In (Zhang and Choi, 2021), the authors also explore time-sensitive open domain
QA, where they provide a Dense Passage Retriever (Karpukhin et al., 2020) with
a query modified with additional lingustic context, but see no improvement for
temporally dependent questions. They also found that retrieval based models
were able to update some world knowledge after swapping the retrieval corpus
and fine-tuning with newer data. However, this approach is not efficient to
keep models’ responses up-to-date. This approach is similar to the approach
used in Atlas, where the authors use a static snapshot of Wikipedia, but swap
the corpus depending on the temporal context, which is not feasible in
scenarios where there are frequent updates in the data.

In (Kasai et al., 2022), the authors proposed to collect temporally evolving
questions by sourcing from news websites. They propose a simple solution of a
RAG model with a wiki snapshot to address the questions. One limiting factor
in this approach is that they use a static index to address dynamically
evolving questions. In our approach, we handle multiple versions of each
document, in order to accommodate cases where information is added, without
needing to swap out or invalidate prior information.

We build on the the work by (Izacard et al., 2022) by proposing and
evaluating, TempRALM, a novel temporally augmented RALM with few-shot learning
capabilities, which can accommodate a document index which contains multiple
versions of documents, and does not require model re-training.

##  3\. Methodology

Figure 3. Example of tabular data to passage conversion. We convert relevant
features of the tabular dataset to textual passages.

In our study, we build on the Retrieval Augmented Language Model (RALM)
architecture, enhanced with few-shot learning, which was first introduced by
the Atlas model (Izacard et al., 2022). Notably, existing RALM approaches,
including Atlas, are not able to take into account the temporal aspects of a
query.

To that end, we augment Atlas with a novel temporal retrieval method and
evaluate the performance of the resulting model. We overview Atlas and the
details of our proposed extensions in the remainder of this section.

###  3.1. RALM Overview

The RALM architecture has three core elements: a document index which contains
documents, a retriever to identify the relevant documents (from the document
index) for a given query/question, and a language model (also called as
reader) that generates the response to the question.

The retriever identifies the $topk$ most relevant documents (or passages) for
a given query from the (typically very large corpus of) text documents stored
in the document index. The language model takes the retrieved documents and
the query as input, and generates an answer to the query.

For our document index, we use a time-sensitive dataset that contains text
passages that refer to tennis tournament details at different time points,
described in Section 4. We use the pre-trained Atlas generator and retriever
components. The retriever is based on the Contriever (Izacard et al., 2021),
an information retrieval approach based on continuous dense embeddings. The
generator is based on a sequence-to-sequence T5 (Raffel et al., 2019)
architecture. It is combined with the retriever with a Fusion-in-Decoder (de
Jong et al., 2023) modification and language modelling adaptation.

While we validate our hypothesis using the RALM architecture under few-shot
conditions, it is important to note that our approach could be used to extend
any retriever that ranks documents using a numerical similarity score between
a query and a document corpus. We show the RALM with fusion-in-decoder
architecture in Figure 2

###  3.2. Semantic Retrieval Score

The core computation involved in deciding which documents are relevant to a
query is based on computing a semantic similarity-based relevance score
between the query and each document in the index and using the dot product of
their encoder representations. In particular, for a given query $q$ and
document $d$, their encoder representations are obtained independently by
encoding them using an encoder $f_{\theta}$, parameterized by $\theta$, the
set of trainable parameters of the encoder.

Then the semantic score, $s(q,d)$ between the query $q$ and the document $d$
is computed as the dot product of their resulting encoder representations:

(1) $s(q,d)=\langle f_{\theta}(q),f_{\theta}(d)\rangle$

###  3.3. Temporal Score

We introduce a new method called TempRALM, which augments the retriever with a
temporal score $\tau(qt,dt)$, where $qt$ is the timestamp of the query $q$,
and $dt$ is the timestamp of document $d$. To calculate $\tau(qt,dt)$, we take
the reciprocal of the time difference between $qt$ and $dt$ to ensure that the
temporal score is inversely proportional to the time difference (and thus
smaller differences in time result in a larger score), and use a scaling
factor $\alpha$ to tune the temporal score’s importance relative to the
importance of the semantic score. The temporal score formulation is as
follows:

(2) $\tau(qt,dt)=\frac{\alpha_{scale}}{qt-dt}\ $

To align the temporal score with the numerical range of the semantic score
$s(q,d)$ from equation 1, we employ a normalization process. In particular, we
subtract from each temporal score, the mean over the temporal scores of all
$(q,d)$ pairs, $\mu_{\tau}$, and divide by the standard deviation over the
temporal scores of all $(q,d)$ pairs, $\sigma_{\tau}$. The normalized temporal
score is then scaled to the range of the relevance score by multiplying with
standard deviation of semantic scores $\sigma_{s}$ and adding mean of semantic
scores $\mu_{s}$ (both computed over the scores of all $(q,d)$ pairs):

(3) $\tau(qt,dt)=\frac{\tau(qt,dt)-\mu_{\tau}}{\sigma_{\tau}}\
\times\sigma_{s}+\mu_{s}$

Figure 4. Questions asking about the winner, runner-up, finalists and score
asked at different timestamps, where the right answer corresponds to the most
recent tennis match that occurred before the query timestamp

###  3.4. TempRALM: A Temporally-Augmented Retriever

To incorporate the temporal score shown in equation (3) in the retriever, we
define an auxiliary retrieval scoring function $TempRet_{t}$, which is the sum
of the semantic score $s(q,d)$ and the temporal score $\tau(qt,dt)$:

(4) $TempRet_{t}(q,d,qt,dt)=s(q,d)+\tau(qt,dt)\ $

In addition, we extend the retriever by removing passages related to future
information with respect to the time of the query. In particular, we mask the
retrieval score for retrieved passages that have time stamps that are larger
than the query timestamp (denoting passages that refer to future information
relative to the time of the event specified in the query), to completely
eliminate these passages from being considered while ranking the $topk$. We
re-write the retrieval computation as follows:

(5) $TempRet_{t}(q,d,qt,dt)=\begin{cases}s(q,d)+\tau(qt,dt)\,&\text{if }qt\geq
dt\\\ -\infty,&\text{otherwise}\end{cases}$

Finally, to ensure a comprehensive coverage of relevant passages, we over-
retrieve of set of documents from the retriever, and from this over-retrieved
set, the $topk$ documents with highest $TempRet_{t}$ scores are passed as
input to the language model. We conduct an extensive set of experiments to
find the optimal number of documents to over-retrieve. It’s important to note
that our extensions do not require re-training any part of the RALM.

###  3.5. Evaluation Metrics

We evaluate the effectiveness of our system on time-specific question
answering test sets. We assume a few-shot training set of (query, timestamp,
answer) triplets,

$Q_{train}=\\{(q_{1},qt_{1},a_{1}),(q_{2},qt_{2},a_{2}),\dots(q_{N},qt_{N},a_{N}\\}$
and evaluate on a test set of held out queries, timestamps, and answers. Our
evaluation metric is exact-match between the predicted and ground-truth answer
for each data point in the test set.

##  4\. Experimental Setup

To explore the effectiveness of our temporally augmented RALM model, we
formulated a retrieval task based on a use case where information evolves over
time. In particular, we curated a dataset based on tennis grand slam data,
where 4 major tournaments take place every year and answers to the queries
depend on when the query is asked, with respect to the timing of the
tournament. We describe the details of the model configuration, our dataset,
and our experiments next. All of our experiments are conducted on a single
NVIDIA A100-PCIE-40GB GPU.

###  4.1. TempRALM Component Configuration

The TempRALM retriever uses the standard Atlas-large (Izacard et al., 2022)
configuration, augmented with our temporal extensions. In particular, it is
based on a general-purpose dense retriever that uses a dual-encoder
architecture, based on the Contriever (Izacard et al., 2021) and a sequence-
to-sequence model built on the Fusion-in-Decoder architecture (de Jong et al.,
2023) which uses T5-1.1 (Raffel et al., 2019) with a language modeling
adaptation. We initialize the retriever and generator with the same pre-
training as Atlas (Izacard et al., 2022).

We chose parameters to configure TempRALM and Atlas-large based on
experimenting with a range of values across their various hyper-parameters,
including $topk$ documents to retrieve for every question, number of training
steps, retriever and language model learning rates, and sampling temperatures.

###  4.2. Document Index Generation

We initially explored using Wikipedia (Wikipedia, 2001) for use as a document
index, as it is commonly used in knowledge intensive tasks. However, we found
that the structure of Wikipedia pages tends to make it harder to connect
events to the dates on which they occurred due to a lot of additional text
present on each page, resulting in poor model performance. In addition, once
we crawled all Wikipedia pages and parsed the documents related to tennis
grand slams, we found that the ATP/WTA Men’s and Women’s Singles Tournament
Tennis data table (kaggle taylorbrownlow, [n. d.]) actually contains was more
detailed information for each event, such as who played which match, dates and
locations of games. In fact, as tables are a common way to summarize details
of events, we believe that exploring their use for training, fine tuning or
prompting LLMs is an important direction to explore. We take an initial step
towards this goal in this paper.

We started out by filtering the ATP table by the following 4 grand slams:
Australian Open, Roland Garros, Wimbledon and US Open. We then converted each
row of the table into a passage of text described in natural language,
covering tournament names, winner names, date of match and final scores. We
also extracted the timestamp from the row, in the format, $YYYY-MM-DD$ and
attached it the passage as metadata. We span tournaments from 1978 to 2019,
yielding to 50,000 documents. The process is illustrated in Figure 3.

| TPQ-2019 Test Set Exact Match | TPQ-2020 Test Set Exact Match
---|---|---
Train Set | Atlas | TempRALM | Atlas | TempRALM
32 | 64.84 | 71.72 | 37.65 | 55.93
64 | 74.68 | 75.78 | 40.93 | 68.44
128 | 76.88 | 77.80 | 41.72 | 72.65
Table 1. Comparing Temporal Proximity with Time based token matching: We
evaluate the retrieval performance of two query sets - TPQ-2019 and TPQ-2020 -
which contain identical questions about an event that occurred in 2019 but
have timestamps from two consecutive years, to simulate the time-frame in
which the queries are made. This table presents the performance comparison
between the Atlas-large model and TempRALM. Both models were fine-tuned on a
limited training sets comprising 32, 64, and 128 time-suffixed question-answer
pairs. A robust model is expected to accurately address queries based on their
temporal closeness to passages relevant to the event, rather than relying
solely on matching year tokens within the queries and passages. Our model
demonstrates strong performance across both query sets.

###  4.3. Training and Evaluation Data

Since our focus is on temporally aware question answering, we use the
following input data format for our model: (query, timestamp, answer). The
time stamp captures the time at which a query is asked, and the answer
corresponds to the answer which is valid at the time the query is posed.

For instance, given the query ”Who won the Australian Open Women’s singles
final?” posed on May 14, 2012, the correct response should indicate the player
who held the title of Australian Open Women’s singles champion as of May 14,
2012. It’s crucial to note that the response to the same query changes as soon
as a new Australian Open tournament completes.

####  4.3.1. Few-Shot Training Set

We create 3 non-overlapping few-shot training sets of size 32, 64 and 128 data
points each. Each data point has the format (query, timestamp, answer). Each
training set is constructed to provide a balanced distribution of query types,
e.g., queries about winners, runners-up, finalists, and final match scores,
spanning all four major tennis tournaments from 1978 to 2018.

We chose the sizes of the few-shot learning training sets based on
recommendations by the Atlas authors (Izacard et al., 2022), who found that 64
training examples typically result in reasonable performance. We added 32 and
128-shot experiments in order to observe the impact of training set size in
our specific scenario.

####  4.3.2. Test sets

We created two evaluation test sets, each of size 128, which use ground truth
triplets using the same format and content as the training set, except
focusing on tournament data for 2019, creating a clear separation between the
training and test sets. We refer to these test sets as Temporal Proximity
Query sets (TPQs), namely TPQ-2019 and TPQ-2020. Both query sets contain
identical questions pertaining to the matches that occurred in 2019 but have
timestamps from two consecutive years, 2019 and 2020 respectively. This
enables us to simulate two temporal contexts. For instance, given the question
’Who won the US Open Women’s singles final?’, TPQ-2019 would time stamp the
query as being posed in 2019, e.g., with 12-31-2019, whereas TPQ-2020 would
timestamp the query as being posed in 2020, e.g., 01-01-2020. The answer in
both cases refers to the U.S. Open tournament which took place in September
2019.

Answering the question correctly requires the model to go beyond textual
pattern matching, and to have (or be able to simulate) temporal awareness that
enables it to determine proximity between dates. Figure 4 shows how we
constructed queries relevant to a particular passage.

##  5\. Results

For our baseline experiment, we compare the performance of TempRALM to Atlas-
large on TPQ-2019 and TPQ-2020 test sets in 32, 64, and 128 few-shot training
scenarios, using an exact match metric. In addition, we evaluate the
retrievers of both models using a recall@1 and recall@5 metrics, which compute
the number of times the correct passage is in the top 1 or top 5 results
returned by the retriever respectively. We discuss our results next.

###  5.1. Comparison between TempRALM and Atlas-large

We compare the performance of TempRALM and Atlas-large on both the TPQ-2019
and TPQ-2020 test sets, using an exact match metric (Table 1). Each experiment
was run 5 times and the results in the table are the average of the 5 runs. We
observe that when the timestamp year of the query matches that of the event
(and thus the date referenced in the corresponding text passage), the
unmodified Atlas model performs comparably to TempRALM. In these scenarios,
the semantic score captures the matching date pattern between the query and
the passage. However, when the year in the timestamp of the query does not
match that in the text passage (TPQ-2020 experiments), TempRALM outperforms
unmodified Atlas by 49% in the 32 few-shot learning experiment, 67% in the 64
few-shot learning experiment, and 74% in the 128 few-shot learning experiment.
This illustrates that our temporal augmentation plays a key role in the
overall answer generations since the the year in the query and passages no
longer match, and the time proximity between different time stamps is not
captured by the similarity score, but only by our temporal extensions.

Our findings also indicate that as the number of examples in the few shot
training increase, our temporal augmentation becomes increasingly more
impactful.

Similarly to the findings in (Izacard et al., 2022), we find that at few-shot
training with 64 examples, model performance starts to be reasonable, and
improves slowly for 128 training examples. We show the Exact match performance
in Table 1

###  5.2. Retriever performance analysis

We observed instances where TempRALM successfully retrieves the gold passage
(which contains the correct answer), but the answer generated by the LLM is
wrong, indicating that the problem in those examples lies with the LLM.

To quantify this behavior, we calculated the number of times the retriever
ranked the gold passage as the top retrieved document (recall@1) and also the
number of times the retriever ranked the gold passage in the top 5 retrieved
passages (recall@5) for both TempRALM and Atlas-large. Our results, shown in
Table 2, illustrate that temporal retrieval improves both Recall@1 and
Recall@5, and this improvement is the most pronounced in the Recall@1 for the
TPQ-2020 experiments, where TempRALM improves over Atlas-large by a factor of
165%. These experiments illustrate both the inability of Atlas-large’s
retriever to handle temporal data, and TempRALM’s impressive ability to
compensate for that.

| TPQ-2019 | TPQ-2020
---|---|---
Model | Recall@1 | Recall@5 | Recall@1 | Recall@5
Atlas-Large | 0.54 | 0.63 | 0.24 | 0.63
TempRALM | 0.63 | 0.75 | 0.64 | 0.75
Table 2. Retriever Recall Metrics: We calculate the recall in 64-shot training
example setting. We choose an experiment closest to the average exact match of
all our 64-shot Atlas and TempRALM experiments.

###  5.3. Model Size Performance: Large vs Base Model

We evaluated the performance on the Atlas large model that has (reader 770M /
retriever 110 M) Parameters. We also compared performance on the Base ( Reader
220M / retriever 110M) model, and show that the temporal retrieval works
better across different model sizes. In table 3, we compare the Exact Match
performance of a 64-shot experiment setting experiment for Atlas and TempRALM.

| Test Set Exact Match |  |
---|---|---|---
Model Size | TPQ-2019 | TPQ-2020 |  |
Atlas Base | 42.97 | 28.13 |  |
TempRALM Base | 56.25 | 35.94 |  |
Atlas Large | 74.68 | 40.93 |  |
TempRALM Large | 75.78 | 68.44 |  |
Table 3. Model Size Performance: We compare the Exact Match in 64-shot
training example setting between Atlas and TempRALM across base and large
models.

##  6\. Conclusions and Future Work

In this study, we introduced and evaluated TempRALM, a Retriever Augmented
Language Model (RALM) augmented with temporal awareness. Unlike conventional
RALM approaches that rely solely on semantic similarity, TempRALM considers
both semantic and temporal relevance when selecting documents to pass to its
Large Language Model (LLM) in response to a given query. Our results indicate
an improvement in performance of up to 74% compared to the Atlas-large model,
even when multiple versions of documents (from different time points) are
present in the document index. Notably, we achieve this without the need for
model pre-training, replacing the document index with an updated index, or
adding any of other computationally intensive elements. We plan to explore a
number of avenues for building on the work presented in this paper, such as
implementing and evaluating different learning strategies for the parameters
of our temporal relevance function, and exploring the interplay between the
retriever and LLM. Furthermore, we plan to explore the use of our temporal
retrieval approach in other tasks such as fact checking, recommender systems,
and retrieval augmented dialog agents.

## References

  * (1)
  * Arxiv (1991) Arxiv. 1991.  Arxiv.  https://arxiv.org
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * de Jong et al. (2023) Michiel de Jong, Yury Zemlyanskiy, Joshua Ainslie, Nicholas FitzGerald, Sumit Sanghai, Fei Sha, and William Cohen. 2023.  FiDO: Fusion-in-Decoder optimized for stronger performance and faster inference.  arXiv:2212.08153 [cs.CL]
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _CoRR_ abs/1810.04805 (2018).  arXiv:1810.04805 http://arxiv.org/abs/1810.04805
  * Dhingra et al. (2021) Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2021.  Time-Aware Language Models as Temporal Knowledge Bases.  _CoRR_ abs/2106.15110 (2021).  arXiv:2106.15110 https://arxiv.org/abs/2106.15110
  * Google et al. (2023) Rohan Anil Google, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023.  PaLM 2 Technical Report.  arXiv:2305.10403 [cs.CL]
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Towards Unsupervised Dense Information Retrieval with Contrastive Learning.  _CoRR_ abs/2112.09118 (2021).  arXiv:2112.09118 https://arxiv.org/abs/2112.09118
  * Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.  Atlas: Few-shot Learning with Retrieval Augmented Language Models.  arXiv:2208.03299 [cs.CL]
  * kaggle taylorbrownlow ([n. d.]) kaggle taylorbrownlow. [n. d.].
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering.  arXiv:2004.04906 [cs.CL]
  * Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. 2022.  RealTime QA: What’s the Answer Right Now?  arXiv:2207.13332 [cs.CL]
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.  _CoRR_ abs/2005.11401 (2020).  arXiv:2005.11401 https://arxiv.org/abs/2005.11401
  * Liu et al. (2023) Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. 2023\.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.  _arXiv preprint arXiv:2304.01852_ (2023).
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2019.  Language Models as Knowledge Bases?  _CoRR_ abs/1909.01066 (2019).  arXiv:1909.01066 http://arxiv.org/abs/1909.01066
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _CoRR_ abs/1910.10683 (2019).  arXiv:1910.10683 http://arxiv.org/abs/1910.10683
  * Wikipedia (2001) Wikipedia. 2001.  Wikipedia, The Free Encyclopedia.  https://wikipedia.org
  * Zhang and Choi (2021) Michael J. Q. Zhang and Eunsol Choi. 2021.  SituatedQA: Incorporating Extra-Linguistic Contexts into QA.  arXiv:2109.06157 [cs.CL]
  * Zhao et al. (2022) Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2022.  Dense Text Retrieval based on Pretrained Language Models: A Survey.  arXiv:2211.14876 [cs.IR]