# Sensing with Shallow Recurrent Decoder Networks

Jan P. Williams∗, Olivia Zahn∗∗ and J. Nathan Kutz† ∗Department of Mechanical
Engineering, University of Washington, Seattle, WA  ∗∗ Department of Physics,
University of Washington, Seattle, WA  † Departments of Applied Mathematics
and Electrical and Computer Engineering, University of Washington, Seattle, WA

###### Abstract

Sensing is a universal task in science and engineering. Downstream tasks from
sensing include inferring full state estimates of a system (system
identification), control decisions, and forecasting. These tasks are
exceptionally challenging to achieve with limited sensors, noisy measurements,
and corrupt or missing data. Existing techniques typically use current
(static) sensor measurements to perform such tasks and require principled
sensor placement or an abundance of randomly placed sensors. In contrast, we
propose a SHallow REcurrent Decoder (SHRED) neural network structure which
incorporates (i) a recurrent neural network (LSTM) to learn a latent
representation of the temporal dynamics of the sensors, and (ii) a shallow
decoder that learns a mapping between this latent representation and the high-
dimensional state space. By explicitly accounting for the time-history, or
trajectory, of the sensor measurements, SHRED enables accurate reconstructions
with far fewer sensors, outperforms existing techniques when more measurements
are available, and is agnostic towards sensor placement. In addition, a
compressed representation of the high-dimensional state is directly obtained
from sensor measurements, which provides an on-the-fly compression for
modeling physical and engineering systems. Forecasting is also achieved from
the sensor time-series data alone, producing an efficient paradigm for
predicting temporal evolution with an exceptionally limited number of sensors.
In the example cases explored, including turbulent flows, complex spatio-
temporal dynamics can be characterized with exceedingly limited sensors that
can be randomly placed with minimal loss of performance.

##  I Introduction

Emerging sensor technologies are transforming every science and engineering
domain, with the quantity and quality of data collected also driving
fundamental advances through data-science and machine learning methods. In
many areas of interest, measurements of the full-state are at best impractical
and often impossible. Thus sensors are commonly used to infer the current and
future behavior of high-dimensional systems with a limited number of sensor
locations. With severely limited and noisy sensor measurements, this task is
exceptionally difficult and frequently requires principled sensor placement
schemes to yield faithful reconstructions manohar_data-driven_2018 . Accurate
and robust reconstruction techniques are vital in enabling downstream tasks
such as system identification, forecasting, and control. While data-driven
techniques can learn mappings from sensor measurements to the full-state space
erichson_shallow_2020 ; yu_flowfield_2019 ; callaham_robust_2019 ;
bolton_applications_2019 , this is typically done with the current sensor
values only. We advocate, instead, learning a mapping from sensor
trajectories, which contain the time history of the sensors, to the full
state-space by using a recurrent neural network in partnership with a decoder
network. As will be shown, not only is there a significant performance
increase, but a minimal number of sensors, randomly placed, can be used.

\begin{overpic}[width=416.27809pt]{Diagram.png} \put(2.0,26.0){{A}}
\end{overpic} (a)

\begin{overpic}[width=138.76157pt]{sensor_measurements.png}
\put(3.0,75.0){{B}} \end{overpic} (b)

\begin{overpic}[width=138.76157pt]{hiddenstates.png} \put(3.0,75.0){{C}}
\end{overpic} (c)

\begin{overpic}[width=138.76157pt]{summaryresults.png} \put(3.0,75.0){{D}}
\end{overpic} (d)

Figure 1: Summary diagram of SHallow REcurrent Decoder networks (SHRED) for
flow reconstruction from sensor measurements. A A graphical representation of
the SHRED method. The mulitvariate time-series of sensor measurements,
$\\{\mathbf{y}_{i}\\}_{t-k}^{t}$, is fed into a stacked long short-term memory
layer. The final output of the recurrent layer, $\mathbf{h}_{t}$, serves as an
input to a fully-connected shallow decoder mapping from the hidden state to
the high-dimensional field. B The time series of sensor measurements fed into
the SHRED model. C The evolution of the output hidden state generated by the
input sequence of sensor measurements. D Reconstruction errors of traditional
methods (QR/POD), shallow decoders (SDN), and SHRED on a turbulent flow when
three sensors are available.

\begin{overpic}[width=433.62pt]{ISOSnapshots.png} \put(2.0,95.0){{A}}
\end{overpic} (a)

\begin{overpic}[width=424.94574pt]{ISOResults.png} \put(2.0,55.0){{B}}
\end{overpic} (b)

\begin{overpic}[width=424.94574pt]{ISONoise.png} \put(2.0,55.0){{C}}
\put(61.0,4.25){$(\alpha)$} \end{overpic} (c)

Figure 2: A Example reconstructions obtained via SHRED and QR/POD of a
turbulent flow when three sensors are available. Ground truth included for
comparison. B Reconstruction errors of the current state-of-the-art methods
and SHRED with a varying number of available sensors. The solid lines denote
the median error from 32 trained estimators and the shaded region denotes the
interquartile range. C Performance of reconstruction methods in the presence
of varying levels of added Gaussian white noise. The added noise has mean zero
and standard deviation equal to $\alpha$ times the mean absolute value of the
field. The number in parentheses denotes the number of available sensors.

\begin{overpic}[width=433.62pt]{SSTSnapshots.png} \put(2.0,95.0){{A}}
\end{overpic} (a)

\begin{overpic}[width=424.94574pt]{SSTResults2.png} \put(2.0,55.0){{B}}
\end{overpic} (b)

\begin{overpic}[width=424.94574pt]{SSTNoise.png} \put(2.0,55.0){{C}}
\put(61.0,4.25){$(\alpha)$} \end{overpic} (c)

Figure 3: A Example reconstructions obtained via SHRED and QR/POD of sea-
surface temperature when three sensors are available. Ground truth included
for comparison. B Reconstruction errors of the current state-of-the-art
methods and SHRED with a varying number of available sensors. The solid lines
denote the median error from 32 trained estimators and the shaded region
denotes the interquartile range. C Performance of reconstruction methods in
the presence of varying levels of added Gaussian white noise. The added noise
has mean zero and standard deviation equal to $\alpha$ times the mean absolute
value of the field. The number in parentheses denotes the number of available
sensors.

\begin{overpic}[width=433.62pt]{AO3Snapshots.png} \put(2.0,95.0){{A}}
\end{overpic} (a)

\begin{overpic}[width=424.94574pt]{AO3Results.png} \put(2.0,55.0){{B}}
\end{overpic} (b)

\begin{overpic}[width=424.94574pt]{AO3Noise.png} \put(2.0,55.0){{C}}
\put(61.0,4.25){$(\alpha)$} \end{overpic} (c)

Figure 4: A Example reconstructions obtained via SHRED and QR/POD of
atmospheric ozone concentration when three sensors are available. Only one of
thirty elevations is shown. Ground truth included for comparison. B
Reconstruction errors of the current state-of-the-art methods and SHRED with a
varying number of available sensors. The solid lines denote the median error
from 32 trained estimators and the shaded region denotes the interquartile
range. C Performance of reconstruction methods in the presence of varying
levels of added Gaussian white noise. The added noise has mean zero and
standard deviation equal to $\alpha$ times the mean absolute value of the
field. The number in parentheses denotes the number of available sensors.

The reconstruction of spatio-temporal dynamics from limited sensors relies on
low-rank features of the data. Computing low-rank embeddings of such high-
dimensional data is often achieved through the singular value decomposition
(SVD), also known as proper orthogonal decomposition (POD) kutz_data-
driven_2013 ; brunton_data-driven_2019 . The coefficients of the dominant
correlated modes are determined by solving a linear inverse problem and the
modes themselves serve as a linear map between measurements and the spatio-
temporal state space. Many of these linear methods are built upon the
mathematical framework of gappy POD and have been successful across many
disciplines everson_karhunenloeve_1995 ; drmac_new_2016 ;
barrault_empirical_2004 ; chaturantabut_nonlinear_2010 ; manohar_data-
driven_2018 . More recently, shallow decoder networks (SDN) have leveraged
advances in machine learning and AI to learn end-to-end, nonlinear maps
between measurements and high-dimensional state spaces. SDNs have been
demonstrated to outperform their linear counterparts, particularly when the
number of available sensors is exceedingly low erichson_shallow_2020 ;
carter_data-driven_2021 ; sahba_wavefront_2022 ; williams_data-driven_2022 .
Both derivatives of gappy POD and SDNs rely on measurements at a single
snapshot in time to reconstruct the corresponding high-dimensional state at
that time.

As a result of this static sensing, linear reconstruction methods are heavily
reliant on optimally placed sensors for the inverse problem to be well-
conditioned manohar_data-driven_2018 ; drmac_new_2016 . In general,
determining optimal sensor locations is a combinatorially hard problem and is
infeasible for large search spaces. For smaller spaces, there exist a number
of well-known techniques for sensor placement boyd_convex_2004 ;
joshi_sensor_2009 ; caselton_optimal_1984 ; krause_near-optimal_2008 ;
lindley_measure_1956 ; sebastiani_maximum_2000 ; paninski_asymptotic_2005 ,
but they become computationally intractable in high-dimensional systems.
Greedy algorithms, such as the QR decomposition with column pivoting, offer
approximate solutions in larger search spaces and are thus critical in
enabling accurate reconstructions for high-dimensional systems manohar_data-
driven_2018 ; drmac_new_2016 . However, greedy algorithms suffer from the fact
that there is no guarantee that the sensor locations found are physically
implementable; for instance, measuring sea-surface temperature in the middle
of the Pacific Ocean is significantly more challenging than taking
measurements along the coast. Variations of greedy search algorithms can
incorporate additional constraints, such as cost and sensor fidelity
clark_sensor_2020 ; clark_multi-fidelity_2021 ; clark_greedy_2019 ;
manohar_optimized_2019 , but often at the expense of reconstruction accuracy.
The use of SDNs helps mitigate the dependence on sensor location, but still is
greatly aided by principled sensor placement schemes williams_data-driven_2022
.

The data-driven method presented here incorporates temporal trajectories of
sensor measurements to improve reconstruction accuracy, robustness to noise,
and eliminate the need for optimally placed sensors. We use a type of
recurrent network layer, long short-term memory networks (LSTM)
hochreiter_long_1997 , to process a time-series of sensor measurements. The
latent representation of the LSTM is the input into a fully-connected, shallow
decoder for reconstruction. We demonstrate that these SHallow REcurrent
Decoders (SHRED) outperform existing linear and nonlinear techniques on three
example datasets: a forced isotropic turbulent flow li_public_2008 , weekly
sea-surface temperature reynolds_improved_2002 , and atmospheric ozone
concentration bey_global_2001 . In all cases, SHRED with as few as three
randomly placed sensors achieves superior reconstruction accuracy than
existing techniques using a far greater number of optimally placed sensors.
Moreover, because the inputs to a trained SHRED model consist of just a few
sensor measurements, SHRED offers an on-the-fly compression for modeling
physical and engineering systems. Fig. 1 gives a graphical summary of SHRED.

##  II Reconstruction of high-dimensional spatio-temporal fields

A SHRED model is a neural network mapping from a trajectory of sensor
measurements to a high-dimensional, spatio-temporal state. The architecture
can be expressed as

$\mathcal{H}\left(\\{\mathbf{y}_{i}\\}_{i=t-k}^{t}\right)=\mathcal{F}\left(\mathcal{G}(\\{\mathbf{y}_{i}\\}_{i=t-k}^{t};\mathbf{W}_{RN});\mathbf{W}_{SD}\right)$
(1)

where $\mathbf{y}_{t}$ consists of measurements of the high-dimensional state
$\mathbf{x}_{t}$, $\mathcal{F}$ is a fully-connected, feed-forward neural
network parameterized by weights $\mathbf{W}_{SD}$, and $\mathcal{G}$ is a
LSTM network parameterized by weights $\mathbf{W}_{RN}.$ The desired network
$\mathcal{H}$ minimizes the reconstruction loss,

$\mathcal{H}\in\operatorname*{arg\rm{}min}_{\widetilde{\mathcal{H}}\in\mathscr{H}}\sum_{i=1}^{N}||\mathbf{x}_{i}-\widetilde{\mathcal{H}}\left(\\{\mathbf{y}_{j}\\}_{i-k}^{i}\right)||_{2}$
(2)

given a set of training states $\\{\mathbf{x}_{i}\\}_{i=1}^{N}$ and
corresponding measurements $\\{\mathbf{y}_{i}\\}_{i=1}^{N}$. We train the
network to minimize reconstruction loss using the ADAM optimizer
kingma_adam_2017 .

We demonstrate the performance of SHRED on three example datasets. In each
case, we compare the reconstruction error obtained by SHRED with randomly
placed sensors to SHRED with QR placed sensors (QR SHRED), shallow decoder
networks with randomly placed sensors (R-SDN), shallow decoder networks with
QR placed sensors (Q-SDN), and linear reconstructions with QR placed sensors
(QR/POD). The considered reconstruction error is defined to be the averaged
mean square error over each state in a test set,

$\text{Error}=\frac{1}{T}\sum_{i=1}^{T}\frac{||\mathcal{H}\left(\\{\mathbf{y}_{j}\\}_{i-k}^{i}\right)-\mathbf{x}_{i}||_{2}}{||\mathbf{x}_{i}||_{2}}.$
(3)

Because SHRED models rely on trajectories of sensor measurements to perform
state estimation, we truncate each data set to reconstruct only the final
$N-k$ temporal snapshots, where $N$ is the initial number of samples and $k$
is the length of the utilized trajectories. This length can be viewed as a
hyper-parameter that can be tuned according to the data. Detailed network
parameters and training protocols can be found in the SI. Finally, we note
that in this section, training, validation, and test samples are temporally
interspersed. In later sections, we demonstrate results in the case that
training and test sets are temporally distinct.

###  II.1 Forced isotropic turbulent flow

The first application we consider is that of a forced isotropic turbulent flow
from the Johns Hopkins Turbulence Database li_public_2008 . The flow was
generated by direct numerical simulation using $1024^{3}$ nodes and the
pseudo-spectral method. We select a 350 by 350 cutout of computed pressure
over 1667 evenly spaced temporal snapshots from the simulation. We seek to
reconstruct these high-dimensional states from trajectories of point
measurements of the states. In this case, the length of the utilized
trajectories is selected to be 100. Correspondingly, the final 1567 temporal
snapshots are randomly split into training, validation, and test sets
consisting of 1100, 234, and 233 snapshots, respectively. For each considered
number of sensors, we generate 32 reconstructions with all considered methods.
We plot the median performance and denote the interquartile range by the
shaded region. These results are shown in panel B of Fig. 2. Even with only a
single, randomly placed sensor, the reconstructions obtained by SHRED yield
significantly lower error than competing methods with as many as 50 sensors.
Moreover, placement via the greedy QR algorithm appears to have a negligible
impact on reconstructive performance. Panel A of Fig. 2 shows sample
reconstructions obtained by SHRED and QR/POD with three sensors. While QR/POD
is only able to identify large scale features, SHRED accurately reconstructs
fine grain features as well.

In the vast majority of real world applications, and in contrast to numerical
simulations, data is often corrupted by noise. As a result, methods for state
estimation must exhibit resilience to noise. To measure this resilience, we
corrupt the data with Gaussian noise of mean zero and standard deviation of
$\alpha\times|\bar{x}|$, where $|\bar{x}|$ is the average absolute value over
all points in all snapshots of the training set. We then generate 32 state
estimates exactly as before using all considered methods. The resulting median
reconstruction error and interquartile range for varying $\alpha$ is shown in
panel C of Fig. 2. Again, SHRED outperforms competing techniques using a far
greater number of sensors.

###  II.2 Sea-surface temperature

The second application we examine is that of sea-surface temperature (SST). We
consider weekly mean sea-surface temperature from the years 1992 to 2019 as
reported by NOAA reynolds_improved_2002 . Unlike the previous example of a
simulated turbulent flow, SST is a sensor generated dataset for which
governing equations are not known and thus represents a more practical
application of SHRED. The data consists of 1400 snapshots of a 180 by 360
grid, of which 44219 spatial locations correspond to the sea-surface. We allow
the input trajectories to SHRED to have a length of 52, corresponding to one
year of measurements. The final 1348 samples are divided into training, test,
and validation sets consisting of 1000, 174, and 174 snapshots, respectively.
Analogous to Fig. 2, Fig. 3 shows example reconstructions and reconstruction
error distributions of SHRED and existing techniques both in the presence and
absence of added Gaussian noise. Again, the reconstructions obtained by SHRED
are both visually and empirically superior, while requiring far fewer sensors,
which can be randomly placed without a loss of accuracy.

###  II.3 Atmospheric ozone concentration

The last system for which we consider the performance of SHRED is a simulation
of atmospheric chemistry. Chemical transport models (CTM) simulate the
evolution of an ensemble of interacting chemical species through a transport
operator velegar_scalable_2019 ; bey_global_2001

$\frac{\partial n_{i}}{\partial t}=-\nabla\cdot(n_{i}\mathbf{U})$ (4)

and a chemical operator

$\frac{dn_{i}}{dt}=(P_{i}-L_{i})(\mathbf{n})+E_{i}-D_{i},$ (5)

where each entry $\mathbf{n}=\begin{bmatrix}n_{1}&n_{2}&\cdots
n_{K}\end{bmatrix}^{T}$ represents the number density of a specific chemical
species, $\mathbf{U}$ is the wind vector, $(P_{i}-L_{i})(\mathbf{n})$ is the
local chemical production and loss term, $E_{i}$ the emission rate of a
species, and $D_{i}$ the deposition rate. The output of a CTM, then, consists
of concentrations for $K$ chemical species for a grid of latitudes,
longitudes, and elevations. The data we consider is drawn from the work of
Velegar et al. velegar_scalable_2019  and contains simulated atmospheric ozone
concentration generated by the CTM software GEOS-Chem bey_global_2001  over
the course of a year with dynamical time steps of 20 minutes. The accessed
data from velegar_scalable_2019  is a compressed SVD representation using the
first 50 POD modes. The decompressed data matrix consists of 26,208 snapshots
of a 46 by 72 by 30 (latitude, longitude, elevation) grid, from which we
further downsample to obtain 2,600 evenly temporally spaced global ozone
concentration fields. To account for the fact that the data matrix is
inherently rank 50, we add Gaussian noise with standard deviation
$\alpha=0.05$ to the data before performing any analyses. The resulting
snapshots are divided into training, validation, and test sets consisting of
2000, 300, and 300 entries, respectively.

Fig. 4 shows the performance of SHRED and existing techniques on this
atmospheric ozone experiment. SHRED still outperforms SDNs and QR/POD while
requiring fewer sensors and being agnostic towards sensor placement. However,
in this case when many sensors are available ($>50$) QR/POD performs
comparably to nonlinear reconstructions. This performance is an artifact of
the use of a compressed, rank 50 representation of the data.

\begin{overpic}[width=411.93767pt]{SSTBothPreds.png} \put(2.0,55.0){{A}}
\end{overpic} (a)

\begin{overpic}[width=411.93767pt]{SSTRandomPreds.png} \put(2.0,55.0){{B}}
\end{overpic} (b)

Figure 5: Forecasting error for sea-surface temperature. A LSTM is trained to
forecast QR placed, panel A, or randomly placed, panel B, sensor measurements
which are subsequently used to perform reconstructions using SHRED and gappy
POD. 16 forecasts are performed and the median error is denoted by the solid
lines, with the 25th and 75th percentiles of reconstruction error defining the
shaded region. The dashed line represents an ensembled forecast.

##  III Forecasting sensor measurements to predict the evolution of spatio-
temporal data

In this section, we explore how the expressiveness of SHRED models with few
sensors can be leveraged to perform forecasts of high-dimensional spatio-
temporal states using only a limited subsampling of the state. Forecasting the
evolution of high-dimensional states from sensor measurements is an
exceedingly challenging task, owing to the fact that doing so combines the
problem of system identification with the difficulties of forecasting. We
propose a two step approach that leverages the success of LSTMs for low-
dimensional time-series forecasting lindemann_survey_2021  and SHRED for state
estimation.

Let $\\{\mathbf{x}_{i}\\}_{1}^{T_{t}}$ represent a training dataset of
temporally ordered states with corresponding sensor measurements
$\\{\mathbf{y}_{i}\\}_{1}^{T_{t}}$. We train an LSTM network,
$\mathcal{G}\left(\\{\mathbf{y}_{i}\\}_{t-k}^{t}\right)$, to map between a
trajectory of sensor measurements and the subsequent measurement,
$\mathbf{y}_{t+1}$, by finding

$\mathcal{G}\in\operatorname*{arg\rm{}min}_{\widetilde{\mathcal{G}}\in\mathscr{G}}\sum_{i=1}^{N}||\mathbf{y}_{t+1}-\widetilde{\mathcal{G}}\left(\\{\mathbf{y}_{j}\\}_{i-k}^{i}\right)||_{2}$
(6)

using the ADAM optimizer on the training data. Doing so, we then forecast
beyond the time interval of the training data to obtain forecasted sensor
measurements $\\{\hat{\mathbf{y}}_{i}\\}_{T_{t}+1}^{T_{t}+p}$ for $p>1.$ The
forecasted measurements are used by a SHRED model, trained on
$\\{\mathbf{x}_{i}\\}_{1}^{T_{t}}$, to obtain forecasts of the high-
dimensional state $\mathbf{x}.$ Detailed training protocols are included in
SI.

The forecasted states $\\{\hat{\mathbf{x}}_{i}\\}_{T_{t}+1}^{T_{t}+p}$ are
evaluated against the ground truth for each $\Delta t$ forecast rather than
the mean error across all forecasts to demonstrate performance over forecasts
of varying length. We also include reconstructions from forecasted sensor
measurements obtained by gappy POD for comparison, akin to the work of
parish_time-series_2020 . Finally, we only show results for SST and turbulent
flow data, as the compressed form of the atmospheric ozone data is biased
towards both QR placement and POD reconstructions.

###  III.1 Sea-surface temperature

For the SST example, we select the first 85% of the dataset to act as training
data, the subsequent 20 snapshots as validation data, and the remainder as the
test set. A LSTM for forecasting is trained on the training data and sensor
measurements are forecast beyond the validation set. These forecasted sensor
measurements are then used to construct a forecast of the high-dimensional
spatio-temporal field using a trained SHRED model and gappy POD. We consider
the cases that the sensors are placed via QR or randomly. We perform 16 runs
for each experiment, and consider an ensembled forecast found by averaging the
forecast of each run. The results for QR placed and randomly placed sensors
are shown in Fig. 5. With QR placement, the forecasts obtained by SHRED
outperform that of POD in the short-term and are similar for longer forecast
horizons. However, with randomly placed sensors SHRED is still able to obtain
comparably accurate forecasts while POD fails to consistently yield faithful
reconstructions.

###  III.2 Turbulent flow

Unlike sea-surface temperature, even medium-range forecasts of a turbulent
flow are impossible due to the fact that the flow is not quasi-periodic or
stationary in nature. For this reason, we focus on achieving accurate short-
term forecasts in this application. To do so, we use the same scheme as
before, with the exception that the validation set is selected to occur
earlier in the training data. Of the 1567 snapshots of the turbulent flow with
sufficiently long preceding measurement histories, the first 1000 are selected
as training, followed by 50 samples selected for validation. The next 100
samples are also used for training and the remainder constitute the test set.
This scheme allows for more accurate short-term forecasts because the forecast
occurs directly after the training data, as opposed to directly after the
validation set. Figure 6 shows the results for the cases that sensors are
placed via QR and randomly. In both cases, the forecast obtained by SHRED
deteriorates quickly but greatly outperforms that obtained by POD
reconstructions.

##  IV Discussion

We have demonstrated SHRED as a method for system identification/state
estimation and the forecasting of high-dimensional time-series data. By
including sensor trajectories in the model for reconstruction, SHRED
outperformed competing state-of-the-art techniques based on static
reconstructions from gappy POD or shallow decoder networks. We considered
three high-dimensional, spatio-temporal datasets, a synthetically generated
forced turbulent flow, a simulation of atmospheric ozone concentration from a
chemical transport model, and weekly mean sea-surface temperature.

The superior performance of SHRED held across both interpolatory
(reconstruction) and extrapolatory (forecasting) regimes. In Section II, the
task was purely an interpolatory reconstruction. Training and test data, upon
which we evaluate reconstruction accuracy, were drawn from the same
distribution. As a result, overfitting to the training data was not an issue,
and SHRED was able to capture fine grain details of complex flows with as few
as one or three sensors. Applied in this manner, SHRED offers an attractive
method for compression of large datasets as well as a framework for developing
reduced order models of dynamical systems. The performance of SHRED in this
section is also indicative of the promise of SHRED for high-dimensional data
that is stationary or periodic. Notably, SHRED demonstrated remarkable
indifference towards sensor placement. In this interpolatory regime, issues
arising from the necessity of optimal sensor placement can be mitigated by
increasing the information content of inputs to a neural network through the
use of sensor trajectories. These results show that the sensor trajectories
encode a significant amount of information, just as is expected of time-
delayed embeddings of dynamics brunton2017chaos ; arbabi2017ergodic ;
lee2019linking ; pan2020structure ; kamb2020time ; bakarji2022discovering .

In Section III, the task at hand was forecasting the evolution of high-
dimensional spatio-temporal data. In contrast to Section II, training and test
sets are temporally separated and thus the results are extrapolatory in
nature. High-dimensional forecasts were accomplished by forecasting low-
dimensional sensor measurements and constructing states from the predicted
measurements. Forecasting is an inherently difficult task for complex systems,
especially in the high-dimensional fields often encountered in the engineering
and the physical sciences. Still, SHRED’s use of sensor trajectories allows
for improved forecasting performance as compared to existing methods.
Furthermore, SHRED for forecasting exhibits robustness to sensor placement not
found in other state-of-the-art methods. We thus advocate for the use of SHRED
for both the tasks of state estimation and state prediction from forecasted
sensor measurements. Further work must be done to improve the accuracy of low-
dimensional sensor forecasts to be used in conjunction with SHRED for state
prediction. Improving the forecasts of high-dimensional data can enable on-
the-fly compression and novel sensing mechanisms.

\begin{overpic}[width=411.93767pt]{ISOBothPreds.png} \put(2.0,55.0){{A}}
\end{overpic} (a)

\begin{overpic}[width=411.93767pt]{ISORandomPreds.png} \put(2.0,55.0){{B}}
\end{overpic} (b)

Figure 6: Forecasting error for turbulent flow. A LSTM is trained to forecast
QR placed, panel A, or randomly placed, panel B, sensor measurements which are
subsequently used to perform reconstructions using SHRED and gappy POD. In
this experiment, the validation set is selected from earlier in the data so
that the forecast occurs immediately after following training data. 16
forecasts are performed and the median error is denoted by the solid lines,
with the 25th and 75th percentiles of reconstruction error defining the shaded
region. The dashed line represents an ensembled forecast.

##  V Materials and methods

###  V.1 Proper orthogonal decomposition and QR

Traditional techniques for high-dimensional state reconstruction from limited
sensor measurements typically rely on the SVD to compute a low-rank embedding
of the spatio-temporal dynamics. Given $N$ snapshots of an $m$ dimensional
state, we construct a data matrix
$\mathbf{X}=[\mathbf{x}_{1}\;\mathbf{x}_{2}\;\dots\mathbf{x}_{N}]$ where
$\mathbf{x}_{i}\in\mathbb{R}^{m}$ is the $i$-th temporal snapshot of the
state. The SVD factors the data matrix into the product of an orthogonal
matrix, $\mathbf{U}\in\mathbb{R}^{m\times m}$, a diagonal matrix with
decreasing entries, $\mathbf{\Sigma}\in\mathbb{R}^{m\times N},$ and another
orthogonal matrix $\mathbf{V}\in\mathbb{R}^{N\times N}.$ An optimal rank $r$
approximation of $\mathbf{X}$ can be directly obtained through the SVD by
retaining only the first $r$ columns of $\mathbf{U}$,

$\mathbf{X}\approx\hat{\mathbf{X}}=\mathbf{U}_{r}\mathbf{\Sigma}_{r}\mathbf{V}_{r}^{T}.$
(7)

Thus, each temporal snapshot $\mathbf{x}_{i}$ can be approximated by a linear
combination of the first $r$ columns of $\mathbf{U}$, otherwise known as the
dominant POD modes.

POD based methods estimate a high-dimensional state, $\mathbf{x}$, drawn from
the same distribution as that of the training data by solving for these $r$
coefficients using a subsampling of the state,

$\mathbf{y}=\mathbf{C}\mathbf{x}\approx\mathbf{C}\mathbf{U}_{r}\mathbf{b},$
(8)

where $\mathbf{C}$ is a “sensing” matrix consisting of rows of the $m\times m$
identity matrix and $\mathbf{b}\in\mathbb{R}^{r}$. A full-state estimate from
a set of sensor measurements, $\mathbf{y}$, can then be obtained by

$\mathbf{x}\approx\hat{\mathbf{x}}=\mathbf{U}_{r}(\mathbf{C}\mathbf{U}_{r})^{-1}\mathbf{y}.$
(9)

In practice, it is critical to design $\mathbf{C}$ such that the inversion in
(9) is well-conditioned. An indirect bound on the condition number of
$\mathbf{C}\mathbf{U}_{r}$ can be found by determining $\mathbf{C}$ where
$|\det(\mathbf{C}\mathbf{U}_{r})|$ is maximized. That is, we seek

$\displaystyle\mathbf{C}.=\operatorname*{arg\rm{}max}_{\mathbf{C}}|\det\mathbf{C}\mathbf{U}_{r}|=\operatorname*{arg\rm{}max}_{\mathbf{C}}\prod_{i}|\lambda_{i}(\mathbf{C}U_{r})|$
$\displaystyle=\operatorname*{arg\rm{}max}_{\mathbf{C}}\prod_{i}\sigma_{i}(\mathbf{C}U_{r}).$
(10)

Unfortunately, for large systems this $NP$-hard, combinatorial search becomes
computationally intractable. The QR decomposition with column pivoting of
$\mathbf{U}_{r}^{T}$ offers an approximate greedy solution. The decomposition
utilizes Householder transformations to find a column permutation matrix
$\mathbf{P}^{T}$ such that

$\mathbf{U}_{r}^{T}\mathbf{P}^{T}=\mathbf{Q}\mathbf{R}$ (11)

where $\mathbf{Q}$ is orthogonal and $\mathbf{R}$ is upper triangular with
decreasing, positive entries on the diagonal. Because at each iteration of the
algorithm the diagonal entry of $\mathbf{R}$ is selected to be as large as
possible and $\mathbf{Q}$ is orthogonal, we have a a greedy maximization of
$\det(\mathbf{U}_{r}^{T}\mathbf{P}^{T})=\det(\mathbf{P}\mathbf{U}_{r}).$ Thus,
setting $\mathbf{C}=\mathbf{P}$ can improve the reconstructions obtained by
POD based methods. We refer to reconstructions obtained in this manner as
QR/POD.

###  V.2 Shallow decoder networks

Although QR/POD methods have demonstrated success in a variety of fields, the
method can, at most, estimate the coefficients of the first $n$ POD modes if
there are $n$ sensors available. The included POD modes are insufficiently
expressive to obtain accurate reconstructions when exceedingly few sensors are
available. This limitation motivates the development of more expressive,
nonlinear methods for reconstruction. Among these are shallow decoder networks
(SDN) erichson_shallow_2020 .

As before, let

$\mathbf{y}=\mathbf{C}\mathbf{x}$ (12)

be a point sampling of a high-dimensional state $\mathbf{x}.$ SDNs are
shallow, fully-connected neural networks that learn a mapping from the space
of sensor measurements, $\mathbf{y}$, back to $\mathbf{x}.$ SDNs can be
denoted as

$\mathcal{F}(\mathbf{y};\mathbf{W}_{SD}):=R(\mathbf{W}^{b}R(\mathbf{W}^{b-1}\cdots
R(\mathbf{W^{1}s})))$ (13)

where $\mathbf{y}$ is the input sensor data, $R$ is a scalar, nonlinear
activation function, $\mathbf{W}_{SD}=\\{\mathbf{W^{i}}\\}_{i=1}^{b}$ are
trainable weights, and $k$ denotes the number of layers in the SDN. Formally,
we seek

$\mathcal{F}\in\operatorname*{arg\rm{}min}_{\widetilde{\mathcal{F}}\in\mathscr{F}}\sum_{i=1}^{N}||\mathbf{x}_{i}-\widetilde{\mathcal{F}}(\mathbf{y}_{i})||_{2},$
(14)

so that reconstruction error is minimized over a set of $N$ training states
$\\{\mathbf{x}_{i}\\}_{i=1}^{N}$. The parameters of the network are randomly
initialized and then trained by a gradient descent method. Once the network is
trained, reconstructions given subsequent sensor measurements are found by

$\hat{\mathbf{x}}=\mathcal{F}(\mathbf{y}).$ (15)

SDNs can be trained for any set of sensor measurements, and unlike QR/POD are
not entirely reliant on principled sensor placement schemes. Still, previous
empirical work suggests that even neural-network based reconstructions can be
improved through the use of QR placement williams_data-driven_2022 .

###  V.3 Recurrent neural networks and shallow recurrent decoders

Both SDNs and QR/POD have been used successfully in the reconstruction of
high-dimensional dynamical systems, but also to broader classes of images
manohar_data-driven_2018 . This extension is possible because both methods
rely only on static, point measurements and each reconstruction is performed
individually. Dynamical systems, on the other hand, inherently depend on the
temporal evolution of the state, and incorporating this dependence into a
method for reconstruction offers a natural improvement upon existing
techniques. Our work includes these dynamics through the use of a trajectories
of sensor measurements

To do so, we rely on recurrent neural networks (RNN) rumelhart_learning_1987
and, more specifically, long short-term memory networks (LSTM). The most basic
RNN accepts as an input a sequence of vectors $\\{\mathbf{v}_{i}\\}_{i=1}^{t}$
and outputs a single hidden state, $\mathbf{h}_{t}.$ This can be expressed
through the recursion relation

$\mathbf{h}_{t}=R(\mathbf{W}_{c}\mathbf{v}_{t}+\mathbf{W}_{r}\mathbf{h}_{t-1}+\mathbf{b}_{r})$
(16)

where $R$ is a scalar, nonlinear activation function, $\mathbf{W}_{c}$,
$\mathbf{W}_{r}$ and $\mathbf{b}_{r}$ are trainable weights, and
$\mathbf{h_{0}}=\begin{bmatrix}0&\cdots&0\end{bmatrix}^{T}$. Generic RNNs are
notorious for suffering from the vanishing gradient problem, causing them to
have difficulty in identifying long term dependencies. LSTMs address this
issue through the introduction of a so-called “gradient super-highway.”
Instead of outputting a single hidden state, LSTMs incorporate an additional
“cell state” which only undergoes minor, pointwise operations, allowing
gradients to flow easily from many time steps in the past. LSTMs are perhaps
the most commonly used class of RNN at the time of writing, due to their
ability to learn long and short term dependencies lindemann_survey_2021 . They
are frequently used in time-series forecasting, natural language processing,
and video analysis, among many other domains lindemann_survey_2021 . For a
more complete discussion of LSTMs, we direct the reader to
hochreiter_long_1997 . The recursion relation of an LSTM can be written as

$\displaystyle\mathbf{h}_{t}=\sigma\left(\mathbf{W}_{o}\begin{bmatrix}\mathbf{h}_{t-1},\mathbf{v}_{t}\end{bmatrix}+\mathbf{b}_{o}\right)\odot\tanh(\mathbf{c}_{t})$
(17)
$\displaystyle\mathbf{c}_{t}=\sigma\left(\mathbf{W}_{f}\begin{bmatrix}\mathbf{h}_{t-1},\mathbf{v}_{t}\end{bmatrix}+\mathbf{b}_{f}\right)\odot\mathbf{c}_{t-1}$
(18)
$\displaystyle+\sigma\left(\mathbf{W}_{i}\begin{bmatrix}\mathbf{h}_{t-1},\mathbf{v}_{t}\end{bmatrix}+\mathbf{b}_{f}\right)\odot\tanh\left(\mathbf{W}_{g}\begin{bmatrix}\mathbf{h}_{t-1},\mathbf{v}_{t}\end{bmatrix}+\mathbf{b}_{g}\right)$

for trainable weights and biases
$\mathbf{W}_{RN}=\\{\mathbf{W}_{o},\mathbf{W}_{f},\mathbf{W}_{i},\mathbf{W}_{g},\mathbf{b}_{o},\mathbf{b}_{f},\mathbf{b}_{i},\mathbf{b}_{g}\\}$.
Both $\sigma$, the sigmoid function, and $\tanh$ operate pointwise. Note that
$\mathbf{h}_{t}$ can be written as function in terms of only
$\\{\mathbf{v}_{i}\\}_{i=1}^{t}$ and parameterized by the trainable weights
and biases,

$\mathbf{h}_{t}=\mathcal{G}(\\{\mathbf{v}_{i}\\}_{i=1}^{t};\mathbf{W}_{RN}).$
(19)

We now introduce the SHallow REcurrent Decoder (SHRED), which merges an LSTM
with the SDN architecture. Suppose we have a multivariate time-series of a
high-dimensional state $\\{\mathbf{x}_{i}\\}_{i=1}^{T}$ and corresponding
measurements of the system $\\{\mathbf{y}_{i}\\}_{i=1}^{T}.$ Rather than
reconstruct $\mathbf{x}_{i}$ using only $\mathbf{y}_{i}$, as is done by QR/POD
and SDN, we let an LSTM learn a latent representation using the previous $k$
sets of sensor measurements. This latent representation is then used by a
shallow decoder to reconstruct the high-dimensional state. The SHRED
architecture can be written as

$\mathcal{H}\left(\\{\mathbf{y}_{i}\\}_{i=t-k}^{t}\right)=\mathcal{F}\left(\mathcal{G}(\\{\mathbf{y}_{i}\\}_{i=t-k}^{t};\mathbf{W}_{RN});\mathbf{W}_{SD})\right)$
(20)

using eqs. 13 and 19. As in the case of SDNs, we use a gradient descent method
to train the network weights to minimize reconstruction loss,

$\mathcal{H}\in\operatorname*{arg\rm{}min}_{\widetilde{\mathcal{H}}\in\mathscr{H}}\sum_{i=1}^{N}||\mathbf{x}_{i}-\widetilde{\mathcal{H}}\left(\\{\mathbf{y}_{i}\\}_{i=t-k}^{t}\right)||_{2}.$
(21)

$\mathcal{H}$, so defined, represents a map from sensor trajectories to the
full-state.

## Acknowledgements

The authors acknowledge support from the National Science Foundation AI
Institute in Dynamic Systems (grant number 2112085). JNK further acknowledges
support from the Air Force Office of Scientific Research (FA9550-19-1-0011 and
FA9550-19-1-0386).

## References

## References

  * (1) K. Manohar, B. W. Brunton, J. N. Kutz, and S. L. Brunton, “Data-Driven Sparse Sensor Placement for Reconstruction: Demonstrating the Benefits of Exploiting Known Patterns,” IEEE Control Systems Magazine, vol. 38, pp. 63–86, June 2018.  Conference Name: IEEE Control Systems Magazine.
  * (2) N. B. Erichson, L. Mathelin, Z. Yao, S. L. Brunton, M. W. Mahoney, and J. N. Kutz, “Shallow neural networks for fluid flow reconstruction with limited sensors,” Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 476, p. 20200097, June 2020.  Publisher: Royal Society.
  * (3) J. Yu and J. S. Hesthaven, “Flowfield Reconstruction Method Using Artificial Neural Network,” AIAA Journal, vol. 57, pp. 482–498, Feb. 2019.  Publisher: American Institute of Aeronautics and Astronautics.
  * (4) J. L. Callaham, K. Maeda, and S. L. Brunton, “Robust flow reconstruction from limited measurements via sparse representation,” Physical Review Fluids, vol. 4, p. 103907, Oct. 2019.  Publisher: American Physical Society.
  * (5) T. Bolton and L. Zanna, “Applications of Deep Learning to Ocean Data Inference and Subgrid Parameterization,” Journal of Advances in Modeling Earth Systems, vol. 11, no. 1, pp. 376–399, 2019.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1029/2018MS001472.
  * (6) J. N. Kutz, Data-Driven Modeling & Scientific Computation: Methods for Complex Systems & Big Data.  Oxford: Oxford University Press, 1st edition ed., Sept. 2013.
  * (7) S. L. Brunton and J. N. Kutz, Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control.  Cambridge: Cambridge University Press, 1st edition ed., Apr. 2019.
  * (8) R. Everson and L. Sirovich, “Karhunen–Loève procedure for gappy data,” JOSA A, vol. 12, pp. 1657–1664, Aug. 1995.  Publisher: Optica Publishing Group.
  * (9) Z. Drmač and S. Gugercin, “A New Selection Operator for the Discrete Empirical Interpolation Method—Improved A Priori Error Bound and Extensions,” SIAM Journal on Scientific Computing, vol. 38, pp. A631–A648, Jan. 2016.  Publisher: Society for Industrial and Applied Mathematics.
  * (10) M. Barrault, Y. Maday, N. C. Nguyen, and A. T. Patera, “An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations,” Comptes Rendus Mathematique, vol. 339, pp. 667–672, Nov. 2004.
  * (11) S. Chaturantabut and D. C. Sorensen, “Nonlinear Model Reduction via Discrete Empirical Interpolation,” SIAM Journal on Scientific Computing, vol. 32, pp. 2737–2764, Jan. 2010.  Publisher: Society for Industrial and Applied Mathematics.
  * (12) D. W. Carter, F. D. Voogt, R. Soares, and B. Ganapathisubramani, “Data-driven sparse reconstruction of flow over a stalled aerofoil using experimental data,” Data-Centric Engineering, vol. 2, p. e5, 2021.  Publisher: Cambridge University Press.
  * (13) S. Sahba, C. C. Wilcox, A. McDaniel, B. D. Shaffer, S. L. Brunton, and J. N. Kutz, “Wavefront sensor fusion via shallow decoder neural networks for aero-optical predictive control,” in Interferometry XXI, vol. 12223, pp. 11–17, SPIE, Oct. 2022.
  * (14) J. Williams, O. Zahn, and J. N. Kutz, “Data-driven sensor placement with shallow decoder networks,” Feb. 2022.  arXiv:2202.05330 [math].
  * (15) S. Boyd and L. Vandenberghe, “Convex Optimization,” Mar. 2004.  ISBN: 9780511804441 Publisher: Cambridge University Press.
  * (16) S. Joshi and S. Boyd, “Sensor Selection via Convex Optimization,” IEEE Transactions on Signal Processing, vol. 57, pp. 451–462, Feb. 2009.  Conference Name: IEEE Transactions on Signal Processing.
  * (17) W. F. Caselton and J. V. Zidek, “Optimal monitoring network designs,” Statistics & Probability Letters, vol. 2, pp. 223–227, Aug. 1984.
  * (18) A. Krause, A. Singh, and C. Guestrin, “Near-Optimal Sensor Placements in Gaussian Processes: Theory, Efficient Algorithms and Empirical Studies,” The Journal of Machine Learning Research, vol. 9, pp. 235–284, June 2008.
  * (19) D. V. Lindley, “On a Measure of the Information Provided by an Experiment,” The Annals of Mathematical Statistics, vol. 27, no. 4, pp. 986–1005, 1956.  Publisher: Institute of Mathematical Statistics.
  * (20) P. Sebastiani and H. P. Wynn, “Maximum Entropy Sampling and Optimal Bayesian Experimental Design,” Journal of the Royal Statistical Society. Series B (Statistical Methodology), vol. 62, no. 1, pp. 145–157, 2000\.  Publisher: [Royal Statistical Society, Wiley].
  * (21) L. Paninski, “Asymptotic theory of information-theoretic experimental design,” Neural Computation, vol. 17, pp. 1480–1507, July 2005.
  * (22) E. Clark, J. N. Kutz, and S. L. Brunton, “Sensor Selection With Cost Constraints for Dynamically Relevant Bases,” IEEE Sensors Journal, vol. 20, pp. 11674–11687, Oct. 2020.  Conference Name: IEEE Sensors Journal.
  * (23) E. Clark, S. L. Brunton, and J. N. Kutz, “Multi-Fidelity Sensor Selection: Greedy Algorithms to Place Cheap and Expensive Sensors With Cost Constraints,” IEEE Sensors Journal, vol. 21, pp. 600–611, Jan. 2021.  Conference Name: IEEE Sensors Journal.
  * (24) E. Clark, T. Askham, S. L. Brunton, and J. Nathan Kutz, “Greedy Sensor Placement With Cost Constraints,” IEEE Sensors Journal, vol. 19, pp. 2642–2656, Apr. 2019.  Conference Name: IEEE Sensors Journal.
  * (25) K. Manohar, E. Kaiser, S. L. Brunton, and J. N. Kutz, “Optimized Sampling for Multiscale Dynamics,” Multiscale Modeling & Simulation, vol. 17, pp. 117–136, Jan. 2019.  Publisher: Society for Industrial and Applied Mathematics.
  * (26) S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, pp. 1735–1780, Nov. 1997.
  * (27) Y. Li, E. Perlman, M. Wan, Y. Yang, C. Meneveau, R. Burns, S. Chen, A. Szalay, and G. Eyink, “A public turbulence database cluster and applications to study Lagrangian evolution of velocity increments in turbulence,” Journal of Turbulence, vol. 9, p. N31, Jan. 2008.  Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/14685240802376389.
  * (28) R. W. Reynolds, N. A. Rayner, T. M. Smith, D. C. Stokes, and W. Wang, “An Improved In Situ and Satellite SST Analysis for Climate.,” Journal of Climate, vol. 15, pp. 1609–1625, July 2002.  ADS Bibcode: 2002JCli…15.1609R.
  * (29) I. Bey, D. J. Jacob, R. M. Yantosca, J. A. Logan, B. D. Field, A. M. Fiore, Q. Li, H. Y. Liu, L. J. Mickley, and M. G. Schultz, “Global modeling of tropospheric chemistry with assimilated meteorology: Model description and evaluation,” Journal of Geophysical Research: Atmospheres, vol. 106, no. D19, pp. 23073–23095, 2001.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1029/2001JD000807.
  * (30) D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” Jan. 2017.  arXiv:1412.6980 [cs].
  * (31) M. Velegar, N. B. Erichson, C. A. Keller, and J. N. Kutz, “Scalable diagnostics for global atmospheric chemistry using Ristretto library (version 1.0),” Geoscientific Model Development, vol. 12, pp. 1525–1539, Apr. 2019.  Publisher: Copernicus GmbH.
  * (32) B. Lindemann, T. Müller, H. Vietz, N. Jazdi, and M. Weyrich, “A survey on long short-term memory networks for time series prediction,” Procedia CIRP, vol. 99, pp. 650–655, Jan. 2021.
  * (33) E. J. Parish and K. T. Carlberg, “Time-series machine-learning error models for approximate solutions to parameterized dynamical systems,” Computer Methods in Applied Mechanics and Engineering, vol. 365, p. 112990, June 2020\.
  * (34) S. L. Brunton, B. W. Brunton, J. L. Proctor, E. Kaiser, and J. N. Kutz, “Chaos as an intermittently forced linear system,” Nature communications, vol. 8, no. 1, pp. 1–9, 2017.
  * (35) H. Arbabi and I. Mezic, “Ergodic theory, dynamic mode decomposition, and computation of spectral properties of the koopman operator,” SIAM Journal on Applied Dynamical Systems, vol. 16, no. 4, pp. 2096–2126, 2017.
  * (36) S. Lee, F. Dietrich, G. E. Karniadakis, and I. G. Kevrekidis, “Linking gaussian process regression with data-driven manifold embeddings for nonlinear data fusion,” Interface focus, vol. 9, no. 3, p. 20180083, 2019\.
  * (37) S. Pan and K. Duraisamy, “On the structure of time-delay embedding in linear models of non-linear dynamical systems,” Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 30, no. 7, p. 073135, 2020.
  * (38) M. Kamb, E. Kaiser, S. L. Brunton, and J. N. Kutz, “Time-delay observables for koopman: Theory and applications,” SIAM Journal on Applied Dynamical Systems, vol. 19, no. 2, pp. 886–917, 2020.
  * (39) J. Bakarji, K. Champion, J. N. Kutz, and S. L. Brunton, “Discovering governing equations from partial measurements with deep delay autoencoders,” arXiv preprint arXiv:2201.05136, 2022.
  * (40) D. E. Rumelhart and J. L. McClelland, “Learning Internal Representations by Error Propagation,” in Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations, pp. 318–362, MIT Press, 1987.  Conference Name: Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations.


# “The Michael Jordan of Greatness”:
Extracting Vossian Antonomasia from Two Decades of
the _New York Times_ , 1987–2007††thanks:  This is a preprint. The final
published version of this article is available at DOI:10.1093/llc/fqy087.

Frank Fischer
National Research University
Higher School of Economics, Moscow
<EMAIL_ADDRESS>
Robert Jäschke
Humboldt-Universität zu Berlin
<EMAIL_ADDRESS>

###### Abstract

Vossian Antonomasia is a prolific stylistic device, in use since antiquity. It
can compress the introduction or description of a person or another named
entity into a terse, poignant formulation and can best be explained by an
example: When Norwegian world champion Magnus Carlsen is described as “the
Mozart of chess”, it is Vossian Antonomasia we are dealing with. The pattern
is simple: A source (Mozart) is used to describe a target (Magnus Carlsen),
the transfer of meaning is reached via a modifier (“of chess”). This
phenomenon has been discussed before (as ‘metaphorical antonomasia’ or, with
special focus on the source object, as ‘paragons’), but no corpus-based
approach has been undertaken as yet to explore its breadth and variety. We are
looking into a full-text newspaper corpus (_The New York Times_ , 1987–2007)
and describe a new method for the automatic extraction of Vossian Antonomasia
based on Wikidata entities. Our analysis offers new insights into the
occurrence of popular paragons and their distribution.

##  1 Introduction and Related Works

When Peter Paul Rubens is described as “the Tarantino of Baroque painting” (in
_Tagesspiegel_ , 2014) or Slavoj Žižek as “the Elvis of cultural theory” (in
_Slate_ , 2014), we deal with a phenomenon called Vossian Antonomasia (VA).
This trope is named after Gerardus Vossius (1577–1649), the Dutch humanist and
professor of rhetoric, who first described VA as a specific sub-phenomenon of
antonomasia. Generally speaking, an antonomasia incorporates the idea of a
particular property of a person standing in for this person (e. g., “The
Voice” standing in for Frank Sinatra). In the case of _Vossian_ Antonomasia, a
person is attributed a specific property by reference to another (more well-
known, more popular, more notorious) person. An “atypological, modifying
signal (pronoun, adjective, genitive)” provides for the transfer of meaning
(Lausberg, 1960). “Baroque painting” and “cultural theory” work as such in the
above-mentioned examples. In other words, VA uses a ’modifier’ to establish a
relationship between a ‘source’ and a ‘target’ (Bergien, 2013). In our paper,
we will study this popular variant, although VA generally works without such
modifier (as in “Go and denounce me, you Judas!”, where Judas stands for
"traitor" and no modifier is presented). Entities can occur both as source and
as target, as is shown by the example of Obama in Bergien (2013): until 2011
he mostly appeared as target, but increasingly as source thereafter. Following
Lakoff (1987), the entity serving as ‘source’ is also referred to as
‘paragon’, “a specific example that comes close to embodying the qualities of
the ideal” (Bergien, 2013).

Internationally, the term ‘Vossian Antonomasia’ is rarely used; instead, we
encounter a distinction between ‘Antonomasia1’ and ‘Antonomasia2’: ‘metonymic’
vs. ‘metaphorical antonomasia’ (Holmqvist and Płuciennik, 2010). In this
classification scheme, Vossian Antonomasia would occur within ‘Antonomasia2’,
in the form of “comparisons with paragons from other spheres of culture:
Lyotard is a pope of postmodernism, Bush is no Demosthenes; and we can buy the
Cadillac of vacuum cleaners.” (Holmqvist and Płuciennik, 2010)

The use of this stylistic device dates back to classical antiquity. Nowadays,
it is a welcome element in many journalistic genres and often occurs in
headings, since it can be both informative and enigmatic, and is notorious for
its unique entertaining qualities. A larger collection of examples111See
https://www.umblaetterer.de/datenzentrum/vossianische-antonomasien.html and
Fischer and Wälzholz (2014). gave the impetus to research this phenomenon more
systematically, with historical perspective and on the basis of larger English
and German newspaper corpora (Jäschke et al., 2017). The aim of this paper is
an exploratory automated analysis of VA in the _New York Times_ (1987–2007).
The object of study was chosen because we were looking for a corpus of a
general contemporary English-language newspaper and found the NYT Annotated
Corpus a perfect fit being readily available as it was released to the
research community (Sandhaus, 2008).

##  2 Extraction and Evaluation

The method described in Jäschke et al. (2017) was based on NLTK’s NER module
and complex hand-crafted regular expressions. This resulted in low precision
(few of the resulting candidates were VA) and recall (only few VA were
detected). Reason enough to work on a novel method for the extraction of VA.
We did so by restricting our search to the clear-cut pattern “the SOURCE of”.
This pattern limitation lead to the extraction of a much larger set of VA than
before (by a factor of 10) with just minimal additional effort. It has to be
noted that this strategy works well for our English-language corpus, but would
have to be adjusted for languages that do not use articles or prepositions to
determine cases.

To overcome the shortcomings of current NER implementations we resorted to
Wikidata, the crowd-sourced fact database (Vrandečić and Krötzsch, 2014), from
which we extracted a list of possible sources of a VA expression. In most
other use-case scenarios, using Wikidata for named-entity matching would lead
to a deterioration or limitation of the results. But in our very special case,
this simple idea was the breakthrough to achieve clean extractions and to
immensely increase precision. Because one prerequisite for being a source of a
VA is to be famous or popular or notorious for something, and this arguably
goes along with being present on at least one of the hundreds of Wikipedia
versions and thus having a Wikidata entry.

By help of the Wikidata Toolkit222See
https://www.mediawiki.org/wiki/Wikidata˙Toolkit. we downloaded and filtered
the Wikidata dump from August 14, 2017. We used the instanceOf333See
https://www.wikidata.org/wiki/Property:P31. property to retrieve all instances
of the class ‘human’.444See https://www.wikidata.org/wiki/Q5. For each
individual we retrieved the English representation of their name and potential
alias names. The resulting list comprises 2,801,931 individuals with 2,955,761
unique names.

From the XML-encoded _New York Times_ corpus (Sandhaus, 2008), we extracted
the full text for each of the 1,854,726 articles and split it into sentences
using the Natural Language Toolkit, NLTK (Bird et al., 2009). Within each
sentence we then searched for character sequences matching the regular
expression `(\\bthe\\s+([\\w.,’-]+\\s+){1,5}?of\\b)`.

It matches all phrases beginning with “the” and ending with “of”, with up to
five words in-between. We had to increase this margin to be able to include
dots, commas, apostrophes and hyphens, allowing us to match sources like
“Robert Downey, Jr.”, “Shaquille O’Neal”, or “Jean-Luc Godard”. Each sentence
containing a matched phrase was stored together with the extracted source
candidate.

We then tried to match each source candidate against the Wikidata list of
individuals. This effectively reduced the number of candidate sentences from
11,452,615 to 96,731, but still included one-word source candidates like
“Church” or “Hall”, caused by ambiguous nicknames or aliases of some persons
stored in Wikidata,555E. g., https://www.wikidata.org/wiki/Q21508608. or
incomplete data.666 E. g., missing first names, see
https://www.wikidata.org/wiki/Q5116540. We decided to filter out such cases
with a curated blacklist, which mainly concerned one-word aliases (of the
1,289 entries in the blacklist, only 62 consist of more than one word).777The
blacklist and all other data can be found in our working repository at
https://github.com/weltliteratur/vossanto and via the project website at
https://vossanto.weltliteratur.net/. After filtering the source candidates by
help of our blacklist, we were down to 3,753 sentences from 20 years of _New
York Times_ containing possible VA.

While we cannot provide numbers for recall due to missing gold annotations for
the VA phenomenon, we can present numbers for the precision of our extraction
method. Compared to other stylistic devices, VA are a notably rare phenomenon
– we were able to evaluate a candidate list of this magnitude manually.

The final result is a list of 2,775 sentences containing VA, making for a
73.9% precision score. (While these are all cases of valid VA, we have
eliminated duplicates for the rankings in Section 3 if an expression
reoccurred in a text or if an article was republished, trimming the list to
2,646 unique occurrences of VA.) Figure 1a shows these numbers on a per-year
basis (the corpus ends early in 2007, which explains the abrupt decrease in
that year). The slight increase of the curves in Figure 1b (relative
frequency) suggests that the use of VA increased over the years covered by our
corpus. Since VA are typically built around a popular person, a paragon, it is
unlikely that Wikipedia/Wikidata has significant gaps for the time covered by
our corpus, but we have to at least consider that Wikipedia, and as a
consequence, Wikidata, too, might have a contemporary bias.

(a) Absolute numbers (b) Relative frequency (one per thousand)

Figure 1: VA candidates and true VA extracted per year. The right vertical
axes measure the resulting precision (percentage of true VA among the
candidates).

##  3 Results

###  3.1 Most Frequent Sources

The 40 most frequent sources are shown in Table 1. They include artists,
people from sports, business, the culture industry, historical figures, and
scientists. While especially the upper segment of the table is dominated by
(US-American) men, Madonna is the most frequent female paragon in our corpus,
followed by Martha Stewart, Greta Garbo, Mother Teresa, Oprah Winfrey, Rosa
Parks, Susan Lucci, and Julia Child.

Table 1: The forty most frequent VA sources extracted from our corpus (‘the …of’). count | source
---|---
68 | Michael Jordan
58 | Rodney Dangerfield
36 | Babe Ruth
32 | Elvis Presley
31 | Johnny Appleseed
23 | Bill Gates
21 | Pablo Picasso
21 | Michelangelo
21 | Donald Trump
21 | Jackie Robinson
21 | Madonna
20 | P. T. Barnum
20 | Tiger Woods
18 | Martha Stewart
16 | Henry Ford
16 | William Shakespeare
16 | Wolfgang Amadeus Mozart
15 | Adolf Hitler
14 | Greta Garbo
14 | John Wayne
14 | Mother Teresa
13 | Napoleon
13 | Ralph Nader
12 | Leonardo da Vinci
12 | Cal Ripken
12 | Leo Tolstoy
12 | Oprah Winfrey
12 | Rosa Parks
12 | Susan Lucci
11 | Walt Disney
11 | Willie Horton
11 | Rembrandt
10 | Albert Einstein
10 | Thomas Edison
10 | Mike Tyson
10 | Julia Child
9 | Ross Perot
9 | Dennis Rodman
8 | James Dean
8 | Mikhail Gorbachev

The top source, Michael Jordan, is followed by Rodney Dangerfield, a stand-up
comedian preeminently known for his catchphrase “I don’t get no respect!”. He
thus finally gets some respect, at least as a highly productive source for VA.
Next in the ranking are Babe Ruth, an American baseball player, and Elvis
Presley. Altogether, Table 1 assembles international celebrities, some of
which are prone to appear beyond the _New York Times_. Yet, many references
are not likely as sources for VA in other nations’ newspapers (just take all
the baseball references), which confirms the notion that VA is a language- and
culture-dependent phenomenon (Kövecses, 2005; Jäschke et al., 2017).

###  3.2 High-Frequent Modifiers

Looking at the most frequent modifiers, we find proof for one of the main
functions of VA, which is inculturation (Holmqvist and Płuciennik, 2010). Next
to unspecific modifiers such as “of his [or her] day”, “of his [or her] time”,
etc., we can find toponyms such as Japan, China, and Brazil (Table 2). This
becomes even clearer if we limit the output to just countries (Table 3).

The same can be said about other kinds of modifiers, such as sports (baseball,
tennis, basketball, hockey, football). They can transfer meaning between
different disciplines, like in this sentence: “It’s one thing to win a gold
medal, but the way he [Andre Agassi] dominated, he was the Michael Johnson of
the tennis field today.” (NYT, 1996) But given the right source, they can also
inherit meaning from other domains, as shows this example: “[MBL pitcher Greg]
Maddux is the Picasso of baseball.” (NYT, 2001)

Table 2: Most frequent modifiers. count | modifier
---|---
55 | his day
33 | his time
29 | Japan
16 | tennis
16 | his generation
16 | baseball
15 | China
13 | her time
13 | her day
12 | our time
11 | the 1990’s
10 | the Zulus
10 | the 90’s
10 | politics
10 | hockey
10 | Brazil
10 | basketball
10 | ballet
9 | jazz
9 | fashion
8 | today
8 | Israel
8 | his era
8 | hip-hop
8 | golf
8 | dance
7 | the art world
7 | the 19th century
7 | Long Island
7 | Iran
Table 3: Selected countries serving as modifiers. count | country
---|---
29 | Japan
15 | China
10 | Brazil
8 | Israel
7 | Iran
7 | India
4 | South Africa
4 | Mexico
3 | Spain
3 | South Korea
3 | Russia
3 | Poland
3 | Pakistan

###  3.3 Distribution of VA

We also gained insights into the distribution of VA across our corpus. VA as a
phenomenon is particularly popular in the cultural pages (“Arts”, “Books”,
“Movies”, “Theater”, “Magazine”) and the “Sports” section, as indicated in
Table 4. In contrast to that, authors in the “Business” section of the _New
York Times_ (the section containing the most articles in our corpus, see Table
5) clearly think twice before operating a VA expression into their texts. The
popularity of VA especially on the cultural pages is confirmed by the example
of German weekly _Die Zeit_ studied in Jäschke et al. (2017) where the paper’s
“Feuilleton” section exhibits the highest frequency of VA. Our quantitative
approach also backs up findings of qualitative research on antonomasia in the
“Sports” section of Croatian media (Grgić and Nikolić, 2013).

Table 4: Occurrences of VA by newspaper section. Categories are as extracted from corpus, this includes empty category fields. VA | 2,646 | category | articles | 1,854,726
---|---|---|---|---
336 | 12.7% | Sports | 160,888 | 8.7%
334 | 12.6% | Arts | 88,460 | 4.8%
290 | 11.0% | New York and Region | 221,897 | 12.0%
237 | 9.0% | Arts; Books | 35,475 | 1.9%
158 | 6.0% | Movies; Arts | 27,759 | 1.5%
109 | 4.1% | Business | 291,982 | 15.7%
102 | 3.9% | Opinion | 134,428 | 7.2%
96 | 3.6% | U.S. | 89,389 | 4.8%
95 | 3.6% | Magazine | 11,464 | 0.6%
62 | 2.3% | Style | 65,071 | 3.5%
61 | 2.3% | Arts; Theater | 13,283 | 0.7%
46 | 1.7% | World | 79,786 | 4.3%
39 | 1.5% | Home and Garden; Style | 13,978 | 0.8%
32 | 1.2% | Travel | 10,440 | 0.6%
31 | 1.2% | Technology; Business | 23,283 | 1.3%
27 | 1.0% |  | 42,157 | 2.3%
25 | 0.9% | Week in Review | 17,107 | 0.9%
25 | 0.9% | Home and Garden | 5,546 | 0.3%
17 | 0.6% | World; Washington | 24,817 | 1.3%
17 | 0.6% | Style; Magazine | 1,519 | 0.1%
Table 5: Number of articles by newspaper category. articles | 1,854,726
---|---
categories | 1,580
Business | 291,982
Sports | 160,888
Opinion | 134,428
U.S. | 89,389
Arts | 88,460
World | 79,786
Style | 65,071
Obituaries | 19,430
Magazine | 11,464
Travel | 10,440

###  3.4 Authors with a Thing for VA

One next step was to see if there are certain authors with a thing for VA. If
so, we could assume that at least some authors would use VA very consciously
being well aware of its stylistic impact. Table 6 presents some authors with
an inclination to use VA.

Table 6: Ranking of VA-embracing authors (absolute number of VA per author – percentage among all VA extracted – total number of articles per author – percentage among all articles in the corpus). Some author fields in the corpus were empty, so from the corpus itself we do not know for 15.5% of extracted VA who authored them. VA | 2,646 | author | articles | 1,854,726
---|---|---|---|---
411 | 15.5% |  | 961,052 | 51.8%
30 | 1.1% | Holden, Stephen | 5,098 | 0.3%
29 | 1.1% | Maslin, Janet | 2,874 | 0.2%
26 | 1.0% | Vecsey, George | 2,739 | 0.1%
23 | 0.9% | Sandomir, Richard | 3,140 | 0.2%
22 | 0.8% | Ketcham, Diane | 717 | 0.0%
20 | 0.8% | Kisselgoff, Anna | 2,661 | 0.1%
19 | 0.7% | Dowd, Maureen | 1,647 | 0.1%
19 | 0.7% | Berkow, Ira | 1,704 | 0.1%
18 | 0.7% | Kimmelman, Michael | 1,515 | 0.1%
17 | 0.6% | Brown, Patricia Leigh | 568 | 0.0%
16 | 0.6% | Pareles, Jon | 4,090 | 0.2%
16 | 0.6% | Chass, Murray | 4,544 | 0.2%
15 | 0.6% | Smith, Roberta | 2,497 | 0.1%
15 | 0.6% | Lipsyte, Robert | 817 | 0.0%
15 | 0.6% | Grimes, William | 1,368 | 0.1%
15 | 0.6% | Barron, James | 2,188 | 0.1%
15 | 0.6% | Anderson, Dave | 2,735 | 0.1%
14 | 0.5% | Stanley, Alessandra | 1,437 | 0.1%
14 | 0.5% | Haberman, Clyde | 2,492 | 0.1%

###  3.5 Anti-VA

As we have shown in Table 1, Michael Jordan is a universal super ‘source’ for
VA, a meme in its own right (Lovinger, 1999). The _Wall Street Journal_ runs a
website collecting all kinds of occurrences of “Michael Jordans of …”,
counting 1,505 occurrences in different media to date (Cohen et al., 2015). On
top of that, Barack Obama, in his honorific speech for Michael Jordan when
awarding him the Medal of Freedom in November 2016, said the following: “There
is a reason you call someone the Michael Jordan of [something]. They know what
you’re talking about because Michael Jordan is the Michael Jordan of
greatness. He is the definition of somebody so good at what they do that
everybody recognizes it. That’s pretty rare.” (Payne, 2016)

Obama stretches VA past its breaking point by implementing recursivity, using
an identical source and target (“Michael Jordan is the Michael Jordan of
greatness”). Another way of doing that would be to use a non-modifying
modifier, as has been acted out by German author Joachim Lottmann, who
ironically referred to himself as “the German version of Rainald Goetz”
(Seidl, 2004). Since Rainald Goetz, the ‘source’ for this VA, is a German
author himself, the whole formulation stops making sense as the modifier
wasn’t used to transfer the meaning into a different field, which lets this VA
expression explode in irony. Another example for this sub-phenomenon will
demonstrate that we are not talking about a single occurrence here: “I am
[right now] in Norway, the Sweden of Scandinavia,” said Swiss satirist Gabriel
Vetter in a recent interview, also leaving the modifier un-modified (Fopp,
2016).

We will conclude this parade of examples with a third class of anti-VA,
quoting a passage which was returned when processing the NYT corpus: “During
their decade-long marriage, Elizabeth Taylor quipped that Burton was ‘the
Frank Sinatra of Shakespeare.”’ (NYT, 12 March 1989) The use of another human
as modifier (here standing _pars pro toto_ for Shakespeare’s oeuvre, too) is
another attempt to wrap VA around itself. The same pattern is found in a well-
known German example, Eckhard Henscheid’s satirical opera guide “Verdi ist der
Mozart Wagners” (1979). The description of these three ways to direct VA
against itself was only possible because our exploratory large-scale analysis
sharpened our senses for these kinds of absurd antonomasia.

##  4 Summary and Outlook

We started this project with the notion that Vossian Antonomasia is a
stylistic device with a simple structure. It turned out, however, that it is a
far richer phenomenon than we thought. Writing extraction rules for this
phenomenon is anything but trivial. By resorting to Wikidata, we circumvented
shortcomings of available NER tools. By discussing the most frequent VA
extracted from the _New York Times_ corpus we added a quantifying approach to
the discourse around paragons. We also presented first insights into the
distribution of VA and found that certain journalistic genres have an
inclination towards using this device, whereas others avoid it (“Sports” and
“Arts” sections vs. “Business”). We also confirmed that VA is a language- and
culture-dependent phenomenon, much like other stylistic devices (Kövecses,
2005).

Our approach comes with known limitations. For one thing, we have focused our
attention on just one grammatical variant of VA. Although our pragmatic
limitation to the pattern “the …of” aimed at the most common shape of VA, we
thereby missed expressions like “the American Oscar Wilde”, “Harlem’s Mozart”,
or “the Japanese Nolan Ryan” (all taken from the NYT corpus), which certainly
leaves room for further work. The same can be said about our limitation to
persons as source entities (living or dead, but in any case non-fictional and
non-mythological). While working with our corpus we made some observations,
though, that we don’t want to withhold:

  * •

Productive ‘sources’ for VA can be real persons as well as mythological or
fictional characters (productive examples for the latter include Woody Allen’s
“Zelig”, Cervantes’ “Don Quixote” or James Hilton’s “Mr. Chips”, but also
sources like “the [Santa Claus, Satan, Midas, Godzilla, King Kong, Cinderella,
Pied Piper, Robin Hood, Energizer Bunny, Cassandra, Jupiter, Icarus] of –”.

  * •

Animals can be targets, even if the source is a human, like in “ _Sea Hero_ is
the Bobo Holloman of racing” (NYT, 1993) or “ _Bonfire_ , the Michael Jordan
of dressage horses” (NYT, 1998).

  * •

Personifications can be part of a VA expression, that is, the use of
individual persons/characters as sources for companies, clubs, bands or places
as ‘targets’, like in “ _Sturm, Ruger_ is the _Benedict Arnold_ of the gun
industry” (NYT, 1989), “ _Aerosmith_ , the _Dorian Gray_ of rock bands” (NYT,
1993), “the _Hudson_ has been the _John Barrymore_ of rivers, noble in profile
but a sorry wreck” (NYT, 1996), “the _National Collegiate Athletic
Association_ , the _Kenneth Starr_ of sports” (NYT, 1998).

In conclusion, we would like to propose more holistic approaches to the
phenomenon rather than just building on Lakoff’s ‘paragons’, that is, the
‘sources’. With quantitative explorations of Vossian Antonomasia like the one
presented here, we can capture the phenomenon as a whole, including source,
target and (if provided) modifier. We will find a network of people, connected
by a diversity of modifiers, with nodes that can be sources or targets (or
both, as shown by the Obama example), a network that can help us understand
the hidden patterns of role models and how they differ from country to
country, from language to language.

##  5 Acknowledgement

We would like to thank the Information School at the University of Sheffield
for granting one of the authors (F. F.) a visiting scholarship in spring 2017
which helped to finish our research.

## References

  * Bergien (2013) Angelika Bergien. 2013.  Names as frames in current-day media discourse.  In _Name and Naming. Proceedings of the second international conference on onomastics_ , pages 19–27, Cluj-Napoca. Editura Mega.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper. 2009.  _Natural Language Processing with Python_ , 1st edition.  O’Reilly Media, Inc.
  * Cohen et al. (2015) Ben Cohen, Geoff Foster, Matthew Oshinsky, Jon Keegan, and Tom McGinty. 2015.  The Michael Jordan of ….  Website.
  * Fischer and Wälzholz (2014) Frank Fischer and Joseph Wälzholz. 2014.  Jeder kann Napoleon sein. Vossianische Antonomasie: Eine Stilkunde.  _Frankfurter Allgemeine Sonntagszeitung_ , page 34.  Dec 21, 2014.
  * Fopp (2016) Andrea Fopp. 2016.  “Die Wurst ist Geheimnis, ist Gefahr, ist Geisterbahn” [interview with Gabriel Vetter].  _TagesWoche_.  Sep 1, 2016.
  * Grgić and Nikolić (2013) Ana Grgić and Davor Nikolić. 2013.  The Cowboys, the Poets, the Professor …– Antonomasia in Croatian Sports Discourse.  In G. Kišiček and I.Ž. Žagar, editors, _What Do We Know about the World? Rhetorical and Argumentative Perspectives_ , pages 408–428. University of Windsor; Pedagoški inštitut, Windsor; Ljubljana.
  * Holmqvist and Płuciennik (2010) Kenneth Holmqvist and Jarosław Płuciennik. 2010.  Princess Antonomasia and the Truth: Two Types of Metonymic Relations.  In Armin Burkhardt and Brigitte Nerlich, editors, _Tropical Truth(s). The Epistemology of Metaphor and Other Tropes_ , pages 373–381. De Gruyter, Berlin/New York.
  * Jäschke et al. (2017) Robert Jäschke, Jannik Strötgen, Elena Krotova, and Frank Fischer. 2017.  “Der Helmut Kohl unter den Brotaufstrichen”. Zur Extraktion Vossianischer Antonomasien aus großen Zeitungskorpora.  In _Proceedings of DHd 2017_ , DHd ’17, pages 120–124. Digital Humanities im deutschsprachigen Raum.
  * Kövecses (2005) Zoltán Kövecses. 2005.  _Metaphor in Culture: Universality and Variation_.  Cambridge University Press, Cambridge.
  * Lakoff (1987) George Lakoff. 1987.  _Women, Fire, and Dangerous Things. What Categories Reveal about the Mind_.  The University of Chicago Press, Chicago.
  * Lausberg (1960) Heinrich Lausberg. 1960.  _Handbuch der literarischen Rhetorik. Eine Grundlegung der Literaturwissenschaft_.  Hueber, München.
  * Lovinger (1999) Caitlin Lovinger. 1999.  They’re Sort of Like Mike.  _New York Times_.  Jan 17, 1999.
  * Payne (2016) Marissa Payne. 2016.  Obama tells Michael Jordan he’s ‘more than just an Internet meme,’ then makes MJ tear up.  _Washington Post_.  Nov 22, 2016.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The New York Times Annotated Corpus LDC2008T19.  DVD, Linguistic Data Consortium, Philadelphia.
  * Seidl (2004) Claudius Seidl. 2004.  Jugend ohne Sex. Ein Manifest gegen die Wirklichkeit: Joachim Lottmanns Roman “Die Jugend von heute”.  _Frankfurter Allgemeine Sonntagszeitung_ , page 34.  Oct 17, 2004.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A Free Collaborative Knowledgebase.  _Communications of the ACM_ , 57(10):78–85.


# Finite temperature QCD at fixed Q with overlap fermions

JLQCD Collaboration: a, Sinya Aokib, Shoji Hashimotoa,c, Takashi Kanekoa,c,
Hideo Matsufurua, Jun-ichi Noakia, Eigo Shintanid

aTheory Center, IPNS, High Energy Accelerator Research Organization (KEK),
Tsukuba 305-0801, Japan
bGraduate School of Pure and Applied Sciences, University of Tsukuba, Tsukuba
305-8571, Japan
cSchool of High Energy Accelerator Science, The Graduate University for
Advanced Studies (Sokendai), Tsukuba 305-0801, Japan
dRIKEN-BNL Research Center, Upton, NY 11973-5000, USA E-mail:
<EMAIL_ADDRESS>
###### Abstract:

We present some preliminary results of the project on finite temperature QCD
with overlap fermions at KEK. We performed a series of simulations to assess
the effects of fixing the topological sector at finite temperature and we will
show the first calculations of topological susceptibility and meson masses for
quenched and full QCD.

##  1 Introduction and motivation

Among several features of QCD the chiral symmetry breaking is one of the most
interesting ones. At low temperature this symmetry is spontaneously broken and
the vacuum develops a quark anti-quark condensate, $\langle\bar{q}q\rangle\neq
0$. Massless Nambu-Goldstone (NG) bosons should appear in the spectrum of the
massless theory. In real QCD, of course, the light quark masses explicitly
break the chiral symmetry, giving a small mass to the NG bosons that have been
identified as the 8 lightest mesons (pions, kaons, $\eta$). Classically, the
pattern of chiral symmetry breaking is the following ($N_{f}$ being the number
of light quarks)111$U(1)_{V}$ gives the conserved baryon number and
$SU(N_{f})_{V}$ is only softly broken by the small quark mass difference.:

$SU(N_{f})_{V}\times SU(N_{f})_{A}\times U(1)_{V}\times U(1)_{A}\rightarrow
SU(N_{f})_{V}\times U(1)_{V}$ (1)

that should actually give 9 NG bosons, but the ninth particle is absent in the
spectrum. This apparent problem was solved noticing that the flavor-singlet
axial $U(1)_{A}$ rotation is no more a symmetry at the quantum level (even in
the massless limit); it is anomalous [1] due to the presence of instanton-like
configurations. A direct effect of the anomaly is the large splitting in the
mass of flavor-singlet and non-singlet pseudoscalar mesons (see Witten-
Veneziano formula [2, 3]).

While at zero temperature the physics is quite clear, at finite temperature
still there is no definite answer to the question if axial $U(1)_{A}$ symmetry
is restored or not. If it is restored, an interesting problem is to establish
if this happens at the same critical temperature of chiral symmetry
restoration. This would have relevant effects on the pattern of symmetry
breaking and so on the critical exponents of the phase transition [4].

By semiclassical calculations of dilute instanton gas at very high temperature
$T\gg T_{c}$, we expect a strong suppression but not an exact restoration of
the $U(1)_{A}$ symmetry. The most advanced lattice result in this context is
the one by Vranas [5], where, using domain wall fermions, he concluded that
just above chiral phase transition the $U(1)_{A}$ symmetry remains broken but
by a very small amount. It is an open question what is the effect that may
have to the order of the transition.

Our aim is to study the fate of chiral and $U(1)_{A}$ symmetry (and the mass
of particles) at finite temperature around and above the phase transition
using the fermionic action that retains the maximal amount of chirality on the
lattice, i.e. the overlap formulation [6]. The JLQCD and TWQCD collaborations
have performed large scale QCD simulations using the overlap action [7]. All
simulations were done at zero temperature and we investigated the chiral
behavior of spectra, low energy constants, chiral condensate and topological
susceptibility [8, 9, 10, 11]. A non-zero topological susceptibility,
indicating anomalous breaking of $U(1)_{A}$ symmetry, was clearly observed in
those simulations at $T=0$ and also the linear dependence with sea quark mass
was obtained, as predicted by chiral perturbation theory.

In order to simulate QCD using HMC with dynamical overlap fermions, fixing the
topological sector was crucial because allowing for topology change would be
extremely expensive [12]. To run a simulation at fixed topological charge $Q$
it was introduced an irrelevant term in the action [12], that suppresses the
occurrence of zero eigenvalues of the hermitian Wilson-Dirac operator that
have to be crossed in order to change topological sector. Fixing topology, of
course, creates a bias in physical results that must be corrected. A full
theory describing the effects of working at fixed $Q$ was developed [13]: the
effects at zero temperature are understood, under control and $O(1/V)$. We
will discuss with more details the subject in the following section.

In order to obtain reliable results at finite temperature we need to check
whether the same methods used at zero temperature to correct for fixed
topology effects work even in this case. So, we started with some exploratory
studies using quenched theory but fixing topology in order to compare with
previous results in the literature. We measured the topological susceptibility
and several meson correlators at finite temperature. We will report the
results of these simulations and the preliminary results in full QCD to
investigate $U(1)_{A}$ restoration.

##  2 Simulations and results

Before discussing the results, let us briefly describe the methods used to
measure correlators and the topological susceptibility at fixed topology.
Detailed derivation of equations can be found in [13].

By using a saddle point expansion of the QCD partition function in a finite
volume we can derive an expression for $Z_{Q}$, partition function at fixed
topology ($V$ is the 4-volume):

$Z_{Q}=\frac{1}{\sqrt{2\pi\chi_{t}V}}\exp\Bigl{[}-\frac{Q^{2}}{2\chi_{t}V}\Bigr{]}\Bigl{[}1-\frac{c_{4}}{8V\chi_{t}^{2}}+O\Bigl{(}\frac{1}{V^{2}}\Bigr{)}\Bigr{]},$
(2)

a gaussian distribution for topological charge that can be used to show that:

$\lim_{|x|\rightarrow\infty}\langle\rho(x)\rho(0)\rangle=\frac{1}{V}\Bigl{(}\frac{Q^{2}}{V}-\chi_{t}-\frac{c_{4}}{2\chi_{t}V}\Bigr{)}+O(V^{-3}).$
(3)

which implies that the topological susceptibility can be extracted from a long
range correlation of the topological charge density $\rho(x)$. At first order
in $1/V$ we can ignore the contribution of the $c_{4}$ term, and check later
the consistency of the assumption.

Using the overlap operator we can define an object that has the same
properties as the topological charge density:

$\rho_{m}(x)=m\,{\rm tr}[\gamma_{5}(D_{c}+m)_{x,x}^{-1}],$ (4)

where $(D_{c}+m)^{-1}$ is the valence quark propagator constructed using the
chirally symmetric overlap operator. An alternative way is to consider the
pseudoscalar isosinglet $\eta^{\prime}$ correlator, whose disconnected part is
equal to $\langle\rho_{m}(x)\rho_{m}(0)\rangle$ at large distances and couples
only to the fast decaying $\eta^{\prime}$ state, making it a better choice in
order to estimate the long distance limit.

We measured the connected and disconnected part of pseudoscalar correlators at
long distance to extract $\chi_{t}$ [8]. Since we are working at finite
temperature the correlators are measured and averaged over spatial directions.
We reconstructed the correlators by using the first 50 eigenvectors of the
overlap operator assuming low mode dominance. For example, the connected
scalar correlator is given by

$C(x,y)_{N}={\rm
Tr}\sum_{ij}^{N}\frac{\psi_{\lambda_{i}}(x)\psi^{\dagger}_{\lambda_{i}}(y)}{i\lambda_{i}+m}\frac{\psi_{\lambda_{j}}(y)\psi^{\dagger}_{\lambda_{j}}(x)}{i\lambda_{j}+m}$
(5)

and similar expression of the pseudoscalar (just
$i\lambda_{i}\rightarrow-i\lambda_{i}$). We checked that the saturation with
50 eigenmodes is sufficiently accurate for the infrared behavior.

###  2.1 Pure gauge simulations

By introducing the topology fixing term in pure gauge simulations we can check
if, even at finite temperature, we can reconstruct topological susceptibility
using the method described above by comparing with the literature.

The setup is the following: Iwasaki action + topology fixing term at
temperatures ranging from $[0.8,1.3]T_{c}$ on two different volumes
$16^{3}\times 6$ and $24^{3}\times 6$. The critical point was estimated to be
at $\beta_{c}=2.445$ by inspecting the Polyakov loop.

We first check whether the eigenvalue distribution behaves as expected. The
typical distribution is shown in Figure 1. We do not find any discrepancy with
previous results (e.g. [14]). The presence of a peak for small eigenvalues in
the high temperature side ($T>T_{c}$) was confirmed. In [14] these modes are
associated with the presence of dilute gas of instantons-anti instantons.

Figure 1: Spectral density of the overlap-Dirac operator at finite temperature
on the $24^{3}\times 6$ quenched lattice.

The most interesting result at this stage is the behavior of the topological
susceptibility at finite temperature in comparison with the results of
Gattringer et al. [15], shown in Figure 2. The asymptotic value for the
disconnected correlator (see equation (3) ) was estimated using a joint fit of
the connected and disconnected parts and assuming a double pole form for the
last one in the quenched theory. The decay is dominated by pionic states for
both of them. Then we use (3) to extract $\chi_{t}$ assuming that the $c_{4}$
term is negligible.

Figure 2: Comparison of topological susceptibility results with [15], lattice
$24^{3}\times 6$

In [15] the topological susceptibility was measured using the index theorem by
just counting the number of zero modes of an approximated version of the
overlap operator, which is a clean definition without the ambiguities due to
cooling. By also performing a simulation without the topology fixing term, we
checked their results also with our exact overlap operator, finding no
significant deviations within errors (cross symbol in Figure 2, temperature
$T/T_{c}=1.1$).

Below the transition temperature there is agreement between the two sets of
data, showing that the method works very well at least until $T_{c}$. Above
the critical temperature our results are systematically lower than the
reference ones. We are currently investigating the source of this discrepancy
which could be traced back in the assumptions leading to (2) or, the $c_{4}$
term could be non negligible in this regime. We are trying to estimate this
quantity but still we are getting too large error bars for the four-point
spatial correlators.

Another possibility to check the validity of (3) is to measure it on different
topological sectors. Simulations are on the way at the time of writing.

###  2.2 Full QCD simulations

In full QCD with two flavors of dynamical overlap fermions we concentrated on
the channels $\pi,\delta,\eta^{\prime},\sigma$ given by the correlators of the
operators
$\bar{\psi}\gamma_{5}\vec{\tau}\psi,\bar{\psi}\vec{\tau}\psi,\bar{\psi}\gamma_{5}\psi,\bar{\psi}\psi$,
respectively. Here, $\tau$ is the Pauli matrix in the flavor space.

If chiral symmetry is restored we expect that the pairs $(\sigma,\pi)$,
$(\eta^{\prime},\delta)$ become degenerate. Similarly, if the flavour-singlet
axial symmetry is restored too at high temperatures we would have that all the
channels become degenerate. The $\pi$ and the $\eta^{\prime}$ differ just by
the disconnected part, which is essentially given by the near-zero modes. This
observation implies that the $U(1)_{A}$ breaking is driven by near-zero modes.
By looking at the spatial correlators in those channels, we can check whether
it happens. The problem of establishing if the axial symmetry is restored at
the critical point is still open and has a relevance on the possible order of
the phase transition [4].

Some details on the simulation follow. The algorithm is HMC, using Iwasaki
action with topology fixing term and two flavors of sea quarks. The size of
the lattice is $16^{3}\times 8$. We choose $N_{t}=8$ to ensure that the
configurations are smooth enough. Masses start from $am=0.05$ down to
$am=0.01$ giving a pion mass of around 400 MeV at lowest temperature. The
$\beta$s were chosen to be in the temperature range $T=[171,243]$ MeV,
$T/T_{c}=[0.97,1.39]$ (assuming the critical temperature to be $T_{c}=175$
MeV). A comment is in order here: we couldn’t estimate directly the transition
temperature because it requires long history runs to measure susceptibilities.
Anyway we checked that, by looking at eigenvalues of the Dirac operator
(Figure 3), and the Polyakov loop, we simulated $\beta$s just below and above
the transition temperature. The topological sector is mainly $Q=0$ but we have
some simulations also at $Q=2$. All configurations were generated and analyzed
using the BlueGene/L installation at KEK.

Figure 3: Eigenvalues density in full QCD simulations, lattice $16^{3}\times
8$. Density around zero gives the chiral condensate by Banks-Casher relation.

Some preliminary results are shown in Figure 4. At this stage we cannot say
anything quantitative since the volume seems still too small to extract
masses. We can just check by inspection of the plots when the correlators
become equal. We observe that the correlators start to become almost
degenerate after the transition temperature and that as the sea quark mass is
decreased this degeneracy is improved. We are collecting more data points to
extrapolate this result toward the transition temperature.

Figure 4: Scalar and pseudoscalar spatial correlators at finite temperature.
The estimate for the $\sigma$, green area, correlator has still huge errors in
comparison to the others. We see some degeneracy for the correlators at
$\beta=2.30,T=208$ MeV, right panel.

##  3 Conclusions

We described our project on finite temperature QCD with overlap fermions.
Simulating overlap fermions forces to fix the topology in our case. A method
to extract physical results from fixed $Q$ simulations was previosly developed
for the zero temperature regime. To obtain reliable results we must check that
we can apply the same method at finite temperature or find the necessary
modifications. We started the investigation analyzing the behavior of
topological susceptibility in pure gauge theory at $Q=0$. We found that our
results differ at high temperature from the previous works. We are currently
investigating the source of this discrepancy. A deep understanding of this
problem is essential in the interpretation of full QCD results where we found
restoration of axial symmetry at least from temperatures above $1.1T_{c}$.

## Acknowledgements

Numerical simulations are performed on Hitachi SR11000 and IBM System Blue
Gene Solution at High Energy Accelerator Research Organization (KEK) under a
support of its Large Scale Simulation Program (No. 09/10-09). This work is
supported in part by the Grant-in-Aid of the Ministry of Education, Culture,
Sports, Science and Technology (No. 21674002, No. 20105005, No. 21684013, No.
220340047, No. 21105508) and by Grant-in-Aid for Scientific Research on
Innovative Areas (No. 20105001, No. 20105003).

## References

  * [1] G. ’t Hooft, Phys. Rev. Lett. 37, 8-11 (1976).
  * [2] E. Witten, Nucl. Phys. B156, 269 (1979).
  * [3] G. Veneziano, Nucl. Phys. B159, 213-224 (1979).
  * [4] A. Butti, A. Pelissetto, E. Vicari, JHEP 0308, 029 (2003). [hep-ph/0307036].
  * [5] P. M. Vranas, Nucl. Phys. Proc. Suppl. 83, 414-416 (2000). [hep-lat/9911002].
  * [6] M. Luscher, Phys. Lett. B428, 342-345 (1998).
  * [7] T. Kaneko et al. [ JLQCD Collaboration ], PoS LAT2006, 054 (2006).
  * [8] S. Aoki et al. [JLQCD and TWQCD Collaborations], Phys. Lett. B 665, 294 (2008) [arXiv:0710.1130 [hep-lat]].
  * [9] J. Noaki et al. [JLQCD and TWQCD Collaborations], Phys. Rev. Lett. 101, 202004 (2008) [arXiv:0806.0894 [hep-lat]].
  * [10] S. Aoki et al. [JLQCD Collaboration and TWQCD Collaboration], Phys. Rev. D 80, 034508 (2009) [arXiv:0905.2465 [hep-lat]].
  * [11] H. Fukaya, S. Aoki, S. Hashimoto, T. Kaneko, J. Noaki, T. Onogi and N. Yamada [JLQCD collaboration], Phys. Rev. Lett. 104, 122002 (2010) [Erratum-ibid. 105, 159901 (2010)] [arXiv:0911.5555 [hep-lat]].
  * [12] H. Fukaya et al. [ JLQCD Collaboration ], Phys. Rev. D74, 094505 (2006). [hep-lat/0607020].
  * [13] S. Aoki, H. Fukaya, S. Hashimoto, T. Onogi, Phys. Rev. D76, 054508 (2007).
  * [14] R. G. Edwards, U. M. Heller, J. E. Kiskis and R. Narayanan, Phys. Rev. D 61, 074504 (2000)
  * [15] C. Gattringer, R. Hoffmann and S. Schaefer, Phys. Lett. B 535, 358 (2002)