# Temporal Gaussian Process Regression in Logarithmic Time

Adrien Corenflos, , Zheng Zhao, , and Simo Särkkä This work was supported
Academy of Finland (projects 321900 and 321891) and Aalto ELEC doctoral
school.Adrien Corenflos (adrien.corenflos@aalto.fi), Zheng Zhao, and Simo
Särkkä were with Department of Electrical Engineering and Automation, Aalto
University, Finland.

###### Abstract

The aim of this article is to present a novel parallelization method for
temporal Gaussian process (GP) regression problems. The method allows for
solving GP regression problems in logarithmic $O(\log N)$ time, where $N$ is
the number of time steps. Our approach uses the state-space representation of
GPs which in its original form allows for linear $O(N)$ time GP regression by
leveraging the Kalman filtering and smoothing methods. By using a recently
proposed parallelization method for Bayesian filters and smoothers, we are
able to reduce the linear computational complexity of the temporal GP
regression problems into logarithmic span complexity. This ensures logarithmic
time complexity when run on parallel hardware such as a graphics processing
unit (GPU). We experimentally demonstrate the computational benefits on
simulated and real datasets via our open-source implementation leveraging the
GPflow framework.

###### Index Terms:

Gaussian process, state-space, parallelization, logarithmic time, Kalman
filter and smoother

##  I Introduction

Gaussian processes (GPs) are a family of useful function-space priors used to
solve regression and classification problems arising in machine learning [1].
In their naive form their complexity scales as $O(N^{3})$, where $N$ is the
number of training data points, which is problematic for large datasets. One
approach to alleviating this is to use state-space methods [2, 3] which reduce
the GP regression problem into a Kalman filtering and smoothing problem with
linear time complexity $O(N)$ in the number of data points. This linear
complexity is optimal on single-threaded computational architectures, as
processing data needs to be done sequentially. However it is suboptimal on
hardware where parallelization is possible, such as multi-core central
processing units (CPUs) or, more importantly, on massively threaded
architectures such as graphics processing units (GPU). The aim of this letter
is therefore to develop parallel state-space GPs (PSSGPs) methods which reduce
the computational complexity (in the sense of parallel span complexity) of
state-space GPs to logarithmic $O(\log N)$ (see Fig. 1). To do so we leverage
the parallel Bayesian filtering and smoothing methodology of [4].

Over the recent years, several other approaches to parallelization of GPs have
been proposed. For instance in [5, 6] the authors consider mini-batching the
dataset to form mixtures of local GP experts, incurring a cubic cost in the
size of the batches. More closely related to this letter are the works in [7,
8] which proposed to leverage the sparse Markovian structure of Markovian and
state-space GPs (SSGPs) via the use of parallel matrix computations, thereby
reaching $O(\log N)$ span complexity in the dataset size in some special
cases. However, the methods outlined in [7, 8] effectively require
computations with large (albeit sparse) matrices and the logarithmic span
complexity is hard to guarantee for all subproblems [7]. Orthogonally to these
parallelization efforts, different approximation methods were introduced in
order to reduce the computational complexity of GPs. These include for example
inducing points, spectral sampling, or basis function methods (see e.g., [1,
9, 10, 11, 12]).

Figure 1: Average run time in seconds of predicting $M=10\,000$ test points
for noisy sinusoidal data with RBF covariance function for standard GP, SSGP,
and PSSGP (ours). $x$-axis is the number of training data points $N$. We can
see that our method outperforms GP and SSGP by a factor $>10$.

The contribution of our paper is three-fold:

  1. 1.

We combine the state-space formulation of GPs with parallel Kalman filters and
smoothers [4].

  2. 2.

We extend the parallel formulation to missing measurements needed in
prediction with state-space GPs.

  3. 3.

We experimentally demonstrate the computational gains of our proposed methods
on simulated and real datasets111We implemented the method as an open-source
extensible library. The code can be found at https://github.com/EEA-
sensors/parallel-gps..

##  II Gaussian processes in state-space form

In this section, we quickly recall results about the state-space formulation
of Gaussian processes before presenting their temporal parallelization
formulation. Given covariance function $C(t,t^{\prime})$ and observations
$\\{y_{k}\colon k=1,\ldots,N\\}$, a temporal GP regression problem of the form

$\begin{split}f(t)&\sim\mathrm{GP}(0,C(t,t^{\prime})),\\\
y_{k}&=f(t_{k})+e_{k},\qquad
e_{k}\sim\mathcal{N}(0,\sigma_{k}^{2}),\end{split}$ (1)

can be converted into a smoothing problem for a $n_{x}$-dimensional
continuous-discrete state-space model

$\begin{split}\frac{\mathop{}\\!\mathrm{d}\mathbf{x}(t)}{\mathop{}\\!\mathrm{d}t}=\mathbf{G}\,\mathbf{x}(t)+\mathbf{L}\,\mathbf{w}(t),\quad
y_{k}=\mathbf{H}\,\mathbf{x}(t_{k})+e_{k},\end{split}$ (2)

where $\mathbf{x}(t)$ is the state, $y_{k}$ is the measurement,
$\mathbf{w}(t)$ is a white noise process with a spectral density $\mathbf{Q}$,
and $e_{k}$ is the Gaussian measurement noise [2, 3]. The dimensionality
$n_{x}$ of the state as well as the matrices $\mathbf{G}$, $\mathbf{L}$,
$\mathbf{H}$, and $\mathbf{Q}$ in the model depend on (and define) the
covariance function at hand.

In the state-space formulation (2), the Gaussian process in (1) has the
representation $f(t)=\mathbf{H}\,\mathbf{x}(t)$. In the case of Matérn
covariance functions this representation happens to be exact and available in
closed form [2], while more general stationary covariance functions can be
approximated up to an arbitrary precision by using Taylor series or Páde
approximants [2, 13, 14, 15, 16, 17] in spectral domain.

The continuous-time state-space model (2) can be discretized into an equvalent
discrete-time linear Gaussian state space model (LGSSM, e.g., [18]) of the
form

$\mathbf{x}_{k}=\mathbf{F}_{k-1}\,\mathbf{x}_{k-1}+\mathbf{q}_{k-1}\quad
y_{k}=\mathbf{H}\,\mathbf{x}_{k}+e_{k},$ (3)

where $\mathbf{q}_{k-1}\sim\mathcal{N}(\mathbf{0},\mathbf{Q}_{k-1})$. The GP
regression problem can now be solved by applying Kalman filter and smoother
algorithms on model (3) in $O(N)$ time [2]. Furthermore, parameters of this
model, such as the hyperparameters $\bm{\theta}$ of the covariance function
$C(t,t^{\prime};\bm{\theta})$ can be estimated with standard state-space
methods (see, [19]).

The aim of this paper is to point out that these sequential Kalman filters and
smoothers as well as the parameter estimation methods can be replaced by their
parallel versions [4] which reduces the computational complexity to $O(\log
N)$. Additionally, parameter estimation using methods such as gradient-based
optimization [20] or Hamiltonian Monte Carlo (HMC) [21] requires computation
of gradients of the log-likelihood function. However, the same libraries that
allow for efficient parallel implementation of the Kalman filters and
smoothers such as TensorFlow [22] also support automatic differentiation which
can be used to compute the gradients without manual effort and within the
parallel computations.

##  III Handling Missing Observations in Parallel Kalman Filter

The formulation in terms of a continuous state space model presented in the
previous section reduces the computational complexity of inference in
compatible GPs to $O(N)$ instead of the cubic cost induced by naive GP
methods, or to a lesser extent by their sparse approximations [1]. In [4], the
authors introduce an equivalent formulation of Kalman filters and smoothers in
terms of an associative operator which enables them to leverage distributed
implementations of scan (prefix-sum) algorithms such as [23] and [24],
reducing the time complexity of Kalman filtering and smoothing down to $O(\log
N)$. However, this formulation does not consider the possibility of missing
observations. This prevents its application for inference in state-space GP
models where test points are treated as missing data [3].

The method in [4] consists in writing the filtering step in terms of an
associative operator of a sequence of five elements
$(\mathbf{A}_{k},\mathbf{b}_{k},\mathbf{C}_{k},\bm{\eta}_{k},\mathbf{J}_{k})$,
which are first initialized in parallel and then combined using parallel
associative scan [23, 24]. At initialization of the original algorithm, these
elements need to be computed so as to correspond to the following quantities:

$\displaystyle p(\mathbf{x}_{k}\mid\mathbf{x}_{k-1},y_{k})$
$\displaystyle=\mathcal{N}(\mathbf{x}_{t}\mid\mathbf{A}_{k}\mathbf{x}_{k-1}+\mathbf{b}_{k},\mathbf{C}_{k}),$
(4) $\displaystyle p(y_{k}\mid\mathbf{x}_{k-1})$
$\displaystyle=\mathcal{N}_{I}(\mathbf{x}_{k-1}\mid\bm{\eta}_{k},\mathbf{J}_{k}),$
(5)

where $\mathcal{N}_{I}$ corresponds to the information form of the Gaussian
distribution. However, when no observation is available at step $k$, these
equations do not hold directly and need to be modified.

By redoing the original derivation it turns out that in the case of missing
measurements, the posterior $p(\mathbf{x}_{k}\mid\mathbf{x}_{k-1},y_{k})$
should be replaced by the transition density
$p(\mathbf{x}_{k}\mid\mathbf{x}_{k-1})=\mathcal{N}(\mathbf{x}_{k}\mid\mathbf{F}_{k-1}\mathbf{x}_{k-1},\mathbf{Q}_{k-1})$
for $k>1$ and $p(\mathbf{x}_{1})$ for $k=1$. That is, the initialization
equations for
$\mathbf{A}_{k},\mathbf{b}_{k},\mathbf{C}_{k},\bm{\eta}_{k},\mathbf{J}_{k}$ in
the case of missing measurements can be written as

$\mathbf{A}_{k}=\mathbf{F}_{k-1},\quad\mathbf{b}_{k}=\mathbf{0},\quad\mathbf{C}_{k}=\mathbf{Q}_{k-1},$
(6)

for $k>1,$ and for $k=1$ as

$\mathbf{A}_{1}=\mathbf{0},\quad\mathbf{b}_{1}=\mathbf{0},\quad\mathbf{C}_{1}=\mathbf{P}_{\infty},$
(7)

with for all $k,$

$\bm{\eta}_{k}=\mathbf{0},\quad\mathbf{J}_{k}=\mathbf{0}.$ (8)

When the quantities
$\mathbf{A}_{k},\mathbf{b}_{k},\mathbf{C}_{k},\bm{\eta}_{k},\mathbf{J}_{k}$
have been initialized for time steps with and without observations, they can
then be combined using parallel scan, with the associative operator defined in
the same way as in [4]:

$\displaystyle\mathbf{A}_{ij}$
$\displaystyle=\mathbf{A}_{j}\left(\mathbf{I}_{n_{x}}+\mathbf{C}_{i}\mathbf{J}_{j}\right)^{-1}\mathbf{A}_{i},$
$\displaystyle\mathbf{b}_{ij}$
$\displaystyle=\mathbf{A}_{j}\left(\mathbf{I}_{n_{x}}+\mathbf{C}_{i}\mathbf{J}_{j}\right)^{-1}\left(\mathbf{b}_{i}+\mathbf{C}_{i}\bm{\eta}_{j}\right)+\mathbf{b}_{j},$
$\displaystyle\mathbf{C}_{ij}$
$\displaystyle=\mathbf{A}_{j}\left(\mathbf{I}_{n_{x}}+\mathbf{C}_{i}\mathbf{J}_{j}\right)^{-1}\mathbf{C}_{i}\mathbf{A}_{j}^{\top}+\mathbf{C}_{j},$
$\displaystyle\bm{\eta}_{ij}$
$\displaystyle=\mathbf{A}_{i}^{\top}\left(\mathbf{I}_{n_{x}}+\mathbf{J}_{j}\mathbf{C}_{i}\right)^{-1}\left(\bm{\eta}_{j}-\mathbf{J}_{j}\mathbf{b}_{i}\right)+\bm{\eta}_{i},$
$\displaystyle\mathbf{J}_{ij}$
$\displaystyle=\mathbf{A}_{i}^{\top}\left(\mathbf{I}_{n_{x}}+\mathbf{J}_{j}\mathbf{C}_{i}\right)^{-1}\mathbf{J}_{j}\mathbf{A}_{i}+\mathbf{J}_{i}.$

Running a parallel scan algorithm on the elements above with the filtering
operator $\otimes$ then produces a sequence of “prefix-sum” elements
$(\mathbf{A}^{*}_{k},\mathbf{b}^{*}_{k},\mathbf{C}^{*}_{k},\bm{\eta}^{*}_{k},\mathbf{J}^{*}_{k}),$
$k=1,\dots,N,$ of which the terms
$\overline{\mathbf{x}}_{k}\triangleq\mathbf{b}^{*}_{k}$ and
$\mathbf{P}_{k}\triangleq\mathbf{C}^{*}_{k}$ will then correspond to the
filtering mean and covariance at time step $k$, respectively.

######  Proposition 1 (Equivalence of sequential and parallel Kalman filters).

For all $k=1,2,\ldots,N+M$ the Kalman filter means and covariances are given
by $\overline{\mathbf{x}}_{k}=\mathbf{b}^{*}_{k}$ and
$\mathbf{P}_{k}=\mathbf{C}^{*}_{k}$, respectively.

###### Proof.

The detailed proof is omitted due to space reasons, but the result follows by
explicitly writing down the forward recursion for the elements
$(\mathbf{A}^{*}_{k},\mathbf{b}^{*}_{k},\mathbf{C}^{*}_{k},\bm{\eta}^{*}_{k},\mathbf{J}^{*}_{k}),$
and checking that equations for $\mathbf{b}^{*}_{k}$ and
$\mathbf{P}_{k}=\mathbf{C}^{*}_{k}$ coincide with the Kalman filter equations.
∎

On the other hand, the parallel smoothing algorithm needs no modifications
compared to [4] in order to handle missing observations, as it only relies on
the result of the filtering algorithm and not directly on the observations.

##  IV Temporal parallelization of Gaussian Processes

A direct consequence of equivalence of the parallel and sequential Kalman
filters and smoothers [4] is the fact that the parallel and sequential
versions of SSGP are equivalent too. In this section, we provide the details
of the steps needed to create the linear Gaussian state-space model (LGSSM)
representation of state-space GPs so that the end-to-end algorithm is
parallelizable across the time dimension, and automatically differentiable,
resulting a total span complexity of $O(\log N)$ in the number of observations
$N$, from training to inference.

###  IV-A Computation of the steady-state covariance

To represent a stationary GP one must start the state-space model SDE from the
stationary initial condition given by the Lyapunov equation [2]. The
complexity of this step is independent of the number of time steps and does
not need explicit time-parallelization. There exist a number of iterative
methods for solving this kind of algebraic equations [25]. However, in order
to make automatic differentiation efficient, in this work, we used the closed
form vectorization solution given in [26] (p. 229) which relies on matrix
algebra, and does not need any explicit looping. This is feasible because we
only need to numerically solve the Lyapunov equation for small state
dimensions. Furthermore, as this solution only involves matrix inversions and
multiplications, it is easily parallelizable on GPU architectures.

###  IV-B Balancing of the state space model

In practice the state-space model obtained via discretization (3) is often
numerically unstable due to the transition matrix having a poor conditioning
number, this would in turn result in inaccuracies in computing both the GP
predictions and the marginal log-likelihood of the observations. To mitigate
this issue we need to resort to balancing [27] in order to obtain a transition
matrix $\mathbf{F}$ which has rows and columns that have equal norms,
therefore obtaining a more stable surrogate state-space model. For any
diagonal matrix $\mathbf{D}\in\mathbb{R}^{n_{x}\times n_{x}},$ the continuous-
discrete model

$\begin{split}\frac{\mathop{}\\!\mathrm{d}\mathbf{z}(t)}{\mathop{}\\!\mathrm{d}t}&=\mathbf{D}^{-1}\,\mathbf{G}\,\mathbf{D}\,\mathbf{z}(t)+\mathbf{D}^{-1}\,\mathbf{L}\,\mathbf{w}(t),\\\
y_{k}&=\mathbf{H}\,\mathbf{D}\,\mathbf{z}_{k}+e_{k},\end{split}$ (9)

with initial condition given by
$\mathbf{z}(t_{0})\sim\mathcal{N}(\mathbf{0},\mathbf{D}^{-1}\,\mathbf{P}_{\infty}\mathbf{D})$
is an equivalent representation of the state-space model (2) with initial
covariance given by
$\mathbf{x}(t_{0})\sim\mathcal{N}(\mathbf{0},\mathbf{P}_{\infty})$ with
$\mathbf{P}_{\infty}$, in the sense that for all $t\geq t_{0}$ we have
$f(t)=\mathbf{H}\,\mathbf{D}\,\mathbf{z}(t)$.

In particular this means that the gradient of the log-likelihood $\log
p(y_{1},\ldots,y_{N}\mid\bm{\theta})$ with respect to the parameters
$\bm{\theta}$ is left unchanged by the choice of scaling matrix $\mathbf{D}$.
This property allows us to condition our state space representation of the
original GP using the scaling matrix $\mathbf{D}^{*}$ defined in [27], and to
compute the gradient of the log-likelihood of the observations with respect to
the GP hyperparameters as if $\mathbf{D}^{*}$ did not depend on $\mathbf{F}$
and therefore the hyperparameters $\bm{\theta}.$ This is crucial to obtain a
stable gradient while not having to unroll the gradient through the loop
necessary to compute $\mathbf{D}^{*}.$

###  IV-C Converting GPs into discrete-time state space

In order to use the parallel formulation of Kalman filters and smoothers in
Section III we need to first form the continuous state space model
representation from the initial Gaussian process definition. This operation is
independent of the number of measurements and therefore has a complexity of
$O(1)$. When it has been formed, we then need to transform it into a discrete-
time LGSSM (3). In practice, the discretization can be implemented using, for
example, matrix fractions or various other methods [28, 18]. These operations
are fully parallelizable across the time dimension and therefore have a span
complexity of $O(1)$ when run on a parallel architecture.

It is worth noting that at hyperparameters learning phase, the discretization
needs only to happen for the training data points. However, when predicting,
it is necessary to insert the $M$ requested prediction times at the right
location in the training data so as to be able to run the Kalman filter and
smoother routines. When done naively, this operation has complexity $O(M+N)$
using merging operation [29], but can be done in parallel with span complexity
$O(\log(\min(M,N))).$ In addition, for some GP models, such as Matérn GPs the
discretization can be done offline, as it admits closed-form solutions [18].

###  IV-D End-to-end complexity of parallelized state space GPs

The complexity analysis of the six stages for running the parallellized state
space GPs are the following:

  1. 1.

Complexity of forming the continuous state-space model is $O(1)$ in both work
and span complexities.

  2. 2.

Complexity of discretizing the state-spate model is $O(N)$ is work complexity
and $O(1)$ in span complexity.

  3. 3.

Complexity of the parallel Kalman filtering and smoothing operations for
training is of $O(N)$ in work complexity and $O(\log N)$ is span complexity.

  4. 4.

Complexity of merging the data for the prediction is of work complexity
$O(N+M)$ and of span complexity $O(\log(\min(M,N)))$.

  5. 5.

Complexity of the parallel Kalman filtering and smoothing operations for
prediction is of $O(N+M)$ in work complexity and $O(\log(N+M))$ is span
complexity.

  6. 6.

Automatic differentiation as an adjoint operation has the same computational
graph as the parallel Kalman filters and smoothers, and therefore has the same
work and span complexities as the training step: $O(N)$ and $O(\log N)$,
respectively.

Putting together the above we can conclude that the total work and span
complexities of doing end-to-end inference to prediction on parallelized state
space GPs are $O(N+M)$ and $O(\log(N+M))$, respectively.

##  V Experiments

In this section, we illustrate the benefits of our approach for inference and
prediction on simulated and real datasets. Because GPUs are inherently
massively parallelized architectures, they are not optimized for sequential
data processing, exposing a lower clock speed than cost-comparable CPUs. This
makes running the standard state space GPs on GPU less attractive than running
them on CPU, contrarily to standard GPs which can leverage GPU-enabled linear
algebra routines. In order to offer a fair comparison between our proposed
methodology, the standard GPs implementation, and the standard state space
GPs, we have therefore chosen to run the sequential implementation of state
space GP on CPU while we run the standard GP and our proposed parallel state
space GP on GPU. We verified empirically that running the standard state space
GP on the same GPU architecture resulted in a tremendous performance loss for
it ($\sim 100$x slower), justifying running it on CPU for benchmarking. All
the results were obtained using an AMD® Ryzen threadripper 3960x CPU with 128
GB DDR4 RAM, and an Nvidia® GeForce RTX 3090 GPU with 24GB memory.

###  V-A Simulation model

We first study the behavior of our proposed methodology on a simple noisy
sinusoidal model given by

$\begin{split}f(t)&=\sin(\pi\,t)+\sin(2\,\pi\,t)+\sin(3\,\pi\,t)\\\
y_{k}&=f(t_{k})+r_{k},\end{split}$ (10)

with observations and prediction times being equally spaced on $(0,4)$. By
increasing the number of training or testing points we can therefore
demonstrate the empirical time complexity (in terms of wall clock or work
span) of our proposed method compared to the standard GP and state space GP.

We have taken the kernels to be successively Matérn32, Matérn52, and RBF
(approximated to the 6th order for the state space GPs), corresponding to
$n_{x}=2,3,6$ respectively. The training dataset and test dataset have sizes
ranging from $2^{12}$ to $2^{15}$. As can be seen in Figure 1, our proposed
method consistently outperforms standard GP and SSGP across the chosen range
of dataset sizes with standard GP eventually running out of memory for larger
datasets.

###  V-B Sunspots dataset

In this section we compare regression and parameter learning via likelihood
maximization using L-BFGS [30, 20] on the monthly sunspot activity dataset by
World Data Center SILSO, Royal Observatory of Belgium, Brussels222The data is
available at http://www.sidc.be/silso/datafiles. The number of total training
points in this dataset is $N=3,200$ for learning the GP hyperparameters. We
interpolate the data for every single day from 1749-01-31 to 2018-07-31 which
results in 96,000 prediction points.

Table I: Running time of hyperparameter learning on sunspot dataset relative to PSSGP. The PSSGP took $39$, $46$, and $48$ milliseconds per function/gradient evaluation when $N=1200,2200$, and $3200$, respectively. N | GP | SSGP | PSSGP
---|---|---|---
1200 | 1.03 | 12.08 | $\mathbf{1}$
2200 | 3.82 | 25.7 | $\mathbf{1}$
3200 | 10.1 | 43.86 | $\mathbf{1}$

The results of running time is shown in the above Table I. PSSGP is shown to
be the fastest for all $N$. Moreover, PSSGP is x10 and 43x faster than GP and
SSGP when $N=3200$. Interpolating daily then took 0.14s for PSSGP, while SSGP
on CPU and GP on GPU where respectively 23 and 33 times slower.

###  V-C CO2 concentration dataset

In order to understand the impact of dimensionality of the SDE we finally
consider the Mauna Loa carbon dioxide concentration dataset333The data is
available at https://www.esrl.noaa.gov/gmd/ccgg/trends/ with HMC sampling [21]
on the parameters. We selected monthly and weekly data from year 1974 to 2021
which contains 3192 training points. To model the periodic pattern of the
data, we use the following composite covariance function
$C_{\mathrm{co2}}(t-t^{\prime})=C_{\mathrm{Per.}}(t-t^{\prime})\,C_{\mathrm{Mat.}}(t-t^{\prime})+C_{\mathrm{Mat.}}(t-t^{\prime})$
whose periodic component was converted to state space form using different
orders of its Taylor series [14], resulting in SDEs of dimensions of
$n_{x}=10$, $14$, and $18$. This model is slightly different from the one
suggested in [14] where the authors also add a RBF covariance to
$C_{\mathrm{co2}}$, however, we did not see any improvement in RMSE from
adding this supplementary degree of freedom and therefore left it out.

Table II: Relative time of sampling from the parameters posterior distribution using HMC with 10 leapfrog steps on CO2 dataset. The GP took $3$ s per sample. Order | GP | SSGP | PSSGP
---|---|---|---
1 | 1 | 4.5 | $\mathbf{0.55}$
2 | $\mathbf{1}$ | 5.73 | 1.36
3 | $\mathbf{1}$ | 6.9 | 2.55

As we can see in Table II, while PSSGP is still competitive compared to SSGP
for high dimensional SDE representations, its complexity increases with the
dimension to the point where it eventually does not outperform the standard GP
anymore.

##  VI Discussion

We have presented a sublinear algorithm for learning and inference in state
space Gaussian processes, leveraging and extending the parallel Kalman filter
and smoother introduced in [4]. This allowed us to reduce dramatically the
training time for regression problems on large datasets as illustrated by our
experiments on synthetic and real data. However, the final experiment also
revealed a limitation of the methodology which is that the method does not
scale well with high-dimensional state-space representations and thus more
work is needed to further reduce the complexity of parallel state-space GPs.
Recent works [31] have suggested that this kind of parallelization techniques
could also be used for non-linear state-space models, which would then make it
possible to extend the present methodology for classification and deep state-
space Gaussian processes [32].

## References

  * [1] C. E. Rasmussen and C. K. Williams, _Gaussian Processes for Machine Learning_. MIT Press, 2006.
  * [2] J. Hartikainen and S. Särkkä, “Kalman filtering and smoothing solutions to temporal Gaussian process regression models,” in _Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2010, pp. 379–384.
  * [3] S. Särkkä, A. Solin, and J. Hartikainen, “Spatiotemporal learning via infinite-dimensional Bayesian filtering and smoothing,” _IEEE Signal Processing Magazine_ , vol. 30, no. 4, pp. 51–61, 2013.
  * [4] S. Särkkä and Á. F. García-Fernández, “Temporal parallelization of Bayesian smoothers,” _IEEE Transactions on Automatic Control_ , vol. 66, no. 1, pp. 299–306, 2021.
  * [5] H. Liu, J. Cai, Y. Wang, and Y. S. Ong, “Generalized robust Bayesian committee machine for large-scale Gaussian process regression,” in _International Conference on Machine Learning_ , 2018, pp. 3131–3140.
  * [6] M. M. Zhang and S. A. Williamson, “Embarrassingly parallel inference for Gaussian processes,” _Journal of Machine Learning Research_ , vol. 20, no. 169, pp. 1–26, 2019.
  * [7] A. Grigorievskiy, N. Lawrence, and S. Särkkä, “Parallelizable sparse inverse formulation Gaussian processes (SpInGP),” in _IEEE International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2017, pp. 1–6.
  * [8] F. Lindgren, H. Rue, and J. Lindström, “An explicit link between Gaussian fields and Gaussian Markov random fields: The stochastic partial differential equation approach,” _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , vol. 73, no. 4, pp. 423–498, 2011.
  * [9] J. Quiñonero-Candela and C. E. Rasmussen, “A unifying view of sparse approximate Gaussian process regression,” _Journal of Machine Learning Research_ , vol. 6, pp. 1939–1959, 2005.
  * [10] M. Lázaro-Gredilla, J. Quiñonero-Candela, C. E. Rasmussen, and A. R. Figueiras-Vidal, “Sparse spectrum Gaussian process regression,” _Journal of Machine Learning Research_ , vol. 11, pp. 1865–1881, 2010.
  * [11] J. Hensman, N. Durrande, and A. Solin, “Variational Fourier features for Gaussian processes,” _Journal of Machine Learning Research_ , vol. 8, no. 151, pp. 1–52, 2017.
  * [12] A. Solin and S. Särkkä, “Hilbert space methods for reduced-rank Gaussian process regression,” _Statistics and Computing_ , vol. 30, no. 2, pp. 419–446, 2020.
  * [13] S. Särkkä and R. Piché, “On convergence and accuracy of state-space approximations of squared exponential covariance functions,” in _Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2014, pp. 1–6.
  * [14] A. Solin and S. Särkkä, “Explicit link between periodic covariance functions and state space models,” in _Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics (AISTATS)_ , vol. 33, 2014, pp. 904–912.
  * [15] ——, “Gaussian quadratures for state space approximation of scale mixtures of squared exponential covariance functions,” in _Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2014, pp. 1–6.
  * [16] T. Karvonen and S. Särkkä, “Approximate state-space Gaussian processes via spectral transformation,” in _Proceedings of IEEE International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2016.
  * [17] F. Tronarp, T. Karvonen, and S. Särkkä, “Mixture representation of the Matérn class with applications in state space approximations and Bayesian quadrature,” in _Proceedings of IEEE International Workshop on Machine Learning for Signal Processing (MLSP)_ , 2018.
  * [18] S. Särkkä and A. Solin, _Applied Stochastic Differential Equations_. Cambridge University Press, 2019.
  * [19] S. Särkkä, _Bayesian Filtering and Smoothing_. Cambridge University Press, 2013.
  * [20] J. Nocedal and S. J. Wright, _Numerical Optimization_. Springer, 1999.
  * [21] R. M. Neal, “MCMC using Hamiltonian dynamics,” in _Handbook of Markov Chain Monte Carlo_ , S. Brooks, A. Gelman, G. L. Jones, and X.-L. Meng, Eds. Chapman & Hall/CRC, 2011, ch. 5.
  * [22] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org.
  * [23] G. E. Blelloch, “Scans as primitive parallel operations,” _IEEE Transactions on Computers_ , vol. 38, no. 11, pp. 1526–1538, 1989.
  * [24] ——, “Prefix sums and their applications,” School of Computer Science, Carnegie Mellon University, Tech. Rep. CMU-CS-90-190, 1990.
  * [25] A. S. Hodel, B. Tenison, and K. R. Poolla, “Numerical solution of the Lyapunov equation by approximate power iteration,” _Linear Algebra and its Applications_ , vol. 236, pp. 205–230, 1996.
  * [26] W. L. Brogan, _Modern Control Theory_ , 3rd ed. Pearson, 2011.
  * [27] E. E. Osborne, “On pre-conditioning of matrices,” _Journal of the ACM_ , vol. 7, no. 4, pp. 338–345, 1960.
  * [28] P. Axelsson and F. Gustafsson, “Discrete-time solutions to the continuous-time differential Lyapunov equation with applications to Kalman filtering,” _IEEE Transactions on Automatic Control_ , vol. 60, no. 3, pp. 632–643, 2014\.
  * [29] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, _Introduction to Algorithms_. MIT press, 2009.
  * [30] C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal, “Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization,” _ACM Transactions on Mathematical Software_ , vol. 23, no. 4, p. 550–560, 1997.
  * [31] F. Yaghoobi, A. Corenflos, S. Hassan, and S. Särkkä, “Parallel iterated extended and sigma-point Kalman smoothers,” _To appear in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021. [Online]. Available: https://arxiv.org/abs/2102.00514
  * [32] Z. Zhao, M. Emzir, and S. Särkkä, “Deep state-space gaussian processes,” _arXiv preprint arXiv:2008.04733_ , 2020.


#  Socratis: Are large multimodal models emotionally aware?

1*Katherine Deng, 1*Arijit Ray, 1*Reuben Tan, 3 Saadia Gabriel, 1Bryan A.
Plummer, 1,2Kate Saenko
1Boston University, 2Meta AI (FAIR), 3MIT
{kdeng, array, rxtan, bplum<EMAIL_ADDRESS><EMAIL_ADDRESS>
∗Equal Contribution

###### Abstract

Existing emotion prediction benchmarks contain coarse emotion labels which do
not consider the diversity of emotions that an image and text can elicit in
humans due to various reasons. Learning diverse reactions to multimodal
content is important as intelligent machines take a central role in generating
and delivering content to society. To address this gap, we propose Socratis, a
societal reactions benchmark, where each image-caption (IC) pair is annotated
with multiple emotions and the reasons for feeling them. Socratis contains 18K
free-form reactions for 980 emotions on 2075 image-caption pairs from 5
widely-read news and image-caption (IC) datasets. We benchmark the capability
of state-of-the-art multimodal large language models to generate the reasons
for feeling an emotion given an IC pair. Based on a preliminary human study,
we observe that humans prefer human-written reasons over 2 times more often
than machine-generated ones. This shows our task is harder than standard
generation tasks because it starkly contrasts recent findings where humans
cannot tell apart machine vs human-written news articles, for instance. We
further see that current captioning metrics based on large vision-language
models also fail to correlate with human preferences. We hope that these
findings and our benchmark will inspire further research on training
emotionally aware models. Our dataset can be found at
https://kdeng55.github.io/socratis-website/.

##  1 Introduction

A crucial prerequisite for effective communication and collaboration is the
ability to possess emotional awareness [7]. As intelligent machines become
increasingly prevalent ranging from generative content creation [19] to
collaborative [13] and embodied AI [15], they need to possess emotional
awareness for effective communication, and greater trust and acceptance [7].
Emotionally unaware messaging undermine efforts to inform people of global
crises [4], spread political division [23], and fail to engage people for the
correct social causes [5]. Existing work on emotion prediction oversimplifies
the problem by categorizing emotions into coarse buckets [14, 25, 3], ignoring
the nuance that the same content can elicit various shades of emotional
reactions for various reasons. For instance, as shown in Figure 1, an image
and caption can generate two conflicting emotions with valid reasons. Learning
this diversity of reactions is crucial to tailor a machine’s interactions to
individual emotional states.

Figure 1: Socratis benchmark. The same image and caption pair can evoke
different emotions and reactions. We release a benchmark dataset of diverse
human-annotated emotions and reactions to images and captions. We show that
current state-of-the-art language models and metrics fail to capture the
nuance of this task.

To encourage further research on emotionally aware AI, we propose the SOCRATIS
benchmark - a dataset of detailed diverse reactions written by humans on
images and captions. SOCRATIS includes 980 shades of emotions and 18K free-
form reactions written by humans on 2075 image and caption pairs collected
from 5 existing news and image-caption datasets. Given an image, a news
caption and an emotion word, our task is to generate a “reaction” caption that
explains why the image and caption may elicit the specified emotion in a
person.

Unlike related benchmarks that focus on reactions to artistic images [12, 1],
niche topics such as gun violence [18], emotionally stylized captions [2], or
on morality [9], we focus on reasons why humans might feel various emotions on
real-life images and captions from widely used news and image-caption
datasets. Our task has higher practical relevance since models that fare well
on this benchmark can be used to create more effective and inclusive messaging
by news agencies and social workers. For instance, as shown in Figure 1, a
writer can look at why someone may be “scared” and update the content to
reflect that transporting the rhinoceros by helicopter is safe in this context
to effectively mitigate the doubt. While a similar benchmark [6] focuses on
reactions to reading a text caption, we focus reactions to both images and
text since content on the web is largely multimodal.

Using Socratis as a testbed, we evaluate a state-of-the-art multimodal
language model [10] and check if commonly used metrics in language-generation
evaluation can distinguish between good “reactions” and poor ones. We generate
“reactions” given an image, caption, and emotion and ask human raters to
blindly choose between the machine generation and the human annotation.

Our results show a stark gap in current capabilities for this task. Humans
prefer human-written reactions two times more often than machine-generated
ones. This starkly contrasts recent findings where humans fare poorly to tell
the difference between machine-generated and AI-generated news articles [27]
or images [19]. This illustrates that while large generative models may be
good at producing believable articles or images, it lacks the nuance required
for emotional awareness. Furthermore, when we separate the generations into
two groups - good generations as rated by humans (when humans couldn’t tell
the difference between human-written and machine-generated) and poor
generations (when humans picked human-written over machine-generated), we
observe a negligible difference in the scoring by commonly used metrics like
BART and CLIP score, which are also based on large language and vision models.
This illustrates the difficulty of the problem since we need better
emotionally aware models to make better metrics.

Hence, we hope these initial results spark further research and discussion
into improving the emotional awareness of large language models. Adding to the
recent discussion that large language models seem to lack rich social
intelligence [20] and theory of mind [21], our benchmark shows that they lack
nuanced emotional awareness as well.

##  2 SOCRATIS Dataset

We propose a benchmark for evaluating the emotional awareness of vision-
language models. Specifically, our task is to predict the reaction a human may
have of a certain emotion while looking at an image and caption (IC) pair. To
this end, we collect a dataset showing human workers an IC pair and ask them
to write the emotion words they feel and the reasons for feeling them. We
interchangeably call these reasons as “reactions” the humans feel of a certain
emotion. We have 18,378 annotated reactions for 980 emotions to 2075 IC pairs
rated by an average of 8 independent workers per IC pair. The most common
emotion words follow that of standard emotion datasets [14, 3] \- happy, sad,
excited, angry. We also have a variety of more subtle emotion words such as
inspire, hopeful, and nostalgic that feature among the top 30 emotion words.
However, the novelty of our dataset isn’t the variety of emotion words, but
the free-form reasons for feeling the different emotions for the same IC pair.
Some examples are shown in Figure 2. We will make our dataset publicly
available.

Figure 2: Qualitative examples from our Socratis dataset and a state-of-the-
art multimodal model, BLIP-v2 generations.

###  2.1 Data Annotation

Image-Caption pair collection We collect image-caption pairs from the Visual
News [11] and the Conceptual Captions [22] dataset. We first randomly sample
1800 images from the Visual News Dataset [11], which consists of an image and
a news headline caption from 4 news datasets - BBC, USA-Today, Guardian and
The Washington Post. We specifically choose news datasets since news headlines
and visuals usually elicit stronger reactions from humans than generic stock
images and are of more practical use. Additionally, to also understand how
people react to generic IC pairs, we sample images and captions from a widely
used image-captioning dataset, Conceptual Captions [22]. IC pairs that
explicitly convey a certain emotion (eg, stock photo of someone smiling) are
less interesting due to a lack of ambiguity from diverse people. Hence, to
sample IC pairs that are likely to elicit diverse emotional reactions, we
choose 500 samples where the emotion of the image does not match the emotion
of the caption. The emotion of the image is predicted by a CLIP [16] model,
fine-tuned on the WebEmo [14] dataset, and a T5 model [17] predicts the text
emotions.

Human Reaction Annotation We show users the IC pairs we collect as described
above. We then ask them to write up to three emotions a person is likely to
feel while viewing the IC pair, along with a reason for why they might feel
each of the emotions they entered. We collect our dataset on Amazon Mechanical
Turk. All data is cleared of personally identifiable information and only
aggregate statistics of model training are reported in the paper. We
incentivize the annotators by awarding them a small bonus if they can match
the most popular emotion word for an IC pair. This encourages workers not to
enter noisy reactions (or extremely uncommon or made-up words) since they are
not likely to match other workers’ responses. To control the quality of
responses, we choose workers with a greater than 98% approval rate (on at
least 50 Human Intelligence Tasks or HIT’s). We also restrict the geographical
location to the US or UK since the images and captions are sourced from news
articles from these two countries. We collect 10 independent annotations for
each image-caption pair to get a representative, diverse set of reactions.

###  2.2 Annotation Quality

We compute some heuristic automatic measures to judge the quality of the
reactions.

Emotion-reaction match We check how often the entered emotion word matches the
emotion of the reaction. For instance, the emotion of “sad” reaction should
also be “sad”. To compute the sentiment, we use a T5 sentiment model [17],
which predicts positive or negative sentiment. We observe an $87.15\%$
accuracy of sentiment match between the emotion word and the reaction. We
manually check some examples where the sentiments of reaction and emotion
words mismatch and find that the T5 model [17] may be noisy. For instance, for
the emotion word “hungry”, the reaction annotated is “the cocoa actually makes
me want to eat something sweet.”. We believe this is a reasonable reaction for
“hungry”. However, the sentiment predicted for “hungry” is “negative”, whereas
the same for the reaction is “positive”, resulting in a mismatch. This further
illustrates that coarse emotion buckets like positive and negative don’t
capture the nuances of reactions of what people feel.

Agreement on reactions To judge the agreement of humans over the reactions for
a given image, caption, and emotion, we further compute the BART [26] scores
of the reactions for the same (I,C,emotion) tuple. We contrast this with the
BART score [26] of 1000 randomly sampled reaction pairs from different IC-
pairs and emotions. We note that the BART scores between reactions for the
same emotion for an IC pair are higher $78\%$ of the time than that of the
random pairs.

##  3 Approach

To understand the emotional awareness of large multimodal models, we benchmark
the capability of a state-of-the-art vision-language model on our proposed
benchmark without further fine-tuning. Specifically, we use the FLAN-T5
variant of BLIP-2 [10] from Huggingface [24]. Our task is to predict the
reason of feeling a certain emotion given an image-caption pair. For a given
image, caption text and emotion word, we prompt BLIP-2 with the image and a
query using the following template: Question: Why does a person feel {emotion}
after seeing this image and reading the news caption ‘{caption}’? Answer:

We input this prompt along with the image to the BLIP-2 model, and use a
greedy approach to generate the response following the standard procedure
outlined on the Huggingface [24] model page. Given an image $i$, caption $c$
and emotion $e$, we can formulate the likelihood of the generated reaction
text $t$ as:

$P(t|i,c,e)=\Pi_{j=1}^{n}P(t_{<j}|i,c,e)$ (1)

where $n$ denotes the number of tokens in the generated text.

##  4 Experiments

Our goal is to evaluate whether large multimodal models are emotionally aware
to generate plausible reasons for why humans might feel a certain reaction for
a given image and text. A pragmatic formulation of this evaluation is whether
a human finds it hard to distinguish a machine generated reason from a human-
written reason. Hence, we first conduct a human evaluation on the multimodal
BLIP-2 generated reasons to see how often humans prefer machine-generated
reasons, human-written reasons, or both.

Human Evaluation To understand human preference, we conduct a preliminary
human study on 500 randomly sampled data points from our dataset. We show an
image, caption, emotion tuple to a user. We then show two choices of reactions
- human-written and machine-generated. The user is unaware of which of the
reactions is machine-generated or human-written. We ask 3 independent workers
to choose the best reaction for the given tuple of image, caption and emotion.
They also have the choice of either choosing that both reactions are
reasonable or that neither is reasonable. Hence, there are four choices for
each image-caption-emotion tuple: We split the human study image-caption-
emotion tuples into three groups:

  1. –

Machine-better: machine was picked over human, indicating that the machine-
generations are good.

  2. –

Both-Good: both human-written and machine-generated reactions are equally
valid.

  3. –

Both-Bad: We discard these examples from further study since the data
annotations are likely noisy.

Metrics Since human evaluations are slow and expensive, we aim to determine if
commonly used captioning metrics can be used to judge good reaction
generations from poor ones to speed up research. We define a good generation
as one which is aligned to or better than human preferences. Hence, a machine
generation is good if a human cannot tell the difference from a human-written
one, or if a human prefers it over a human-written one. Hence, a good metric
should score such cases (machine-preferred and both-good) higher than the
cases where a human-written reason was preferred over a machine-generated one
(human-preferred). We compute three commonly used metrics. First, we compute
BART Score between the machine-generated reaction and the human-generated
reference to measure human-likeness of generations as described in [26]. Next,
we compute CLIP-Score and RefCLIP-Score [8] to see if image-relevance plays a
major factor in distinguishing good generations from poor ones. We use a
prompt like “Human feels {emotion}, when seeing this image with {caption}
because {explanation}” and compute the cosine similarity to the image as
described in [8].

Table 1: Number of times majority ($\frac{2}{3}$) humans prefer human-written reactions, BLIP-2 (machine) reactions, or both out of 382 examples. Machine-better | Human-better | Both-Good | Both-Bad
---|---|---|---
91 | 233 |  47 |  11
Table 2: Evaluations with BART and CLIP-Score on subsets where humans prefer human-written reactions, BLIP-2 (machine) reactions, or both. We want machine-better or both-good generations scored higher than human-better. | Machine-better | Human-better | Both-Good
---|---|---|---
BART | -5.48 | -5.42 | -5.54
CLIP-Score | 0.74 | 0.76 | 0.75
RefCLIP | 0.42 | 0.43 | 0.42
Table 3: Multimodal vs a text-only model on relevance of generation to the image | CLIP $\uparrow$ | Ref-CLIP $\uparrow$
---|---|---
BLIP-2 | 0.75 | 0.42
FLAN-T5 | 0.73 | 0.42

##  5 Results

Humans prefer human-written reactions to machine generations. In Table 1, we
observe that workers pick human reactions over two times more often than
machine-generated ones (233 vs 91). This suggests that state-of-the-art large
vision-and-language models are still limited at extrapolating contextual
information beyond simply correlating the visual concepts in the image with
relevant words.

Current captioning metrics cannot distinguish between good and bad reactions
In Table 2, we see that BART scores do not follow human preferences. The
scores of the machine generations for when the machine was rated better or
equally good (both-good) by humans are not higher than the scores when the
machine generations are poor (human-better). CLIP and RefCLIP also do not seem
to differ across the three sets. This suggests that visual similarity may also
not be important in distinguishing good from poor reactions. Further
investigation is required to check if we can train a custom BART metric based
on a few examples in our dataset.

Multimodal models are slightly more image-relevant We also check the
performance of a language-only model compared to the multimodal BLIP-2 by
using only the langauge model in BLIP2, which is a FLAN-T5 model. In Table 3,
we see that the relevance to the image is understandably higher (acc to CLIP
scores) for BLIP-2. However, based on results in Table 2, this doesn’t
necessarily mean that the generations are more preferred by humans.

Discussion and Conclusion We propose the Socratis benchmark to generate
reactions for why images and captions elicit certain emotions in humans. Our
initial experiments indicate that a state-of-the-art vision-language model is
unable to extrapolate the required information to generate reasonable
reactions. However, further research is required to investigate biases that
exist in our dataset and how these are perpetuated in the models that are
benchmarked on it. Further investigation is also required to see if quick
fixes like changing the generation strategy, adapting a few layers, or in-
context learning (showing examples in prompts) can make these models more
emotionally aware. We hope our Socratis benchmark will encourage future
research on training and evaluating emotionally aware AI algorithms.

Acknowledgements: We are thankful to Praneeth Chandra Bogineni for valuable
discussions in the initial phase of the project. This material is based upon
work supported, in part, by DARPA under agreement number HR00112020054. The
findings in the paper do not reflect the opinions of the US Government or
DARPA.

## References

  * [1] Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J Guibas.  Artemis: Affective language for visual art.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11569–11579, 2021.
  * [2] Kezhen Chen, Qiuyuan Huang, Daniel McDuff, Xiang Gao, Hamid Palangi, Jianfeng Wang, Kenneth Forbus, and Jianfeng Gao.  Nice: Neural image commenting with empathy.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4456–4472, 2021.
  * [3] Vaidehi Dalmia, Hongyi Liu, and Shih-Fu Chang.  Columbia mvso image sentiment dataset, 2016.
  * [4] Ding Ding, Edward W Maibach, Xiaoquan Zhao, Connie Roser-Renouf, and Anthony Leiserowitz.  Support for climate policy and societal action are linked to perceptions about scientific agreement.  Nature Climate Change, 1(9):462–466, 2011.
  * [5] Ajay Divakaran, Karan Sikka, Arijit Ray, Xiao Lin, and Yi Yao.  User targeted content generation using multimodal embeddings, Sept. 23 2021.  US Patent App. 17/191,698.
  * [6] Saadia Gabriel, Skyler Hallinan, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi.  Misinfo reaction frames: Reasoning about readers’ reactions to news headlines.  arXiv preprint arXiv:2104.08790, 2021.
  * [7] Daniel Goleman.  Emotional intelligence. why it can matter more than iq.  Learning, 24(6):49–50, 1996.
  * [8] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  arXiv preprint arXiv:2104.08718, 2021.
  * [9] Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al.  Can machines learn morality? the delphi experiment.  arXiv preprint arXiv:2110.07574, 2021.
  * [10] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023.
  * [11] Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez.  Visual news: Benchmark and challenges in news image captioning.  arXiv preprint arXiv:2010.03743, 2020.
  * [12] Youssef Mohamed, Mohamed Abdelfattah, Shyma Alhuwaider, Feifan Li, Xiangliang Zhang, Kenneth Ward Church, and Mohamed Elhoseiny.  Artelingo: A million emotion annotations of wikiart with emphasis on diversity over language and culture.  arXiv preprint arXiv:2211.10780, 2022.
  * [13] OpenAI.  Gpt-4 technical report, 2023.
  * [14] Rameswar Panda, Jianming Zhang, Haoxiang Li, Joon-Young Lee, Xin Lu, and Amit K Roy-Chowdhury.  Contemplating visual emotions: Understanding and overcoming dataset bias.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 579–595, 2018.
  * [15] Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba.  Watch-and-help: A challenge for social perception and human-ai collaboration.  arXiv preprint arXiv:2010.09890, 2020.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [17] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  The Journal of Machine Learning Research, 21(1):5485–5551, 2020\.
  * [18] Carley Reardon, Sejin Paik, Ge Gao, Meet Parekh, Yanling Zhao, Lei Guo, Margrit Betke, and Derry Tanti Wijaya.  BU-NEmo: an affective dataset of gun violence news.  In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2507–2516, Marseille, France, June 2022. European Language Resources Association.
  * [19] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [20] Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi.  Neural theory-of-mind? on the limits of social intelligence in large LMs.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762–3780, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics.
  * [21] Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov.  Minding language models’ (lack of) theory of mind: A plug-and-play multi-character belief tracker.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13960–13980, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * [22] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556–2565, 2018.
  * [23] Bertie Vidgen, Austin Botelho, David Broniatowski, Ella Guest, Matthew Hall, Helen Margetts, Rebekah Tromble, Zeerak Waseem, and Scott Hale.  Detecting east asian prejudice on social media.  arXiv preprint arXiv:2005.03909, 2020.
  * [24] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Huggingface’s transformers: State-of-the-art natural language processing.  arXiv preprint arXiv:1910.03771, 2019.
  * [25] Jingyuan Yang, Qiruin Huang, Tingting Ding, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  Emoset: A large-scale visual emotion dataset with rich attributes.  arXiv preprint arXiv:2307.07961, 2023.
  * [26] Weizhe Yuan, Graham Neubig, and Pengfei Liu.  Bartscore: Evaluating generated text as text generation.  Advances in Neural Information Processing Systems, 34:27263–27277, 2021.
  * [27] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi.  Defending against neural fake news.  Advances in neural information processing systems, 32, 2019.


# Cosmology of Plane Geometry

Alexander Skutin

###### Key words and phrases:

Plane geometry, Analytic geometry

###### 2010 Mathematics Subject Classification:

51M04, 51N20

##  1\. Introduction

This paper focuses on a new approach to plane geometry and develops important
concepts that can allow researchers to unite and observe plane geometry from a
new, meaningful perspective. The present short note is a first chapter of the
paper, referenced in the last section.

##  2\. Deformation principle

In the area of plane geometry, the deformation principle refers to replacing a
certain configuration of points, lines, and circles with more general ones, in
which equal points, lines, or circles in the original (undeformed)
configuration are replaced by their deformed versions, i.e. points, lines, and
circles that are not equal in the general case but are related to one another.
If one is aware that points, lines, or circles are equal in the case of non-
deformed configuration, one can predict the deformed versions connections in
terms of general configuration. Consideration of the non-deformed case aids in
predicting which points should be connected in the general deformed case.

###  2.1. Basic deformation example

For the first example of application of deformation principle consider the
configuration of a square $ABCD$ with its center $O$. Now, we can look on the
point $O$ as on the third vertices $O_{ab}$, $O_{bd}$, $O_{cd}$, $O_{da}$ of
the triangles $O_{ab}AB$, $O_{bc}BC$, $O_{cd}CD$, $O_{da}DA$ constructed
internally on the sides of $ABCD$ which are isosceles right angled triangles
($\angle AO_{ab}B=90^{\circ}$, $|O_{ab}A|=|O_{ab}B|$ and so on). So, in the
case when $ABCD$ is a square we see that $O=O_{ab}=O_{bc}=O_{cd}=O_{da}$ are
equal, thus, the deformation principle predicts that in the general (deformed)
case of an arbitrary $ABCD$, deformed points $O_{ab}$, $O_{bc}$, $O_{cd}$,
$O_{da}$ should be connected. And in fact, they are: $O_{ab}O_{cd}$ is
perpendicular to $O_{bc}O_{da}$ and they have equal lengths.

Therefore, we can state the next result

Theorem 1 (Deformation of a square with its center). Consider any (convex)
quadrilateral $ABCD$ and let points $O_{ab}$, $O_{bc}$, $O_{cd}$, $O_{da}$ are
chosen inside of $ABCD$ such that $O_{ab}AB$, $O_{bc}BC$, $O_{cd}CD$,
$O_{da}DA$ are isosceles right angled triangles (see picture below).

Then the following properties of the points $O_{ab}$, $O_{bc}$, $O_{cd}$,
$O_{da}$ are true:

  1. (1)

$O_{ab}O_{cd}\perp O_{bc}O_{da}$

  2. (2)

$|O_{ab}O_{cd}|=|O_{bc}O_{da}|$.

$A$$D$$C$$B$$O$ (a) Theorem 1 (Square case)

$A$$D$$B$$C$$O_{da}$$O_{cd}$$O_{bc}$$O_{ab}$ (b) Theorem 1

In the previous theorem we see how the deformation principle works if we
deform a square with its center, similarly we can deform other simple
configurations and get complicated results. Each time when we deform we should
have decided which points we like to deform and how we deform them. For
example, in the previous case we can look on the square’s center $O$ as on the
intersection of the internal angle $\angle ABC$, $\angle BCD$, $\angle CDA$,
$\angle DAB$ bisectors which will provide us with another type of deformation
of $ABCDO$ (in the case of the angle bisectors the resulting deformational
fact may be stated as : Points $O_{1}$, $O_{2}$, $O_{3}$, $O_{4}$ are cyclic).
To see why the deformation principle is natural we can look on it as on the
reverse process to the process of consideration of particular cases of
theorems. For example, Theorem 1 becomes trivial in the particular case of a
square because in this case points $O_{ab}$, $O_{bc}$, $O_{cd}$, $O_{da}$
become equal. So, the particular case should be seen as an evidence of the
general one.

###  2.2. Possible deformations

Theorem 1 is the deformation of four nested points into four connected points
in the general case. Similarly, other objects equal (or trivially connected)
in the non-deformed case can be deformed to understand their connected
deformations. The following table shows which objects may be considered non-
deformed and how their deformations may appear:

Undeformed objects | Possible deformations
---|---
Coincide points | Collinear points, concyclic points
Coincide lines | Concurrent lines
Coincide circles |  | Concurrent circles, coaxial circles, circles having
---
radical line which has many nice properties wrt
original configuration (if there are two circles)
Coincide triangles |  | Perspective triangles, triangles which vertices
---
are lying on the same conic
Concyclic points |  | Concyclic points, points lying on the same conic
---

###  2.3. Deformations of an equilateral triangle

Instead of a square $ABCD$, this paper uses an equilateral triangle $ABC$ and
attempts to deform its center, incircle, circumcircle, and many other closely
related objects. Deformation of an equilateral triangle is the most powerful
tool for producing and verifying results throughout this paper.

As an example of the deformation of an equilateral triangle, we can consider
an equilateral triangle with center $O$ and interpret $O$ as the entire set of
Clark Kimberling centers $X_{i}$ (see [1]). Therefore, the deformation theory
predicts that, in the general case of an arbitrary triangle $ABC$, the
triangle centers $X_{i}$ should have many relations among themselves, such as
collinearity, concyclity, etc. Many of these relationships may be found in
[1].

Next we introduce some first examples of application of the deformation
principle in the case of equilateral triangles.

Example 1. Consider any triangle $ABC$. Let $A^{\prime}BC$ be the equilateral
triangle constructed on the side $BC$ such that $A^{\prime}$, $A$ are on the
same half plane wrt $BC$. Denote the center $O_{a}$ of $A^{\prime}BC$.
Similarly define $O_{b}$, $O_{c}$. In the case of equilateral $ABC$ we get
that $O_{a}=O_{b}=O_{c}$ – coincides with the center $O$ of $ABC$. So, we can
predict that in the general case of an arbitrary $ABC$ we will have that the
points $O_{a}$, $O_{b}$, $O_{c}$ will be connected and have relations with the
base triangle $ABC$. And, in fact, the next such relation can be formulated:
Points $O_{a}$, $O_{b}$, $O_{c}$ form the equilateral triangle which
circumcircle is passing through the first Fermat point of $ABC$.

Example 2. Consider a triangle $ABC$ with the first and second Fermat points
$F_{1}$, $F_{2}$. Let $F_{a}$, $F_{b}$, $F_{c}$ be the second Fermat points of
$F_{1}BC$, $F_{1}AC$, $F_{1}AB$, respectively. In the case of equilateral
$ABC$ we get that $F_{1}$ coincides with the center $O$ of $ABC$ and $F_{2}$
coincides with a point $P$ lying on the circumcircle of $ABC$. Aditionally, in
the equilateral triangle case we have that $F_{a}$, $F_{b}$, $F_{c}$ are the
reflections of center $O$ of $ABC$ wrt its sides and the points $A$, $B$, $C$,
$F_{a}$, $F_{b}$, $F_{c}$ form a regular hexagon which circumcircle contains
$P$. Thus, we can predict that in the general case of an arbitrary $ABC$ the
configuration $ABCF_{a}F_{b}F_{c}F_{2}$ will inherit some properties of a
regular hexagon and a point $P$ on its circumcircle. And, in fact, the next
such relation can be formulated: Point $F_{2}$ lies on the circumcircle of
$F_{a}F_{b}F_{c}$.

Example 3. Consider a triangle $ABC$ and any point $P$. Let $AP$, $BP$, $CP$
meet the circumcircle $(ABC)$ of $ABC$ second time at $A^{\prime}$,
$B^{\prime}$, $C^{\prime}$ (i.e. $A^{\prime}B^{\prime}C^{\prime}$ is the
circumcevian triangle of $P$ wrt $ABC$). Consider the nine-point centers $N$,
$N_{a}$, $N_{b}$, $N_{c}$ of $ABC$, $A^{\prime}BC$, $B^{\prime}AC$,
$C^{\prime}AB$, respectively. Denote

  1. (1)

the reflections $N_{a}^{\prime}$, $N_{b}^{\prime}$, $N_{c}^{\prime}$ of
$N_{a}$, $N_{b}$, $N_{c}$ wrt $BC$, $AC$, $AB$, respectively

  2. (2)

the reflections $N_{a}^{\prime\prime}$, $N_{b}^{\prime\prime}$,
$N_{c}^{\prime\prime}$ of $N_{a}$, $N_{b}$, $N_{c}$ wrt midpoints of $BC$,
$AC$, $AB$, respectively.

In the case of equilateral $ABC$ and $P=O$ – its center we get that
$N_{a}^{\prime}=N_{b}^{\prime}=N_{c}^{\prime}=N=P=O$ and
$N_{a}^{\prime\prime}=N_{b}^{\prime\prime}=N_{c}^{\prime\prime}=N=P=O$. So, we
can predict that in the general case of an arbitrary $ABC$ and an point $P$ we
will have that the points $N_{a}^{\prime}$, $N_{b}^{\prime}$,
$N_{c}^{\prime}$, $N$, $P$ ($N_{a}^{\prime\prime}$, $N_{b}^{\prime\prime}$,
$N_{c}^{\prime\prime}$, $N$, $P$) will be connected with each other. And, in
fact, the next such connections can be formulated:

  1. (1)

Points $N_{a}^{\prime}$, $N_{b}^{\prime}$, $N_{c}^{\prime}$, $N$ lie on the
same circle

  2. (2)

Points $N_{a}^{\prime\prime}$, $N_{b}^{\prime\prime}$, $N_{c}^{\prime\prime}$,
$N$ lie on the same circle.

###  2.4. Near-the-equilateral-triangle shapes

This paper also uses the following shapes (among others), which may be
regarded as shapes close to an equilateral triangle. The deformations of these
shapes will appear in the next sections of the paper.

(c) $\triangle$ with center

(d) $\triangle$ with center 2

(e) $\triangle$ with cevians

(f) $\triangle$ with midpoint triangle

(g) Triangulated $\triangle$

(h) $\triangle$ with incircle

(i) Regular hexagon

(j) Regular hexagon 2

(k) Hexagonal star

(l) Crown shape

##  3\. Complete article

The full version of this article was written by the Author involving geometric
problems contribution by Tran Quang Hung, Kadir Altintas and Antreas
Hatzipolakis.

Download it from the link below

https://www.scribd.com/document/421475794/Cosmology-of-Plane-Geometry-
Concepts-and-Theorems

## References

  * [1] C. Kimberling, Encyclopedia of Triangle Centers – ETC, http://faculty.evansville.edu/ck6/encyclopedia/ETC.html.
  * [2] A. Skutin, Some new theorems in plane geometry, arXiv preprint arXiv:1704.04923 (2017).
  * [3] A. Skutin, Creative geometry, arXiv preprint arXiv:1802.03543 (2018).