# Learning Relationships between Text, Audio, and Video via Deep Canonical
Correlation for Multimodal Language Analysis

Zhongkai Sun,1 Prathusha K Sarma,2 William A Sethares,1 Yingyu Liang1
1University of Wisconsin-Madison, 2Curai
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Work done while at UW-Madison

###### Abstract

Multimodal language analysis often considers relationships between features
based on text and those based on acoustical and visual properties. Text
features typically outperform non-text features in sentiment analysis or
emotion recognition tasks in part because the text features are derived from
advanced language models or word embeddings trained on massive data sources
while audio and video features are human-engineered and comparatively
underdeveloped. Given that the text, audio, and video are describing the same
utterance in different ways, we hypothesize that the multimodal sentiment
analysis and emotion recognition can be improved by learning (hidden)
correlations between features extracted from the outer product of text and
audio (we call this text-based audio) and analogous text-based video. This
paper proposes a novel model, the Interaction Canonical Correlation Network
(ICCN), to learn such multimodal embeddings. ICCN learns correlations between
all three modes via deep canonical correlation analysis (DCCA) and the
proposed embeddings are then tested on several benchmark datasets and against
other state-of-the-art multimodal embedding algorithms. Empirical results and
ablation studies confirm the effectiveness of ICCN in capturing useful
information from all three views.

##  1 Introduction

Human language communication occurs in several modalities: via words that are
spoken, by tone of voice, and by facial and bodily expressions. Understanding
the content of a message thus requires understanding all three modes. With the
explosive growth in availability of data, several machine learning algorithms
have been successfully applied towards multimodal tasks such as sentiment
analysis (?; ?), emotion recognition (?), image-text retrieval (?), and aiding
medical diagnose (?; ?) etc. Among multimodal language sentiment or emotion
experiments involving unimodal features (?; ?; ?; ?), it is commonly observed
that text based features perform better than visual or auditory modes. This is
plausible for at least three reasons: 1) Text itself contains considerable
sentiment-related information. 2) Visual or acoustic information may sometimes
confuse the sentiment or emotion analysis task. For instance: “angry” and
“excited” may have similar acoustic performances (high volume and high pitch)
even though they belong to opposite sentiments. Similarly, “sad” and
“disgusted” may have different visual features though they both belong to the
negative sentiment. 3) Algorithms for text analysis have a richer history and
are well studied.

Based on this observation, learning the hidden relationship between verbal
information and non-verbal information is a key point in multi-modal language
analysis. This can be approached by looking at different ways of combining
multi-modal features.

The simplest way to combine text (T), audio (A) and video (V) for feature
extraction and classification is to concatenate the A, V, and T vectors. An
alternative is to use the outer product (?; ?) which can represent the
interaction between pairs of features, resulting in 2D or 3D arrays that can
be processed using advanced methods such as convolutional neural networks
(CNNs) (?). Other approaches (?; ?; ?; ?) study multi-modal interactions and
intra-actions by using either graph or temporal memory networks with a
sequential neural network LSTM (?). While all these have contributed towards
learning multi-modal features, they typically ignore the hidden correlation
between text-based audio and text-based video. Individual modalities are
either combined via neural networks or passed directly to the final classifier
stage. However, it is obvious that attaching both audio and video features to
the same textual information may enable non-text information to be better
understood, and in turn the non-text information may impart greater meaning to
the text. Thus, it is reasonable to study the deeper correlations between
text-based audio and text-based video features.

This paper proposes a novel model which uses the outer-product of feature
pairs along with Deep Canonical Correlation Analysis (DCCA) (?) to study
useful multi-modal embedding features. The effectiveness of using an outer-
product to extract cross-modal information has been explored in (?; ?). Thus,
features from each mode are first extracted independently at the sentence (or
utterance) level and two outer-product matrices ($T\otimes V$ and $T\otimes
A$) are built for representing the interactions between text-video and between
text-audio. Each outer-product matrix is connected to a convolutional neural
network (CNN) for feature extraction. Outputs of these two CNNs can be
considered as feature vectors for text-based audio and text-based video and
should be correlated.

In order to better correlate the above text-based audio and text-based video,
we use Canonical Correlation Analysis (CCA) (?), which is a well-known method
for finding a linear subspace where two inputs are maximally correlated.
Unlike cosine similarity or Euclidean distance, CCA is able to learn the
direction of maximum correlation over all possible linear transformations and
is not limited by the original coordinate systems. However, one limitation of
CCA is that it can only learn linear transformations. An extension to CCA
named Deep CCA (DCCA) (?) uses a deep neural network to allow non-linear
relationships in the CCA transformation. Recently several authors (?; ?) have
shown the advantage of using CCA-based methods for studying correlations
between different inputs. Inspired by these, we use DCCA to correlate text-
based audio and text-based video. Text-based audio and text-based video
features derived from the two CNNs are input into a CCA layer which consists
of two projections and a CCA Loss calculator. The two CNNs and the CCA layer
then form a DCCA, the weights of the two CNNs and the projections are updated
by minimizing the CCA Loss. In this way, the two CNNs are able to extract
useful features from the outer-product matrices constrained by the CCA loss.
After optimizing the whole network, outputs of the two CNNs are concatenated
with the original text sentence embedding as the final multi-modal embedding,
which can be used for the classification.

We evaluate our approach on three benchmark multi-modal sentiment analysis and
emotion recognition datasets: CMU-MOSI (?), CMU-MOSEI (?), and IEMOCAP(?).
Additional experiments are presented to illustrate the performance of the ICCN
algorithm. The rest of the paper is organized as follows: Section 2 presents
related work, Section 3 introduces our proposed model and Section 4 describes
our experimental setup. Section 5 presents a discussion on the empirical
observations, Section 6 concludes this work.

##  2 Related Work

The central themes of this paper are related to learning (i) multi-modal
fusion embeddings and (ii) cross-modal relationships via canonical correlation
analysis (CCA).

Multi-modal fusion embedding: Early work (?) concatenates the audio, video and
text embeddings to learn a larger multi-modal embedding. But this may lead to
a potential loss of information between different modalities. Recent studies
on learning multi-modal fusion embeddings train specific neural network
architectures to combine all three modalities. In their work (?) propose
improvements to multi-modal embeddings using reinforcement learning to align
the multi-modal embedding at the word level by removing noises. A multi-modal
tensor fusion network is built in (?) by calculating the outer-product of
text, audio and video features to represent comprehensive features. However
this method is limited by the need of a large computational resources to
perform calculations of the outer dot product. In their work (?) developed an
efficient low rank method for building tensor networks which reduce
computational complexity and are able to achieve competitive results. A Memory
Fusion Network (MFN) is proposed by (?) which memorizes temporal and long-term
interactions and intra-actions between cross-modals, this memory can be stored
and updated in a LSTM. (?) learned multistage fusion at each LSTM step so that
the multi-modal fusion can be decomposed into several sub-problems and then
solved in a specialized and effective way. A multimodal transformer is
proposed by (?) that uses attention based cross-modal transformers to learn
interactions between modalities.

Cross-modal relationship learning via CCA: Canonical Correlation Analysis(CCA)
(?) learns the maximum correlation between two variables by mapping them into
a new subspace. Deep CCA (DCCA) (?) improves the performance of CCA by using
feedforward neural networks in place of the linear transformation in CCA.

A survey of recent literature sees applications of CCA-based methods in
analyzing the potential relationship between different variables. For example,
a CCA based model to combine domain knowledge and universal word embeddings is
proposed by (?). Models proposed by (?) use Deep Partial Canonical Correlation
Analysis (DPCCA), a variant of DCCA, for studying the relationship between two
languages based on the same image they are describing. Work by (?)
investigates the application of DCCA to simple concatenations of multimodal-
features, while (?) applied CCA methods to learn joint-representation for
detecting sarcasm. Both approaches show the effectiveness of CCA methods
towards learning potential correlation between two input variables.

##  3 Methodology

This section first introduces CCA and DCCA. Next, the interaction canonical
correlation network (ICCN), which extracts the interaction features of a CNN
in a DCCA-based network, is introduced. Finally, the whole pipe-line of using
this method in a multimodal language analysis task is described.

### CCA and DCCA

Given two sets of vectors $X\in\mathbb{R}^{n_{1}\times m}$ and
$Y\in{\mathbb{R}}^{n_{2}\times m}$, where $m$ denotes the number of vectors,
CCA learns two linear transformations $A\in\mathbb{R}^{n_{1}\times r}$ and
$B\in\mathbb{R}^{n_{2}\times r}$ such that the correlation between $A^{T}X$
and $B^{T}Y$ is maximized. Denote the covariances of $X$ and $Y$ as
$S_{11},S_{22}$, and the cross-covariance of $X,Y$ as $S_{12}$. The CCA
objective is

$\begin{split}A^{*},B^{*}&=\mathop{\arg\max}_{A,B}\textrm{corr}(A^{T}X,B^{T}Y)\\\
&=\mathop{\arg\max}_{A,B}\frac{A^{T}S_{12}B}{\sqrt{A^{T}S_{11}A}\sqrt{B^{T}S_{22}B}}.\end{split}$
(1)

The solution of the above equation is fixed and can be solved in multiple ways
(?; ?). One method suggested by (?) lets $U,S,V^{T}$ be the Singular Value
Decomposition ($SVD$) of the matrix $Z$ =
$S_{11}^{-\frac{1}{2}}S_{12}S_{22}^{-\frac{1}{2}}$. Then $A^{*},B^{*}$ and the
total maximum canonical correlation are

$\begin{split}A^{*}&=S_{11}^{-\frac{1}{2}}U\\\
B^{*}&=S_{22}^{-\frac{1}{2}}V\\\
\textrm{corr}(A^{*T}X,B^{*T}Y)&=\textrm{trace}(Z^{T}Z)^{\frac{1}{2}}.\end{split}$
(2)

One limitation of CCA is that it only considers linear transformations. DCCA
(?) learns non-linear transformations using a pair of neural networks. Let
$f,g$ denote two independent neural networks, the objective of DCCA is to
optimize parameters $\theta_{f},\theta_{g}$ of $f,g$ so that the canonical
correlation between the output of $f$ and $g$, denoted as $F_{X}=$
$f(X;\theta_{1})$ and $F_{Y}=$ $g(Y;\theta_{2})$, can be maximized by finding
two linear transformations $C^{*},D^{*}$. The objective of DCCA is

$\begin{split}\theta_{f}^{*},\theta_{g}^{*}&=\mathop{\arg\max}_{\theta_{f},\theta_{g}}\textrm{CCA}(F_{X},F_{Y})\\\
&=\mathop{\arg\max}_{\theta_{f},\theta_{g}}\textrm{corr}(C^{*T}F_{X},D^{*T}F_{Y}).\end{split}$
(3)

In order to update the parameters of $f,g$, a loss for measuring the canonical
correlation must be calculated and back-propagated. Let $R_{11},R_{22}$ be
covariances of $F_{X},F_{Y}$, the cross-covariance of $F_{X},F_{Y}$ as
$R_{12}$. Let $E=R_{11}^{-\frac{1}{2}}R_{12}R_{22}^{-\frac{1}{2}}$. According
to (2), the canonical correlation loss for updating $F_{x},F_{Y}$ can be
defined by

$\textrm{CCA Loss}=-\textrm{trace}(E^{T}E)^{\frac{1}{2}}.$ (4)

Networks $f(X;\theta_{f}),g(Y;\theta_{g})$’s parameters can be updated by
minimizing the CCA Loss (4) (i.e. maximize the total canonical correlation).

### Text Based Audio Video Interaction Canonical Correlation

Previous work of (?; ?) on multi-modal feature fusion has shown that the
outer-product is able to learn interactions between different features
effectively. Thus, we use the outer-product to represent text-video and text-
audio features. Given that outer-product and DCCA are applied at the utterance
(sentence)-level, we extract utterance level features for each uni-modal
independently in order to test the effectiveness of ICCN more directly. Let
$H_{t}\in\mathbb{R}^{d_{t}}$ be the utterance-level text feature embedding,
and $H_{v}\in\mathbb{R}^{d_{v}\times l_{v}},H_{a}\in\mathbb{R}^{d_{a}\times
l_{a}}$ be the video and audio input sequences. A 1D temporary convolutional
layer is used to extract local structure of the audio and video sequences, and
the outputs of the 1D-CNN are denoted as $H_{a1}\in\mathbb{R}^{d_{a1}\times
l_{a}},H_{v1}\in\mathbb{R}^{d_{v1}\times l_{v}}$. Next, two LSTMs process the
audio and video sequences. The final hidden state of each LSTM is used as the
utterance-level audio or video feature, denoted as
$H_{a2}\in\mathbb{R}^{d_{a2}},H_{v2}\in\mathbb{R}^{d_{v2}}$. Once each
utterance-level feature has been obtained, the text-based audio feature matrix
and text-based video feature matrix can then be learned using the outer-
product on $H_{t},H_{v2},H_{a2}$:

$\begin{split}H_{ta}&=H_{t}\otimes H_{a2},H_{ta}\in\mathbb{R}^{d_{t}\times
d_{a2}}\\\ H_{tv}&=H_{t}\otimes H_{v2},H_{tv}\in\mathbb{R}^{d_{t}\times
d_{v2}}.\end{split}$ (5)

In order to extract useful features from the outer-product matrices
$H_{ta},H_{tv}$, a Convolutional Neural Network is used as the basic feature
extractor. $H_{ta}$ and $H_{tv}$ are connected by multiple 2D-CNN layers with
max pooling. Outputs of the two 2D-CNNs are reshaped to 1D vector and then be
used as inputs to the CCA Loss calculation. 1D-CNN, LSTM, and 2D-CNN’s weights
are again updated using the back-propagation of the CCA Loss (4). Thus the two
2D-CNNs learn to extract features from $H_{tv}$ and $H_{ta}$ so that their
canonical correlation is maximized.

Figure 1: ICCN method for aligning text-based audio features and text-based
video features. Sentence level uni-modal features are extracted independently.
Outer-product matrices of text-audio and text-video are used as input to the
Deep CCA network. After learning the CNN’s weights using the CCA Loss, outputs
of the two CNNs are concatenated with the original text to form the multi-
modal embedding. This can be used as input to independent downstream tasks.

Algorithm 1 provides the pseudo-code for the whole Interaction Canonical
Correlation Network (ICCN).

0: Data $H_{t}\in\mathbb{R}^{d_{t}\times N}$, $H_{a}\in\mathbb{R}^{d_{a}\times
l_{a}\times N}$, $H_{v}\in\mathbb{R}^{d_{v}\times l_{v}\times N}$, $epoch$,
$\eta$

Initialize $W_{a}$ of ($\textrm{CNN1D}_{a},\textrm{LSTM}_{a}$, and
$\textrm{CNN2D}_{ta}$)

Initialize $W_{v}$ of ($\textrm{CNN1D}_{v},\textrm{LSTM}_{v}$, and
$\textrm{CNN2D}_{tv}$)

while $epoch>0$ do

$H_{a1}=\textrm{CNN1D}_{a}(H_{a})$

$H_{v1}=\textrm{CNN1D}_{v}(H_{v})$

$H_{a2}=\textrm{LSTM}{a}(H_{a1})$

$H_{v2}=\textrm{LSTM}{v}(H_{v1})$

$H_{tv}=H_{t}\otimes H_{v2}$, $H_{ta}=H_{t}\otimes H_{a2}$

$K_{tv}=\textrm{CNN2D}_{tv}(H_{tv})$

$K_{ta}=\textrm{CNN2D}_{ta}(H_{ta})$

Compute gradients $\nabla W_{v}$, $\nabla W_{a}$ of:

$\min\limits_{W_{v},W_{a}}\textrm{CCALoss}(K_{tv},K_{ta})$

Update:

$W_{v}\leftarrow W_{v}-\eta\nabla W_{v}$

$W_{a}\leftarrow W_{a}-\eta\nabla W_{a}$

$epoch\leftarrow epoch-1$

end while

$K_{tv},K_{ta}$

Algorithm 1 Interaction Canonical Correlation Network

### Pipe-line for Downstream Tasks

The ICCN method acts as a feature extractor. In order to test its performance,
an additional downstream classifier is also required. Uni-modal features can
be extracted using a variety of simple extraction schemes as well as by
learning features using more complex neural network based models such as a
sequential LSTM. Once uni-modal features for text, video, and audio have been
obtained, the ICCN can be used to learn text-based audio feature $K_{ta}$ and
text-based video feature $K_{tv}$. The final multi-modal embedding can be
formed as the concatenation of the text-based audio, the original text, and
the text-based video features, which are denoted as $[K_{ta};H_{t};K_{tv}]$.
This $[K_{ta};H_{t};K_{tv}]$ can then be used as an input to downstream
classifiers such as logistic regression or multilayer perceptron. Figure 1
shows the pipe-line using the ICCN for downstream tasks in our work.

##  4 Experiment Settings

This section describes the experimental datasets and baseline algorithms
against which ICCN is compared.

### Datasets

The proposed algorithm is tested using three public benchmark multi-modal
sentiment analysis and emotion recognition datasets: CMU-MOSI (?), CMU-MOSEI
(?), and IEMOCAP(?). Both CMU-MOSI and CMU-MOSEI’s raw features, and most of
the corresponding extracted features can be acquired from the CMU-
MultimodalSDK (?).

  * •

CMU-MOSI: This dataset is a multi-modal dataset built on 93 Youtube movie
reviews. Videos are segmented to 2198 utterance clips, and each utterance
example is annotated on a scale of [-3, 3] to reflect sentiment intensity. -3
means an extremely negative and 3 means an extremely positive sentiment. This
data set is divided into three parts, training (1283 samples), validation (229
samples) and test (686 samples).

  * •

CMU-MOSEI: This dataset is similar to the CMU-MOSI, but is larger in size. It
consists of 22856 annotated utterances extracted from Youtube videos. Each
utterance can be treated as an individual multi-modal example. Train,
validation, and test sets contain 16326, 1871, and 4659 samples respectively.

  * •

IEMOCAP: This dataset contains 302 videos in which speakers performed 9
different emotions (angry, excited, fear, sad, surprised, frustrated, happy,
disappointed and neutral). Those videos are divided into short segments with
emotion annotations. Due to the imbalance of some emotion labels, we follow
experiments in previous papers (?; ?) where only four emotions (angry, sad,
happy, and neutral) are used to test the performance of the algorithm. Train,
validation, and test partitions contain 2717, 789, and 938 data samples
respectively.

### Multi-modal Features

The following uni-modal features are used prior to combinations,

  * •

Text Features: For MOSI and MOSEI, we use a pre-trained transformer model BERT
(?) to extract utterance level text features, (many other approaches use Glove
word-level embeddings followed by a LSTM). The motivation behinds using BERT
is 1) BERT is the state-of-the-art in sentence encoding algorithms and has
demonstrated tremendous success in several downstream text applications such
as sentiment analysis, question-answering, semantic similarity tasks etc, 2)
using BERT simplifies the training pipe-line, with a large focus now towards
improving the performance of ICCN on a particular downstream task. We input
the raw text to the pre-trained uncased BERT-Base model (without fine-tuning).
Sentence encodings output from BERT are used as the text features. Each
individual text feature is of size 768. For IEMOCAP, we used InferSent(?) to
encode utterance level text. Since data is provided in the form of word
indices for GLOVE embeddings rather than raw text, we use InferSent; a BiLSTM
layer followed by a max-pooling layer to learn sentence embeddings.

  * •

Audio Features: The audio feature is extracted by using COVAREP (?), which is
a public software used for extracting acoustic features such as pitch, volume,
and frequency. The CMU-MultimodalSDK provides COVAREP feature sequence for
every multi-modal example, the dimension of each frame’s audio feature is 74.

  * •

Video Features: Facet(?) has been used for extracting facial expression
features such as action units and face pose. Similarly, every multimodal
example’s video feature sequence is also obtained from the CMU-MultimodalSDK.
The size of each frame’s video feature is 35.

### Baseline Methods

We consider a variety of baseline methods for multi-modal embedding
comparison. In order to focus on the multi-modal embedding itself, we input
each multi-modal embedding to the same downstream task classifier (or
regressor). Experimental comparisons are reported in two parts 1) we report
the effectiveness of DCCA over the simpler CCA based methods when used as
inputs to the ICCN and 2) we compare ICCN against newer utterance level
embeddings algorithms that learn features for a down stream task in an end-to-
end fashion. The following baselines are used in our comparisons,

  * •

Uni-modal and Concatenation: This is the simplest baseline in which uni-modal
features are concatenated to obtain a multi-modal embedding.

  * •

Linear CCA: CCA (?) considers linear transformations for different inputs. We
use the CCA to learn a new common space for audio and video modes, and combine
the learned audio and video features with the original text embedding. This is
because, (?) showed that using a CCA-based method to correlate audio and video
is more effective that correlating audio-text or video-text.

  * •

Kernel-CCA: Kernel-CCA (?) introduces a nonlinearity via kernel maps. Kernel-
CCA can be used exactly like CCA.

  * •

GCCA: Generalized CCA (?) learns a common subspace across multiple views. We
use GCCA in two ways: 1) use the GCCA output embedding directly and 2) combine
the GCCA output embedding with the original text embedding.

  * •

DCCA: A Deep CCA based algorithm proposed by (?). Audio and video features are
simply concatenated and then be correlated with text features using DCCA.
Outputs of the DCCA are again concatenated with raw text, audio, and video
features to formulate the multimodal embedding.

In the proposed ICCN algorithm, text features are encoded by a pre-trained
BERT transformer or by InferSent. This is unlike most of the state-of-the-art
algorithms that obtain sentence level encodings by passing word embeddings
through variants of RNNs. However, since the idea is to compare modal
features, we also choose the following three state-of-the-art utterance-level
(i.e. sentence-level) fusion models (whose core algorithm is agnostic to the
text encoding architecture) as additional baselines. To make the comparison
fair, these methods use the same features as ours.

  * •

TFN: Tensor Fusion Network (TFN) (?) combines individual modal’s embeddings
via calculating three different outer-product sub-tensors: unimodal, bimodal,
and trimodal. All tensors will then be flattened and used as a multi-modal
embedding vector.

  * •

LMF: Low-Rank Multimodal Fusion (LMF) (?) learns the multimodal embedding
based the similar tensor processing of TFN, but with an additional low-rank
factor for reducing computation memory.

  * •

MFM: Multimodal Factorization Model (MFM) (?) is consists of a discriminative
model for prediction and a generative model for reconstructing input data. A
comprehensive multimodal embedding is learned via optimizing the generative-
discriminative objective simultaneously.

Data View | CMU-MOSI | CMU-MOSEI
---|---|---
| Acc-2 | F-score | MAE | Acc-7 | Corr | Acc-2 | F-score | MAE | Acc-7 | Corr
Audio | 45.15 | 45.83 | 1.430 | 16.21 | 0.248 | 58.75 | 59.23 | 0.785 | 38.59 | 0.298
Video | 48.10 | 49.06 | 1.456 | 15.51 | 0.339 | 59.25 | 59.90 | 0.770 | 36.09 | 0.288
Text | 80.80 | 80.17 | 0.897 | 35.92 | 0.688 | 82.83 | 83.02 | 0.582 | 48.76 | 0.681
Text+Video | 81.00 | 80.91 | 0.920 | 35.11 | 0.676 | 82.86 | 83.01 | 0.581 | 47.92 | 0.674
Text+Audio | 80.59 | 80.56 | 0.909 | 35.08 | 0.672 | 82.80 | 82.96 | 0.582 | 49.02 | 0.689
Audio+Video+Text | 80.94 | 81.00 | 0.895 | 36.41 | 0.689 | 82.72 | 82.87 | 0.583 | 50.11 | 0.692
CCA | 79.45 | 79.35 | 0.893 | 34.15 | 0.690 | 82.94 | 83.06 | 0.573 | 50.23 | 0.690
KCCA | 79.82 | 79.91 | 0.889 | 34.76 | 0.689 | 83.05 | 83.14 | 0.574 | 50.09 | 0.692
GCCA | 62.50 | 62.15 | 1.403 | 17.29 | 0.533 | 75.12 | 75.46 | 0.653 | 45.33 | 0.602
GCCA+Text | 77.80 | 77.87 | 1.107 | 25.94 | 0.658 | 82.75 | 82.90 | 0.613 | 46.06 | 0.644
DCCA | 80.60∗ | 80.57∗ | 0.874 | 35.51 | 0.703 | 83.62∗ | 83.75∗ | 0.579 | 50.12 | 0.707
TFN | 80.82 | 80.77 | 0.901 | 34.94 | 0.698 | 82.57 | 82.09 | 0.593 | 50.21 | 0.700
LMF | 82.53 | 82.47 | 0.917 | 33.23 | 0.695 | 82.03 | 82.18 | 0.623 | 48.02 | 0.677
MFM | 81.72 | 81.64 | 0.877 | 35.42 | 0.706 | 84.40 | 84.36 | 0.568 | 51.37 | 0.717
ICCN | 83.07 | 83.02 | 0.862 | 39.01 | 0.714 | 84.18 | 84.15 | 0.565 | 51.58 | 0.713

Table 1: Results for experiments on CMU-MOSI and CMU-MOSEI. Best numbers are
in bold. For accuracy, F-score, and Correlation, higher is better. For mean
absolute error, lower is better. Results marked with $*$ are reported in
original papers. For TFN, LMF, and MFM, we re-did experiments with using our
features for a fair comparison.

### Ablation studies of ICCN

In order to analyze the usefulness of different components of the ICCN, we
consider the following two questions:

Question 1: Is using canonical correlation better than using other methods
like Cosine-Similarity?

Question 2: Is learning the interactions between text and video (or audio)
useful?

We design several experiments to address these two questions, First, we
replace the CCA Loss with Cosine-Similarity Loss while leaving other parts of
the ICCN unchanged. Second, instead of using the outer-product of audio (or
video) and text as input to the CCA Loss, we use audio and video directly as
the input to DCCA network. We compare different ICCN variants’ performance to
prove the usefulness of each component of the network.

### Evaluation Methods

To evaluate ICCN against previous baselines, the following performance metrics
as described in (?; ?; ?) are reported,

  * •

On the CMU-MOSI and CMU-MOSEI we report four performance metrics, i) binary
accuracy, ii) F1-score iii) mean absolute error and iv) 7-class sentiment
level / Correlation with human labeling.

  * •

On the IEMOCAP we used i) binary accuracy and ii) F1-score for evaluation.

Evaluations Details on CMU-MOSI, CMU-MOSEI: The original MOSI and MOSEI
datasets are labeled in the range [-3,3]. The author of the datasets suggests
a criterion for building binary labels: examples with label in [-3, 0) are
considered to have negative sentiment while examples with label in (0, 3] are
considered to have positive sentiment. 7-class sentiment level is also
calculated based on the label distribution in [-3,3]. The correlation of
predicted results with human labeling is also used as a criteria.

### Hyperparameter Tuning

A basic Grid-Search is used to tune hyperparameters, and the best
hyperparameter settings for the ICCN are chosen according to its performance
on the validation dataset. For ICCN, hyperparameters and tuning ranges are:
learning rate ($1e-5$–$1e-3$), mini-batch size ($128$–$512$), the number of
epoch ($10$–$100$), hidden dimensions of MLP ($64$–$512$), and output
dimension of the CCA projection ($30$–$100$). ReLU is used as the activation
function, RMSProp is used as the optimizer.

Whenever the training of the ICCN with a specific hypyerparameter setting has
finished, features learned from the ICCN are used as input to the same
downstream task models (a simple MLP is used in this work). Test results are
reported by using the best hyperparameter setting learned above.

Data View | IEMOCAP
---|---
Emotions | Happy | Angry | Sad | Neutral
| Acc-2 | F-score | Acc-2 | F-score | Acc-2 | F-score | Acc-2 | F-score
Audio | 84.03 | 81.09 | 85.49 | 84.03 | 82.75 | 80.26 | 63.08 | 59.24
Video | 83.14 | 80.36 | 85.91 | 83.27 | 81.19 | 80.35 | 62.30 | 58.19
Text | 84.80 | 81.17 | 85.12 | 85.21 | 84.18 | 83.63 | 66.02 | 63.52
Text+Video | 85.32 | 82.01 | 85.22 | 85.10 | 83.33 | 82.96 | 65.82 | 65.81
Text+Audio | 85.10 | 83.47 | 86.09 | 84.99 | 83.90 | 83.91 | 66.96 | 65.02
Audio+Video+Text | 86.00 | 83.37 | 86.37 | 85.88 | 84.02 | 83.71 | 66.87 | 65.93
CCA | 85.91 | 83.32 | 86.17 | 84.39 | 84.19 | 83.71 | 67.22 | 64.84
KCCA | 86.54 | 84.08 | 86.72 | 86.32 | 85.03 | 84.91 | 68.29 | 65.93
GCCA | 81.15 | 80.33 | 82.47 | 78.06 | 83.22 | 81.75 | 65.34 | 59.99
GCCA+Text | 87.02 | 83.44 | 88.01 | 88.00 | 84.79 | 83.26 | 68.26 | 67.61
DCCA | 86.99 | 84.32 | 87.94 | 87.85 | 86.03 | 84.36 | 68.87 | 65.93
TFN | 86.66 | 84.03 | 87.11 | 87.03 | 85.64 | 85.75 | 68.90 | 68.03
LMF | 86.14 | 83.92 | 86.24 | 86.41 | 84.33 | 84.40 | 69.62 | 68.75
MFM | 86.67 | 84.66 | 86.99 | 86.72 | 85.67 | 85.66 | 70.26 | 69.98
ICCN | 87.41 | 84.72 | 88.62 | 88.02 | 86.26 | 85.93 | 69.73 | 68.47

Table 2: Results for experiments on IEMOCAP. Best numbers are in bold. For
accuracy and F-score, higher is better. For TFN, LMF, and MFM, we re-did
experiments with using our features for a fair comparison.

Data View | CMU-MOSI | CMU-MOSEI
---|---|---
| Acc-2 | F-score | MAE | Acc-7 | Corr | Acc-2 | F-score | MAE | Acc-7 | Corr
ICCN | 83.07 | 83.02 | 0.862 | 39.01 | 0.714 | 84.18 | 84.15 | 0.565 | 51.58 | 0.713
ICCN1(no text) | 82.13 | 82.05 | 0.874 | 35.51 | 0.703 | 83.01 | 83.10 | 0.575 | 50.12 | 0.707
ICCN2(cos) | 82.32 | 82.27 | 0.876 | 36.01 | 0.702 | 82.98 | 82.90 | 0.575 | 50.63 | 0.700
ICCN3(no text + cos) | 81.49 | 81.58 | 0.889 | 35.77 | 0.692 | 82.59 | 82.73 | 0.578 | 50.21 | 0.696

Table 3: Ablation studies of ICCN on CMU-MOSI and CMU-MOSEI. ICCN1-3 denote
different variants, “no text” means applying DCCA to audio and video directly
instead of applying to the outer-product with text; “cos” means replacing CCA
Loss with Cosine-Similarity Loss;“no text + cca” means removing outer-product
with text and use Cosine-Similarity Loss.

##  5 Discussion of Empirical Results

This section presents and discusses results on the CMU-MOSI, CMU-MOSEI, and
IEMOCAP datasets.

### Performance on Benchmark Datasets

Tables 1 and 2 present results of experiments on the CMU-MOSI, CMU-MOSEI, and
IEMOCAP datasets.

  * •

First, when compared with results of using uni-modal and simple
concatenations, ICCN outperforms all of them in all of the criteria. Note that
the performance of the text feature is always better than the performance of
the audio and video, and that a simple concatenation of text, video, and audio
does not work well. This shows the advance of highly-developed pre-trained
text features capable of improving the overall multimodal task performance.
However, it also shows the challenge of how to effectively combine such a
highly developed text feature with audio and video features.

  * •

Second, ICCN also outperforms other CCA-based methods. The results of using
other CCA-based methods show that they cannot improve the multimodal
embedding’s performance. We argue this occurs because 1) CCA / KCCA / GCCA do
not exploit the power of neural network architectures so that their learning
capacities are limited. 2) Using DCCA without learning the interactions
between text-based audio and text-based video may sacrifice useful
information.

  * •

Third, the ICCN still achieves better or similar results when compared with
other neural network based state-of-the-art methods (TFN, LMF, and MFM). These
results demonstrate the ICCN’s competitive performance.

### Results of Ablation Studies

Table 3 shows results of using variants of ICCN on CMU-MOSI and CMU-MOSEI
datasets.

First, using the CCA Loss performs better than using Cosine-Similarity Loss
with or without the outer-product. This is reasonable as the DCCA is able to
learn the hidden relationships (with the help of non-linear transformations)
but cosine-similarity is restrained by the original coordinates. To further
verify this, we also record changes of canonical correlation and cosine
similarity between text-based audio and text-based video (i.e., between the
two outputs of the CNNs in the ICCN) by using CCA Loss or Cosine Similarity
Loss for the ICCN with using the CMU-MOSI dataset. Curves in Figures 2 and 3
summarize the results of the experiments. Results show that maximizing the
canonical correlation by using the CCA Loss does not necessarily increase the
cosine similarity, and vice versa. This demonstrates that the canonical
correlation is a genuinely different objective function than cosine
similarity, and explains the different behaviors in the downstream
applications. CCA is capable of learning hidden relationships between inputs
that the cosine similarity does not see.

Second, learning the interactions between non-text and text performs better
than using audio and video directly. This also makes sense because audio and
video are more correlated when they are based on the same text, thus learning
text-based audio and text-based video performs better. In summary, Table 3
shows the usefulness of using a text-based outer-product together with DCCA.

Figure 2: Changes of mean canonical correlation and mean cosine similarity
between text-based audio and text-based video when training with CCA Loss: The
network learns to maximize the canonical correlation but the cosine similarity
isn’t affected. Figure 3: Changes of mean canonical correlation and cosine
similarity between text-based audio and text-based video when training with
Cosine Similarity Loss: convergence of cosine similarity doesn’t affect
canonical correlation.

##  6 Conclusion and Future Work

This paper has proposed the ICCN method, which uses canonical correlation to
analyze hidden relationships between text, audio, and video. Testing on a
multi-modal sentiment analysis and emotion recognition task shows that the
multi-modal features learned from the ICCN model can achieve state-of-the-art
performance, and shows the effectiveness of the model. Ablation studies
confirm the usefulness of different part of the network.

There is, of course, considerable room for improvement. Possible directions
include learning dynamic intra-actions in each model together with inter-
actions between different modes; learning the trade-off between maximum
canonical correlation and best downstream task performance; and developing an
interpretable end-to-end multi-modal canonical correlation model. In the
future, we hope to move forward in the development of multi-modal machine
learning.

## References

  * [Akaho 2006] Akaho, S.  2006\.  A kernel method for canonical correlation analysis.  arXiv preprint cs/0609071.
  * [Andrew et al. 2013] Andrew, G.; Arora, R.; Bilmes, J.; and Livescu, K.  2013\.  Deep canonical correlation analysis.  In ICML, 1247–1255.
  * [Busso et al. 2008] Busso, C.; Bulut, M.; Lee, C.-C.; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, J. N.; Lee, S.; and Narayanan, S. S.  2008\.  Iemocap: Interactive emotional dyadic motion capture database.  Language resources and evaluation 42(4):335.
  * [Chen et al. 2017] Chen, M.; Wang, S.; Liang, P. P.; Baltrušaitis, T.; Zadeh, A.; and Morency, L.-P.  2017\.  Multimodal sentiment analysis with word-level fusion and reinforcement learning.  In ICMI, 163–171.  ACM.
  * [Conneau et al. 2017] Conneau, A.; Kiela, D.; Schwenk, H.; Barrault, L.; and Bordes, A.  2017\.  Supervised learning of universal sentence representations from natural language inference data.  arXiv preprint arXiv:1705.02364.
  * [Degottex et al. 2014] Degottex, G.; Kane, J.; Drugman, T.; Raitio, T.; and Scherer, S.  2014\.  Covarep—a collaborative voice analysis repository for speech technologies.  In ICASSP, 960–964.  IEEE.
  * [Devlin et al. 2018] Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Gers, Schmidhuber, and Cummins 1999] Gers, F. A.; Schmidhuber, J.; and Cummins, F.  1999\.  Learning to forget: Continual prediction with lstm.  ICANN.
  * [Haq and Jackson 2011] Haq, S., and Jackson, P. J.  2011\.  Multimodal emotion recognition.  In MAPAS. IGI Global.  398–423.
  * [Hazarika et al. 2018] Hazarika, D.; Poria, S.; Gorantla, S.; Cambria, E.; Zimmermann, R.; and Mihalcea, R.  2018\.  Cascade: Contextual sarcasm detection in online discussion forums.  arXiv preprint arXiv:1805.06413.
  * [Hotelling 1936] Hotelling, H.  1936\.  Relations between two set of variables.  Biometrika 28(3-4):321–377.
  * [iMotions 2017] iMotions.  2017\.  Facial expression analysis.  imotions.com.
  * [Lawrence et al. 1997] Lawrence, S.; Giles, C. L.; Tsoi, A. C.; and Back, A. D.  1997\.  Face recognition: A convolutional neural-network approach.  IEEE transactions on neural networks 8(1):98–113.
  * [Lee et al. 2014] Lee, G.; Singanamalli, A.; Wang, H.; Feldman, M. D.; Master, S. R.; Shih, N. N.; Spangler, E.; Rebbeck, T.; Tomaszewski, J. E.; and Madabhushi, A.  2014\.  Supervised multi-view canonical correlation analysis (smvcca): Integrating histologic and proteomic features for predicting recurrent prostate cancer.  IEEE transactions on medical imaging 34(1):284–297.
  * [Liang et al. 2018] Liang, P. P.; Liu, Z.; Zadeh, A.; and Morency, L.-P.  2018\.  Multimodal language analysis with recurrent multistage fusion.  arXiv preprint arXiv:1808.03920.
  * [Liu et al. 2018] Liu, Z.; Shen, Y.; Lakshminarasimhan, V. B.; Liang, P. P.; Zadeh, A.; and Morency, L.-P.  2018\.  Efficient low-rank multimodal fusion with modality-specific factors.  ACL.
  * [Liu et al. 2019] Liu, F.; Guan, B.; Zhou, Z.; Samsonov, A.; G Rosas, H.; Lian, K.; Sharma, R.; Kanarek, A.; Kim, J.; Guermazi, A.; and Kijowski, R.  2019\.  Fully-automated diagnosis of anterior cruciate ligament tears on knee mr images using deep learning.  Radiology 1\.
  * [Martin and Maes 1979] Martin, N., and Maes, H.  1979\.  Multivariate analysis.  Academic press London.
  * [Morency, Mihalcea, and Doshi 2011] Morency, L.-P.; Mihalcea, R.; and Doshi, P.  2011\.  Towards Multimodal Sentiment Analysis: Harvesting Opinions from The Web.  In ICMI 2011.
  * [Poria et al. 2016] Poria, S.; Chaturvedi, I.; Cambria, E.; and Hussain, A.  2016\.  Convolutional mkl based multimodal emotion recognition and sentiment analysis.  In ICDM, 439–448.  IEEE.
  * [Rotman, Vulić, and Reichart 2018] Rotman, G.; Vulić, I.; and Reichart, R.  2018\.  Bridging languages through images with deep partial canonical correlation analysis.  In ACL(Volume 1: Long Papers), 910–921.
  * [Sarma, Liang, and Sethares 2018] Sarma, P. K.; Liang, Y.; and Sethares, W. A.  2018\.  Domain adapted word embeddings for improved sentiment classification.  arXiv preprint arXiv:1805.04576.
  * [Soleymani et al. 2017] Soleymani, M.; Garcia, D.; Jou, B.; Schuller, B.; Chang, S.-F.; and Pantic, M.  2017\.  A survey of multimodal sentiment analysis.  Image and Vision Computing 65:3–14.
  * [Sun et al. 2019] Sun, Z.; Sarma, P. K.; Sethares, W.; and Bucy, E. P.  2019\.  Multi-modal sentiment analysis using deep canonical correlation analysis.  arXiv preprint arXiv:1907.08696.
  * [Tenenhaus and Tenenhaus 2011] Tenenhaus, A., and Tenenhaus, M.  2011\.  Regularized generalized canonical correlation analysis.  Psychometrika 76(2):257.
  * [Tsai et al. 2018] Tsai, Y.-H. H.; Liang, P. P.; Zadeh, A.; Morency, L.-P.; and Salakhutdinov, R.  2018\.  Learning factorized multimodal representations.  arXiv preprint arXiv:1806.06176.
  * [Tsai et al. 2019] Tsai, Y.-H. H.; Bai, S.; Liang, P. P.; Kolter, J. Z.; Morency, L.-P.; and Salakhutdinov, R.  2019\.  Multimodal transformer for unaligned multimodal language sequences.  arXiv preprint arXiv:1906.00295.
  * [Wang et al. 2018] Wang, Y.; Shen, Y.; Liu, Z.; Liang, P. P.; Zadeh, A.; and Morency, L.-P.  2018\.  Words can shift: Dynamically adjusting word representations using nonverbal behaviors.  arXiv preprint arXiv:1811.09362.
  * [Wang, Li, and Lazebnik 2016] Wang, L.; Li, Y.; and Lazebnik, S.  2016\.  Learning deep structure-preserving image-text embeddings.  In CVPR, 5005–5013.
  * [Zadeh et al. 2016] Zadeh, A.; Zellers, R.; Pincus, E.; and Morency, L.-P.  2016\.  Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos.  arXiv preprint arXiv:1606.06259.
  * [Zadeh et al. 2017] Zadeh, A.; Chen, M.; Poria, S.; Cambria, E.; and Morency, L.-P.  2017\.  Tensor fusion network for multimodal sentiment analysis.  arXiv preprint arXiv:1707.07250.
  * [Zadeh et al. 2018a] Zadeh, A.; Liang, P. P.; Mazumder, N.; Poria, S.; Cambria, E.; and Morency, L.-P.  2018a.  Memory fusion network for multi-view sequential learning.  In AAAI.
  * [Zadeh et al. 2018b] Zadeh, A.; Liang, P. P.; Poria, S.; Vij, P.; Cambria, E.; and Morency, L.-P.  2018b.  Multi-attention recurrent network for human communication comprehension.  In AAAI.
  * [Zadeh et al. 2018c] Zadeh, A. B.; Liang, P. P.; Poria, S.; Cambria, E.; and Morency, L.-P.  2018c.  Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph.  In ACL, 2236–2246.


# Quantum Gravity in Flat Spacetime

Jarmo Mäkelä111Vaasa University of Applied Sciences, Wolffintie 30, 65200
Vaasa, Finland, email<EMAIL_ADDRESS>
###### Abstract

Inspired by Einstein’s Strong Principle of Equivalence we consider the effects
of quantum mechanics to the gravity-like phenomena experienced by an observer
in a uniformly accelerating motion in flat spacetime. Among other things, our
model of quantum gravity, derived from the first principles, predicts the
Unruh effect, and a discrete area spectrum for spacelike two-surfaces.

##  1 Introduction

When Albert Einstein began to formulate his general theory of relativity, he
used as a starting point his Strong Principle of Equivalence. In broad terms,
this principle states that it is impossible to decide, by means of local
measurements, whether one is in a gravitational field or in an accelerating
frame of reference. In this sense the effects of gravity are equivalent with
those of acceleration. [1, 2] For instance, if an observer finds all bodies in
his surroundings to fall with the same acceleration, the observer has no way
to tell, whether he is in a uniform gravitational field, caused by the
curvature of spacetime, or in a uniformly accelerating motion in flat
spacetime.

One of the greatest challenges of modern physics is to create the quantum
theory of gravity, which is supposed to bring together general relativity and
quantum mechanics. It appears to the author that in the attempts to quantize
gravity developed so far the Equivalence Principle has received insufficient
attention. It is possible that we should take the Equivalence Principle as the
starting point of quantum gravity in the same way as Einstein took it as the
starting point of his general relativity.

Inspired by these thoughts we shall, in this paper, quantize the effects of
gravity in flat spacetime from the point of view of an observer in a uniformly
accelerating motion. According to the Equivalence Principle quantization of
gravity in curved spacetime should, at least locally, be equivalent with the
quantization of gravity in flat spacetime from the point of view of an
acccelerating observer. We shall see that the resulting ”quantum gravity in
flat spacetime” may indeed be developed systematically, beginning from the
first principles.

In our approach we consider the gravity-like effects on a spacelike two-plane
at rest with respect to our accelerating observer, and it plays a role
somewhat similar to that of the event horizon of a black hole. The standard
rules of quantum mechanics applied to our model imply that the accelerating
plane has a discrete structure in the sense that it consists of a finite
number of separate constituents, all of them having an equally spaced area
spectrum. It also predicts the Unruh effect, according to which an observer in
an accelerating motion detects thermal radiation with a characteristic
temperature, which is proportional to the proper acceleration of the observer.

Unless otherwise stated, we shall always use the natural units, where
$\hbar=G=c=k_{B}=1$.

##  2 Action

Whenever we attempt to quantize gravity, we may use the classical Einstein-
Hilbert action

$S_{EH}:=\frac{1}{16\pi}\int_{M}R\sqrt{-g}\,d^{4}x$ (2.1)

as the starting point. In this equation $R$ is the Riemann curvature scalar,
and $g$ is the determinant of the metric tensor of spacetime. We have
integrated over the whole spacetime $M$. However, if the spacetime region
under consideration possesses a boundary, we must supplement the Einstein-
Hilbert action with the Gibbons-Hawking boundary term [3]

$S_{GH}:=\frac{1}{8\pi}\int_{\partial M}K\,dV,$ (2.2)

where $K$ is the trace of the exterior curvature tensor induced on the
boundary $\partial M$ of spacetime. $dV$ is the volume element on the
boundary, and we have integrated over the whole boundary. So the whole action
takes the form:

$S=S_{EH}+S_{GH}=\frac{1}{16\pi}\int_{M}R\sqrt{-g}\,d^{4}x+\frac{1}{8\pi}\int_{\partial
M}K\,dV.$ (2.3)

In flat spacetime the Riemann curvature scalar $R\equiv 0$, and we are left
with the Gibbons-Hawking boundary term only. Hence the action becomes to:

$S=\frac{1}{8\pi}\int_{\partial M}K\,dV.$ (2.4)

Even if spacetime were flat, the action may still be non-zero, if the boundary
of spacetime has been chosen appropriately. Because of the presence of the
potentially non-zero boundary term, the quantum-mechanical properties of the
gravity-like effects caused by a choice of a non-inertial frame of reference
are far from trivial.

In this paper we first pick up from the four-dimensional flat spacetime an
inertial frame of reference, where each point of spacetime is identified by
means of four coordinates $(T,X,Y,Z)$. The coordinate $T$ is timelike, and the
coordinates $X$, $Y$ and $Z$ spacelike. When written in terms of these
coordinates the line element of spacetime takes the flat Minkowski form:

$ds^{2}=-dT^{2}+dX^{2}+dY^{2}+dZ^{2}.$ (2.5)

We choose the boundary of spacetime to consist of five timelike hyperplanes,
where $X=X_{1}$, $Y=Y_{1}$, $Y=Y_{2}$, $Z=Z_{1}$ and $Z=Z_{2}$ such that
$Y_{2}>Y_{1}$ and $Z_{2}>Z_{1}$. In other words, the boundary of spacetime is
a three-dimensional, rectangular box, which proceeds in time, and where the
faces stay still in our frame of reference.

One of the faces of the box, however, is not assumed to stay still, but it is
assumed to accelerate with constant proper acceleration $a$ to the direction
of the positive $X$-axis. As a consequence, the $X$-coordinates of the ponts
of the face satisfy an equation:

$X^{2}-T^{2}=\frac{1}{a^{2}},$ (2.6)

which implies:

$X=\sqrt{T^{2}+\frac{1}{a^{2}}}.$ (2.7)

So the boundary of spacetime consists, as a whole, of five timelike
hyperplanes, and of a one timelike hypersurface, which is created, when a
spacelike two-plane accelerates along the $X$-axis.

The exterior curvature of all of the timelike hyperplanes is zero, and so they
do not contribute to the action. The exterior curvature of the hypersurface
created by the accelerating plane, however, is non-zero. To calculate the
trace of the exterior curvature tensor on that hypersurface we introduce the
Rindler coordinates to our spacetime. As it is well known, between the Rindler
coordinates $(t,x)$ and the coordinates $(T,X)$ there is the relationship: [4]

$\displaystyle t$ $\displaystyle=$
$\displaystyle\tanh^{-1}\left(\frac{T}{X}\right),$ (2.8a) $\displaystyle x$
$\displaystyle=$ $\displaystyle\sqrt{X^{2}-T^{2}}.$ (2.8b)

When written in terms of the Rindler coordinates, together with the
coordinates $Y$ and $Z$, the line element of spacetime takes the form:

$ds^{2}=-x^{2}\,dt^{2}+dx^{2}+dY^{2}+dZ^{2}.$ (2.9)

An observer with constant Rindler coordinate $x$, together with constant
coordinates $Y$ and $Z$, has constant proper acceleration $a$, which is
related to $x$ such that

$x=\frac{1}{a},$ (2.10)

which follows from Eqs. (2.6) and (2.8b). Eq. (2.8 a), in turn, implies that
the Rindler time coordinate $t$ gives the boost angle $\phi$ of an observer
with constant proper acceleration.

On the timelike hypersurface created, when a plane is in an accelerating
motion with constant proper acceleration $a$ the Rindler coordinate $x$ is a
constant. The only non-zero component of the exterior curvature tensor on that
hypersurface is:

$K_{tt}=\Gamma_{tt}^{x}=x,$ (2.11)

and therefore its trace is:

$K=g^{tt}K_{tt}=-\frac{1}{x}.$ (2.12)

The volume element on our hypersurface is:

$dV=x\,dY\,dZ\,dt,$ (2.13)

and so we find that the action in Eq. (2.7) takes, if the Rindler time
coordinate $t$ lies within an interval $[t_{i},t_{f}]$, the form:

$S=-\frac{1}{8\pi}A(t_{f}-t_{i}),$ (2.14)

where $A$ is the area of the accelerating plane. [5]

##  3 Hamiltonian

The bridge from the action to the Hamiltonian $H(q_{j},p_{j};t)$ of any system
is formed by the Hamilton-Jacobi equation:

$H(q_{j},\frac{\partial S}{\partial q_{j}};t)+\frac{\partial S}{\partial
t}=0,$ (3.1)

where $S$ is the principal function of the system. The quantities $q_{j}$ are
the coordinates of the configuration space, and the quantities
$p_{j}=\frac{\partial S}{\partial q_{j}}$ the corresponding coordinates of the
momentum space. The principal function of the system has been denoted by the
same symbol as its action for a very good reason: As it is well known, the
principal function agrees with the action integral calculated along the
classical path. Hence it follows that if we know the action $S$ calculated
along the classical path, the Hamiltonian of the system will be: [6]

$H=-\frac{\partial S}{\partial t}.$ (3.2)

Our derivation of the classical Hamiltonian of the gravitational field is
based on Eq. (3.2). Before proceeding to the use of Eq. (3.2), however, we
must decide, which time coordinate to use. The very idea of this paper is to
consider the effects of gravity from the point of view of a uniformly
accelerating observer. It is therefore sensible to take the proper time $\tau$
of the observer as the time coordinate. Eq. (2.9) implies that between the
proper time $\tau$ and the Rindler time $t$ there is the relationship:

$d\tau=x\,dt.$ (3.3)

Writing

$H=-\frac{\partial S}{\partial\tau},$ (3.4)

and using Eqs. (2.10) and (2.14) we therefore find:

$H=\frac{a}{8\pi}A.$ (3.5)

Eq. (3.5) gives the classical Hamiltonian of the gravitational field from the
point of view of an observer moving along a plane with area $A$ and proper
acceleration $a$. Remarkably, the Hamiltonian in Eq. (3.5) is identical to the
one obtained in several papers in spacetimes with horizon, from the point of
view of an observer with constant proper acceleration $a$, just outside of a
horizon with area $A$. [7-14] When the proper acceleration $a$ of the observer
tends to infinity, Eq. (2.10) implies that the Rindler coordinate $x$ tends to
zero, which means that in this limit our plane tends to a Rindler horizon of
flat spacetime.

We may always define a diffeoforphism $f$ from the unit square
$I^{2}:=[0,1]\times[0,1]$ to the accelerating plane, which we shall henceforth
denote by $B$. In general, this map $f:I^{2}\longrightarrow B$ is of the form:

$f(\chi^{1},\chi^{2})=(Y(\chi^{1},\chi^{2}),Z(\chi^{1},\chi^{2}))$ (3.6)

for all $\chi^{1},\chi^{2}\in[0,1]$, where the coordinates
$Y(\chi^{1},\chi^{2})$ and $Z(\chi^{1},\chi^{2})$ of the point $(Y,Z)$ on the
plane $B$ are functions of the coordinates $\chi^{1}$ and $\chi^{2}$ of the
point $\chi=(\chi^{1},\chi^{2})$ in $I^{2}$. Defining the function

$p(\chi):=p(\chi^{1},\chi^{2}):=\frac{1}{8\pi}\left|\begin{matrix}\frac{\partial
Y(\chi^{1},\chi^{2})}{\partial\chi^{1}}&\frac{\partial
Y(\chi^{1},\chi^{2})}{\partial\chi^{2}}\\\ \frac{\partial
Z(\chi^{1},\chi^{2})}{\partial\chi^{1}}&\frac{\partial
Z(\chi^{1},\chi^{2})}{\partial\chi^{2}}\end{matrix}\right|$ (3.7)

we may write the area of the plane $B$ as:

$A=8\pi\int_{I^{2}}p(\chi)\,d^{2}\chi,$ (3.8)

where we have integrated over the unit square. As a consequence, the
Hamiltonian of the gravitational field takes, from the point of view of our
accelerating observer, the form:

$H=a\int_{I^{2}}p(\chi)\,d^{2}\chi.$ (3.9)

It must be emphasized that the proper acceleration $a$ is not a dynamical
variable of the system, but just a parameter, which determines the observer.

We shall now take the quantities $p(\chi)$ associated with the points $\chi$
as the coordinates of the momentum space. The corresponding coordinates of the
configuration space we shall denote by $q(\chi)$. They obey the Hamiltonian
equations of motion:

$\dot{q}(\chi)=\frac{\delta H}{\delta p(\chi)}=a,$ (3.10)

where the dot means the derivative with respect to the proper time $\tau$ of
the accelerating observer. The general solution to Eq. (3.10) is:

$q(\chi)=a\tau+C(\chi),$ (3.11)

where $C(\chi)$ is an arbitrary constant function of the proper time $\tau$,
and depends on the coordinates $(\chi^{1},\chi^{2})$ of the point $\chi$ only.
Eqs. (2.10) and (3.3) imply that the quantity $a\tau$ may be identified as the
boost angle $\phi$ of the observer. Hence we may write Eq. (3.11) as:

$q(\chi)=\phi+C(\chi).$ (3.12)

Since our Hamiltonian in Eq. (3.10) does not involve $q(\chi)$, the
Hamiltonian equations of motion written for the variables $p(\chi)$ read as:

$\dot{p}(\chi)=-\frac{\delta H}{\delta q(\chi)}=0,$ (3.13)

which means that the quantities $p(\chi)$ are constants of motion of the
system.

##  4 Quantization

When we go over from the classical to the quantum-mechanical consideration of
the gravitational field in flat spacetime we must replace the classical
quantities $q(\chi)$ and $p(\chi)$ by the corresponding operators
$\hat{q}(\chi)$ and $\hat{p}(\chi)$ obeying the canonical commutation
relations:

$\displaystyle[\hat{q}(\chi),\hat{q}(\chi^{\prime})]=[\hat{p}(\chi),\hat{p}(\chi^{\prime})]$
$\displaystyle=$ $\displaystyle 0,$ (4.1a)
$\displaystyle[\hat{q}(\chi),\hat{p}(\chi^{\prime})]$ $\displaystyle=$
$\displaystyle i\delta^{2}(\chi,\chi^{\prime})$ (4.1b)

at all points $\chi,\chi^{\prime}\in I^{2}$. In Eq. (4.1b)
$\delta^{2}(\chi,\chi^{\prime})$ is the two-dimensional delta function in
$I^{2}$. When written in terms of these operators the classical Hamltonian $H$
in Eq. (3.9) becomes replaced by the corresponding Hamiltonian operator

$\hat{H}=a\int_{I^{2}}\hat{p}(\chi)\,d^{2}\chi,$ (4.2)

and therefore the time-independent Schrödinger equation

$\hat{H}|\psi\rangle=E|\psi\rangle$ (4.3)

written for the energy eigenstates $|\psi\rangle$ of the gravitational field
takes the form:

$a\int_{I^{2}}\hat{p}(\chi)|\psi\rangle=E|\psi\rangle.$ (4.4)

Identifying the energy $E$ of the gravitational field with the classical
Hamiltonian $H$ in Eq. (3.9) we observe that this equation reduces to:

$\int_{I^{2}}\hat{p}(\chi)|\psi\rangle=\int_{I^{2}}p(\chi)|\psi\rangle.$ (4.5)

Eq. (4.5) has an obvious solution, where $|\psi\rangle$ is the common
eigenstate of all of the operators $\hat{p}(\chi)$, which means that:

$\hat{p}(\chi)|\psi\rangle=p(\chi)|\psi\rangle$ (4.6)

for all $\chi\in I^{2}$. If the eigenstate $|\psi\rangle$ is normed to unity,
we must have:

$\langle\psi|\psi\rangle=1.$ (4.7)

We shall take Eqs. (4.6) and (4.7) as the postulates of our model, and see,
where this will take us.

In the position representation the energy eigenstates $|\psi\rangle$ are
expressed as functionals $\psi[q(\chi)]$ of the functions $q(\chi)$, and they
may be regarded as the wave functions of the gravitational field. The inner
product between the wave functions $\psi_{1}[q(\chi)]$ and $\psi_{2}[q(\chi)]$
is defined as:

$\langle\psi_{1}|\psi_{2}\rangle:=\int\psi_{1}^{*}[q(\chi)]\psi_{2}[q(\chi)]\,\mathcal{D}[q(\chi)],$
(4.8)

where $\mathcal{D}[q(\chi)]$ is an appropriate integration measure in the
space of the functions $q(\chi)$. The momentum operators $\hat{p}(\chi)$ are
expressed as functional differential operators:

$\hat{p}(\chi):=-i\frac{\delta}{\delta q(\chi)},$ (4.9)

and so Eq. (4.6) takes the form:

$-i\frac{\delta\psi[q(\chi)]}{\delta q(\chi)}=p(\chi)\psi[q(\chi)].$ (4.10)

Fortunately, this functional deifferential equation may be solved explicitly.
Its general solution is:

$\psi[q(\chi)]=\mathcal{N}\exp\left[i\int_{I^{2}}p(\chi)q(\chi)\,d^{2}\chi\right],$
(4.11)

where $\mathcal{N}$ is an arbitrary complex number.

As one may observe, the functional $\psi[q(\chi)]$ in Eq. (4.11) is uniquely
determined, up to the constant coefficient $\mathcal{N}$, by the function
$p(\chi)$, which gives, at each point $\chi$, the eigenvalue of the operator
$\hat{p}(\chi)$. If we denote by $p^{\prime}(\chi)$ another eigenvalue of
$\hat{p}(\chi)$, the corresponding eigenfunction is:

$\psi^{\prime}[q(\chi)]=\mathcal{N}^{\prime}\exp\left[i\int_{I^{2}}p^{\prime}(\chi)q(\chi)\,d^{2}\chi\right],$
(4.12)

and since eigenstates associated with different eigenvalues must be
orthogonal, we must have:

$\int\psi^{*}[q(\chi)]\psi^{\prime}[q(\chi)]\,\mathcal{D}[q(\chi)]=1,$ (4.13)

whenever

$p^{\prime}(\chi)\equiv p(\chi),$ (4.14)

and

$\int\psi^{*}[q(\chi)]\psi^{\prime}[q(\chi)]\,\mathcal{D}[q(\chi)]=0,$ (4.15)

if $p^{\prime}(\chi)$ differs from $p(\chi)$ at any point $\chi$. In Eq.
(4.12) $\mathcal{N}^{\prime}$ is an arbitrary complex number such that Eq.
(4.13) is satisfied.

We now divide the unit square $I^{2}$ in $N$ non-intersecting subsets $D_{j}$
$(j=1,2,\dots,N)$ such that $D_{j}\cap D_{k}=\emptyset$ for all $j\neq k$,
where $j,k\in\\{1,2,\dots,N\\}$, and

$I^{2}=D_{1}\cup D_{2}\cup\cdots\cup D_{N}.$ (4.16)

We shall assume that each subset $D_{j}$ has the same area

$\Delta\chi:=\frac{1}{N}.$ (4.17)

We shall also assume that in the limit, where $N$ tends to infinity, the
supremum of the diameters of the subsets $D_{j}$ tends to zero. Picking up
from each of the subsets $D_{j}$ a point $\chi_{j}$ we may write the integral
in the exponential in Eq. (4.11) as a Riemann integral:

$\int_{I^{2}}p(\chi)q(\chi)\,d^{2}\chi=\lim_{N\rightarrow\infty}\left(\sum_{j=1}^{N}p_{j}q_{j}\Delta\chi\right),$
(4.18)

where we have denoted:

$\displaystyle p_{j}$ $\displaystyle:=$ $\displaystyle p(\chi_{j}),$ (4.19a)
$\displaystyle q_{j}$ $\displaystyle:=$ $\displaystyle q(\chi_{j})$ (4.19b)

for all $j=1,2,\dots,N$. Hence we have:

$\psi[q(\chi)]=\mathcal{N}\lim_{N\rightarrow\infty}\left[\prod_{j=1}^{N}\exp(ip_{j}q_{j}\Delta\chi)\right].$
(4.20)

Eq. (4.20) enables us to define the integral on the left hand side of Eq.
(4.13):

$\int\psi^{*}[q(\chi)]\psi^{\prime}[q(\chi)]\,\mathcal{D}[q(\chi)]:=\mathcal{N}^{*}\mathcal{N}^{\prime}\lim_{N\rightarrow\infty}\left\\{\prod_{j=1}^{N}\int_{-L/2}^{L/2}\exp[i(p^{\prime}_{j}-p_{j})q_{j}\,\Delta\chi]\,dq_{j}\right\\},$
(4.21)

where we have denoted ${p^{\prime}}_{j}:=p^{\prime}(\chi_{j})$. As one may
see, we have introduced a new, positive number $L$. We might, of course, have
taken $L$ to infinity, and thereby integrate $q_{j}$ from the negative to the
positive infinity. Such a choice, however, would make the integral in Eq.
(4.21) infinite. Because of that we shall henceforth take $L$ to be a fixed
number. If we want Eqs. (4.13) and (4.15) to be satisfied, we must have, for
every $j=1,2,\dots,N$:

$p_{j}\Delta\chi=n_{j}\frac{2\pi}{L},$ (4.22)

where $n_{j}=0,\pm 1,\pm 2\,\dots$.

The diffeomorphism $f:I^{2}\longrightarrow B$ defined in Eq. (3.6) maps every
subset $D_{j}$ of $I^{2}$ to a certain piece, or constituent

$U_{j}=f(D_{j})$ (4.23)

of the accelerating plane $B$. These constituents have the properties:

$U_{j}\cap U_{k}=\emptyset$ (4.24)

for all $j\neq k$, and

$B=U_{1}\cup U_{2}\cup\cdots\cup U_{N}.$ (4.25)

In the limit, where the number $N$ of the constituents becomes very large, we
may view the quantity

$A_{j}:=8\pi p_{j}\Delta\chi$ (4.26)

as the area of the constituent $j$. According to Eq. (4.22) the possible area
eigenvalues of the constituent are:

$A_{j}=n_{j}\frac{16\pi^{2}}{L}.$ (4.27)

Obviously, negative area eigenvalues do not make sense, and therefore we shall
ignore the negative values of the quantum numbers $n_{j}$, and keep only the
non-negative ones. In other words, $n_{j}=0,1,2,\dots$ in Eq. (4.27).

As one may observe from Eq. (4.27), the areas of the constituents of the
accelerating plane $B$ have a discrete spectrum with an equal spacing. Unless
the plane has an infinite number of constituents with zero area, we are
therefore forced to conclude that the number of the constituents of the plane
is finite. This number we shall denote by $N$. The total area of the plane is
the sum of the areas of its constituents:

$A=A_{1}+A_{2}+\cdots+A_{N}.$ (4.28)

So we see that the postulates (4.6) and (4.7) of our model imply that our
plane necessarily has a discrete structure in the sense that it consists of a
finite number of separate constituents. The possible area eigenvalues of the
plane are of the form:

$A=\frac{16\pi^{2}}{L}(n_{1}+n_{2}+\cdots+n_{N}).$ (4.29)

The eigenfunctions associated with these area eigenvalues are of the form:

$\psi(q_{1},q_{2},\dots,q_{N})=\mathcal{N}\exp\left[\frac{i}{8\pi}(A_{1}q_{1}+A_{2}q_{2}+\cdots+A_{N}q_{N})\right].$
(4.30)

Hence the wave functions are no more functionals, but ordinary functions of a
finite number of variables $q_{1},q_{2},\dots,q_{N}$. The inner product
between the wave functions is defined as:

$\langle\psi_{1}|\psi_{2}\rangle:=\int_{-L/2}^{L/2}dq_{1}\int_{-L/2}^{L/2}dq_{2}\cdots\int_{-L/2}^{L/2}dq_{N}\,\psi^{*}_{1}(q_{1},q_{2},\dots,q_{N})\psi_{2}(q_{1},q_{2},\dots,q_{N}).$
(4.31)

and if we want the wave function $\psi(q_{1},q_{2},\dots,q_{N})$ to be normed
to unity, we may take:

$\mathcal{N}=\frac{1}{\sqrt{L^{N}}}.$ (4.32)

So we find:

$\psi(q_{1},q_{2},\dots,q_{N})=\frac{1}{\sqrt{L^{N}}}\exp\left[\frac{i}{8\pi}(A_{1}q_{1}+A_{2}q_{2}+\cdots+A_{N}q_{N})\right].$
(4.33)

Eqs. (3.5) and (4.29) imply that the energy eigenvalues of the gravitational
field from the point of view of our accelerating observer are:

$E=\frac{\alpha a}{8\pi}(n_{1}+n_{2}+\cdots n_{N}),$ (4.34)

where

$\alpha:=\frac{16\pi^{2}}{L}.$ (4.35)

As the reader may have noticed, quantization of the gravitational field in
flat spacetime from the point of view of an accelerating observer is, in our
model, very similar to the quantization of the system of free particles in
non-relativistic quantum mechanics. In such systems the coordinates of the
momentum space are constants of motion, and the coordinates of the
configuration space are, possibly up to additive constants, linear functions
of time, whereas in our system the momentum variables $p(\chi)$ are constants
, and the corresponding coordinates $q(\chi)$ of the configuration space
depend, up to additive constants, linearly on the proper time $\tau$ of the
observer. When quantizing the system of free particles in non-relativistic
quantum mechanics one must use box normalization for the wave function of the
system. When using the box normalization one assumes that the particles of the
system lie in a rectangular box. Finally, at the end of the calculations, the
edge kength of the box is taken to infinity. In our model we also applied a
sort of box normalization, where the coordinates of the confuguration space of
the system were assumed to lie within the interval $[-L/2,L/2]$. However, we
did not take the ”edge length” $L$ of the box to infinity, but we took $L$ to
be a fixed number. As a consequence, $L$ appears as a parameter of the model,
which must be fixed such that the predictions of the model agree with
experiments.

##  5 Unruh Effect

In Section 4 we completed, in our model, the quantization of gravity in flat
spacetime. The most important outcome of our model was that the area of an
accelerating plane has a discrete spectrum with equal spacing. We are now
prepared to go to the thermodynamical properties of gravity. As it is well
known, the thermodynamical properties of any system may be deduced from its
partition function

$Z(\beta):=\sum_{n}e^{-\beta E_{n}},$ (5.1)

where $\beta$ is the temperature parameter, and we have summed over the energy
eigenstates $n$ of the system. In what follows, we shall base our calculation
of the partition function on Eq. (4.34). We say that constituent $j$ is in
vacuum, if $n_{j}=0$; otherwise the constituent is in an excited state.

When obtaining an expression for the partition function of our system we shall
assume that at least one of the constituents of the accelerated plane is in an
excited state, and when the quantum states of the constituents are
interchanged, the quantum state of the system will also change. With these
assumptions the partition function takes the form:

$\begin{split}Z(\beta)=&\,\,\,\,\sum_{n_{1}=1}^{\infty}\exp\left(-\frac{\alpha\beta
a}{8\pi}n_{1}\right)\\\ &+\sum_{n_{1}=1}^{\infty}\exp\left(-\frac{\alpha\beta
a}{8\pi}n_{1}\right)\sum_{n_{2}=1}^{\infty}\exp\left(-\frac{\alpha\beta
a}{8\pi}n_{2}\right)\\\ &+\cdots\\\
&+\sum_{n_{1}=1}^{\infty}\exp\left(-\frac{\alpha\beta
a}{8\pi}n_{1}\right)\cdots\sum_{n_{N}=1}^{\infty}\exp\left(-\frac{\alpha\beta
a}{8\pi}n_{N}\right).\end{split}$ (5.2)

In the first term on the right hand side of Eq. (5.2) just one of the
constituents is in an excited state, and we have summed over all of those
states. In the second term two of the constituents are in excited states.
Finally, in the last term all of the $N$ constituents are in excited states.
Defining the characteristic temperature

$T_{C}:=\frac{\alpha a}{8\pi\ln(2)}$ (5.3)

we may write Eq. (5.2) as:

$\begin{split}Z(\beta)&=\frac{1}{2^{\beta T_{C}}-1}+\left(\frac{1}{2^{\beta
T_{C}}-1}\right)^{2}+\cdots+\left(\frac{1}{2^{\beta T_{C}}-1}\right)^{N}\\\
&=\frac{1}{2^{\beta T_{C}}-2}\left[1-\left(\frac{1}{2^{\beta
T_{C}}-1}\right)^{N}\right],\end{split}$ (5.4)

whenever $\beta T_{C}\neq 1$. When $\beta T_{C}=1$, we have:

$Z(\beta)=N.$ (5.5)

The energy of the gravitational field may be calculated from its partition
function $Z(\beta)$ as:

$E=-\frac{\partial}{\partial\beta}\ln[Z(\beta)],$ (5.6)

and if we define the energy per constituent as

$\bar{E}:=\frac{E}{N},$ (5.7)

Eqs. (5.4) and (5.6) imply:

$\bar{E}=\bar{E}_{1}+\bar{E}_{2},$ (5.8)

where we have denoted:

$\displaystyle\bar{E}_{1}$ $\displaystyle:=$
$\displaystyle\frac{1}{N}\frac{2^{\beta T_{C}}}{2^{\beta
T_{C}}-2}T_{C}\ln(2),$ (5.9a) $\displaystyle\bar{E}_{2}$ $\displaystyle:=$
$\displaystyle\frac{2^{\beta T_{C}}}{2^{\beta T_{C}}-1-(2^{\beta
T_{C}}-1)^{N+1}}T_{C}\ln(2),$ (5.9b)

whenever $\beta T_{C}\neq 1$. When $\beta T_{C}=1$, a calculation similar to
the one performed in Ref. [7] gives, in the large $N$ limit, the result:

$\bar{E}=T_{C}\ln(2).$ (5.10)

Eqs. (4.34), (5.3), (5.8) and (5.9) imply that the average

$\bar{n}:=\frac{n_{1}+n_{2}+\cdots n_{N}}{N}$ (5.11)

of the quantum numbers $n_{1},n_{2},\dots,n_{N}$ depends of the temperature
parameter $\beta$ as:

$\bar{n}=\frac{1}{N}\frac{2^{\beta T_{C}}}{2^{\beta T_{C}}-2}+\frac{2^{\beta
T_{C}}}{2^{\beta T_{C}}-1-(2^{\beta T_{C}}-1)^{N+1}}.$ (5.12)

So far everything has been quite exact, and no approximations have been made.
At this point we proceed to consider, what happens in the limit, where the
number $N$ of the constituents of the accelerating plane tends to infinity. In
this limit the first term on the right hand side of Eq. (5.12) will vanish,
and we have, in effect:

$\bar{n}=\frac{2^{\beta T_{C}}}{2^{\beta T_{C}}-1-(2^{\beta T_{C}}-1)^{N+1}}.$
(5.13)

When $\beta T_{C}>1$, which means that the temperature $T$ experienced by the
observer moving along the plane is less than the characteristic temperature
$T_{C}$ we have:

$\lim_{N\rightarrow\infty}(2^{\beta T_{C}}-1)^{N+1}=\infty,$ (5.14)

and so the constituents of the plane are, in effect, in vacuum. However, when
$\beta T_{C}<1$, which means that $T>T_{C}$, we find:

$\lim_{N\rightarrow\infty}(2^{\beta T_{C}}-1)^{N+1}=0,$ (5.15)

and so we may write, as an excellent approximation:

$\bar{n}=\frac{2^{\beta T_{C}}}{2^{\beta T_{C}}-1}.$ (5.16)

An interesting property of this expression is that, when $T$ tends to $T_{C}$
from the right hand side, which means that $\beta T_{C}$ tends to $1$ from the
left hand side, we have:

$\bar{n}=2.$ (5.17)

So we find that at the characteristic temperature $T_{C}$ the accelerating
plane performs a phase transition, where the constituents of the plane jump,
in average, from the vacuum to the second excited states. The latent heat per
constituent associated with this phase transition is

$\bar{L}=2T_{C}\ln(2).$ (5.18)

Since the constituents of the plane are effectively in the vacuum, when
$T<T_{C}$, and jump to the second excited states at the characteristic
temperature $T_{C}$, we may view the characteristic temperature $T_{C}$,
defined in Eq. (5.3), as the lowest possible temperature experienced by the
accelerating observer. Interestingly, we find that if we choose:

$\alpha=4\ln(2),$ (5.19)

then:

$T_{C}=\frac{a}{2\pi},$ (5.20)

which exactly agrees with the Unruh temperature [15]

$T_{U}:=\frac{a}{2\pi}$ (5.21)

experienced by the observer. In this sense we have been able to derive the
Unruh effect from our model of quantum gravity in flat spacetime: An
accelerating observer detects thermal radiation with a temperature, which is
proportional to the proper acceleration $a$ of the observer. That radiation is
produced, when the constituents of the plane perform quantum jump from the
excited states to the vacuum.

##  6 Concluding Remarks

In this paper we have constructed, step by step, a model of quantum gravity in
flat spacetime from the point of view of a unformly accelerating observer. An
observer of this kind detects phenomena identical to those in a uniform
gravitational field in his surroundings, and we considered those effects
quantum-mechanically. Among other things, our nodel of quantum gravity
predicted that spacetime has a discrete structure in the sense that the plane
at rest with respect to the accelerating observer consists of separate
constituents, each of which has an area, which is an integer times a certain
fundamental area. As a consequence, the area eigenvalues of the accelerating
plane are of the form:

$A=\alpha(n_{1}+n_{2}+\cdots n_{N}),$ (6.1)

where the quantum numbers $n_{j}$ are non-negative integers for all
$j=1,2,\dots,N$, and $N$ is the number of the constituents of the plane. We
found that with the choice

$\alpha=4\ln(2)$ (6.2)

for the parameter $\alpha$ of our model an observer with constant proper
acceleration $a$ detects thermal radiation with certain minimum temperature,
which agrees with the Unruh temperature

$T_{U}=\frac{a}{2\pi}$ (6.3)

of the observer. In this sense our model of quantum gravity in flat spacetime
implies the Unruh effect.

Basically, the results of the paper were outcomes of our decision to
supplement the standard Einstein-Hilbert action with the standard Gibbons-
Hawking boundary term. We first picked up a rectangular box on a spacelike
hypersurface of the flat Minkowski spacetime spacetime, where the time
coordinate is a constant, and then put one of its faces into a uniformly
accelerating motion, while the other faces were kept at rest in the flat
Minkowski system of coordinates. When the faces of the box proceeded in the
flat spacetime, a three-dimensional boundary was created for spacetime. It
turned out that the trace of the exterior curvature tensor on the three-
dimensional timelike hypersurface created by the accelerating plane was non-
zero, while the exterior curvature tensor on the timelike hypersurfaces
created by the other faces of the box vanished identically. As a consequence,
the Gibbons-Hawking boundary term was non-zero, even though the Einstein-
Hilbert action vanished. Thus, the presence of the Gibbons-Hawking boundary
term produced non-zero gravitational action, despite of the fact that
spacetime was flat.

From the action we obtained the classical Hamiltonian with respect to the
accelerating observer. The discrete structure of the accelerating plane
followed from the straightforward replacement of the classical Hamiltonian by
the corresponding Hamiltonian operator according to the standard rules of
quantum mechanics. The coordinates $p_{j}$ of the momentum space agree, up to
the coefficient $\frac{1}{8\pi}$, with the areas of the constituents of the
plane, whereas the corresponding coordinates $q_{j}$ of the configuration
space agree, up to additive constants, with the boost angle of the observer.
In the process we were compelled to introduce a box normalization for the wave
function of the gravitational field, where the coordinates $q_{j}$ of the
configuration space are confined to live inside of a box with edge length $L$.
222Not to be confused with the edge length of the accelerating box in flat
spacetime! Defining a parameter $\alpha$ in terms of the edge length $L$ as:

$\alpha=\frac{16\pi^{2}}{L}$ (6.4)

we obtained the area spectrum in Eq. (6.1).

As we have seen, quantization of gravity in flat spacetime from the point of
view of an accelerating observer is really very simple and straightforward,
and may be carried out explicitly. Nevertheless, there are still some
shortcomings in our procedure. For instance, our model of quantum gravity
predicted for the areas of the constituents of the accelerating plane both
positive and negative eigenvalues, and we excluded the negative eigenvalues
”by hand”. An even more serious problem in our approach is the presence of an
undetermined parameter, the edge length $L$ of the box used in the
normalization of the wave function. Eqs. (6.2) and (6.4) imply that if we
choose

$L=\frac{4\pi^{2}}{\ln(2)}\approx 57.0,$ (6.5)

our model predicts the Unruh effect. The close relationship between the
variables $q_{j}$ and the boost angle $\phi$ of the observer suggests that the
edge length $L$ sets the bound for $\phi$ such that, for symmetry reasons,
$-\frac{L}{2}\leq\phi\leq\frac{L}{2}$. Whether such interpetation is valid, is
still an open question, as well as are its possible consequences.

Taken as a whole, our paper highlights an aspect of the potential quantum
theory of gravity, which so far has received very little attention: The
quantum states of the gravitational field are not absolute, but relative, and
they depend on the state of motion of the observer. For instance, an
accelerating observer in flat spacetime detects gravity-like effects, which
may be quantized by means of the standard rules of quantum mechanics, whereas
inertial observers detect no effects of gravity whatsoever. It is to be hoped
that the results obtained this paper would pave the way for a proper quantum
theory of gravitation in curved spacetime.

## References

  * [1] A. Einstein, Jahrb. Radioaktivität Elektronik 4 (1907) 414.
  * [2] A. Einstein, Ann. Phys. 35 (1911) 898.
  * [3] G. W. Gibbons, S. W. Hawking, Phys. Rev. D15 (1977) 2752.
  * [4] See, for example, R. M. Wald, General Relativity (The University of Chicago Press, Chicago 1984).
  * [5] There are still conributions to the boundary term emerging from the edges of our box. Those contributions, however, may be cancelled, if necessary, by means of an addition of an appropriate, constant boundary term.
  * [6] See, for example, H. Goldstein, Classical Mechanics (Second Edition) (Addison-Wesley, 1980).
  * [7] J. Mäkelä, Entropy 13 (2011) 1324.
  * [8] J. Mäkelä, Int. J. Mod. Phys. D23 (2014) 1450001.
  * [9] J. Mäkelä, Phys. Rev. D87 (2013) 104040.
  * [10] J. Mäkelä, Phys. Rev. D91 (2015) 124050.
  * [11] J. Mäkelä, Phys. Rev. D93 (2016) 084002.
  * [12] J. Mäkelä, Int. J. Mod. Phys. D28 (2019) 1950129.
  * [13] J. Mäkelä, Phys. Rev. D103 (2021) 027002.
  * [14] J. Mäkelä, Int. J. Mod. Phys. D31 (2022) 2250082.
  * [15] W. G. Unruh, Phys. Rev. D14 (1976) 5670.