# An Analysis of Deep Reinforcement Learning Agents for Text-based Games

Chen Chen<EMAIL_ADDRESS>
Yue Dai<EMAIL_ADDRESS>
Josiah Poon<EMAIL_ADDRESS>
Caren Han caren.han<EMAIL_ADDRESS>
The University of Sydney
Camperdown, NSW 2006 Australia

###### Abstract

Text-based games(TBG) are complex environments which allow users or computer
agents to make textual interactions and achieve game goals.In TBG agent design
and training process, balancing the efficiency and performance of the agent
models is a major challenge. Finding TBG agent deep learning modules’
performance in standardized environments, and testing their performance among
different evaluation types is also important for TBG agent research. We
constructed a standardized TBG agent with no hand-crafted rules, formally
categorized TBG evaluation types, and analyzed selected methods in our
environment.

##  1 Introduction

Text-based games are designed for humans to play and finish quests using text
replies. During one episode of a text-based game, when a player takes one
action, the game environment provides some text descriptions about the current
game state, typically about a place or some effects caused by the player’s
last action. Based on the text description(feedback), the player makes the
next move, keeps collecting information, and tries to reach a winning state
with the least steps possible.

Figure 1: Text-based Game Agent Workflow

Research on Text-based games mainly focuses on building goal-oriented smart
agents. The methods in Text-based games could be applied to interactive
language-related applications, such as the Internet of Things (IoT), smart
assistants, online technical support/chatbots, and so on. The latest methods
in this area include Reinforcement Learning and Natural Language Understanding
modules. This combination enables smart agents to be trained through live
language interactions and learn to understand the environment, improve
themselves to perform actions and achieve goals.

The language models are typically trained offline in many traditional natural
language tasks. However, in wide-ranging applications, like the smart agents
in home pods, we expect the smart agent to adapt to different users in day-to-
day interactions and figure out actions that need to take based on specific
user language features and habits without hard-coded settings and improve user
satisfaction. Text-based Games research is performed in a similar environment,
where the agent explores the text environment, takes actions, collects
results, and finally finds an optimal mapping of language input and actions to
take without manually labelling the input data and training a model offline.

Some early papers in this field use hand-crafted rules as the core of the
agents. As we expect Text-based game agents capable of online learning and
improvement, this paper focuses only on Deep Reinforcement Learning agents. We
also emphasize the deep learning module capability and efficiency analysis in
this field, which is the critical component of agent online learning
performance. The development path of this field is described in table 2. At
the early years, the models were established on single-setting games, later
with the development of Text-based Game experiment platforms. The agents are
trained on multiple games with different settings. More methods that have been
successfully applied to the Natural Language Processing field are introduced
into Text-based game agents, including graphs and external knowledge.

The most regularly used platform in this field is Textworld. Due to the
complexity of Text-based game agents, even trained on the same platform, hand-
crafted rules/language features and other component differences make it hard
to compare the effectiveness of deep learning module performance alone. To
offer a clearer view of method effectiveness among benchmarked works, We
collected some deep learning modules and methods from related research works,
benchmarked them under the same environments, and experimented with the same
hyperparameters in different game scenarios.

Some new approaches to the Text-based game task have shown outstanding
results, like graph-aided techniques. Rather than learning games in one set,
or using as many features as possible, new challenges in this field includes
using less hand-crafted features and training the agent to solve unseen games.

The motivation for this survey is to examine literature in this field after
2015, analyze it in depth and benchmark some methods in a standardized
environment. We summarize approaches introduced so far, present our benchmark
results and analysis, and point out some future directions. In this paper, we
consider three questions:

  * •

Q1: How do different agent models balance efficiency and performance?

  * •

Q2: How do some modules perform on an agent without hand-crafting rules and
features in standardized environments?

  * •

Q3: Can models perform consistently in different evaluation types?

###  1.1 Related analysis and surveys

Most of the Text-based Games agent field methods are evaluated in separate
research works. Though each research work selects benchmarks to present the
significance of the findings, the evaluations are conducted with different
games and settings. They might incorporate different levels of hand-crafted
rules, with additional modules on top of the Deep Neural Network structure.
Several surveys in this field collected methods and results within the
research field. Inspired by these works, we decided to focus our analysis on
deep learning modules used in TBG agents, review the related methods more in-
depth, test selected models in a standardized evaluation setting and present
our results and analysis.

###  1.2 Structure of the analysis

This analysis begins with an overview of the literature and related
methodology. We introduce the methodology development trend in this field,
review and classify the methods, and explain how they are applied. We review
the evaluation platforms and metrics used in related research works and then
introduce our ablation test on selected methods within a standardized
environment. Finally, we summarize our analysis of the methods and findings
and address the issues in this field for future direction.

##  2 Overview of the literature

Text-based game agents are designed to explore the game environment, extract
useful information from text features provided, and learn optimal policy to
give correct text actions. Efforts have been made to improve text encoding
components, processing text features, and efficiently producing text actions.
Issues are learning global policy, accurately constructing game state
representations, and reducing state space.

Text-based game is generally framed as a reinforcement learning problem. In
early years, many hand-crafted methods are introduced to try to solve games by
analyzing text patterns and crafting templates. Deep Reinforcement learning is
later introduced into this field. Test environments like Textworld are
introduced to enable learning from a batch of games, and testing agent
generalization ability on unseen games. Neural networks such as MLP, RNN are
applied into agent design as text encoder and being developed over ensuing
years.

Table 1:  Historical overview of text-based game papers Year |  Network Architecture/Training |  Reinforcement Learning Methods
---|---|---
2015 |  MLP, Word Embeddings |  State Encoding, Deep Q-Learning
2018 |  State Recurrence, Multiple games, Zero-shot Evaluation |  Exploration Reward, Replay Memory
2019 |  Graph Attention, pre-trained knowledge graph |  Graph Representation, Actor-Critic

Table1 summarizes major methodologies introduced into this field, ranked by
timeline. The two earliest papers(?, ?) (2015, table 1) introducing deep
reinforcement learning into text-based games are inspired by NLP methods.
Popularly methods like word embeddings and MLP are introduced for agents to
process and encode text descriptions. A word embedding sequence represents the
game state, and vanilla DQN is used as RL algorithm. With the two earliest
works, more variations of neural network architecture, such as RNN and
Transformer are introduced. The actor-critic algorithm was introduced as an
alternative Reinforcement Learning framework to DQN. Various graph methods are
also introduced as a new type of state representation. More pre-trained
modules are included in text-based game agents for enhancement.

Text-based game platforms are also under continuous development. The earliest
works only focus on one particular game. Later platforms allow the evaluation
to be done on different games. On top of different games,
Textworld(?)(2018,table1) offers a large batch generation of similar games,
which allow agents to be trained on a large training set to allow testing on
unseen games.

The latest development of text-based game agent research is more focused on a
graph representation of game state(2019,table1), and several graph generation
methods are introduced. Recently, researchers also focus on enhancing agent
language understanding capability by including pre-trained language processing
components into agents’ neural network architecture.

##  3 Reinforcement Learning Methods

As text-based game agent training requires interaction with the game
environment, most related research works are built under the Reinforcement
Learning(RL) framework. Various RL methods have been applied, including game
states and action interpretation, RL algorithms and reward functions. In this
section, we introduce basic concepts of major methodologies and classify
related works in an ontology (Figure 3) for reader’s reference.

###  3.1 State Representation

Under the Reinforcement Learning framework, a text-based game environment can
be modelled as POMDP, and the agents do not have direct access to the game
states. One of the challenges for solving text-based games, even for a human,
is interpreting visible game features and choosing the correct data formation
to represent the game’s state as accurately as possible.

####  3.1.1 Embedding only

Research works like (?, ?) only use word embedding sequences generated from
game state text description as game representations. Pre-trained, and non-
pretrained methods are adopted, which is discussed in the word embedding
section. Some researchers also concatenate features like game file names as
game state representations.

####  3.1.2 Graph aided

Graph has been widely used as a feature to solve problems in many fields, such
as bioinformatics and computer vision. For text-based games like Textworld
games, the game states can be accurately represented by the graph. The
accurate graph representation is usually not accessible to game players. Using
pretrained neural networks or hand-crafted rules, some researchers try turning
game state text description and agent experience into knowledge graphs. Neural
network architectures like GCN can encode the generated knowledge graph.

###  3.2 Action Space

Action space flexibility describes a text-based game agent’s ability to adapt
to complex action text and action sets. Some agents can only cope with a small
fixed action set, and others can deal with action text with varying lengths,
large vocabulary, and complex combinations.

####  3.2.1 Non-flexible

Text-based game agents’ deep learning models could be designed with a fixed
output fit a small action set. For example, (?) fixed their network output
shapes to 2 and 5, to deal with a fixed 2*5 (10 different action choices in
total) game action set. This design has different sets of neural network layer
weights for different action choices, which could be beneficial to predict
value more accurately for each action. Yet due to the network output size only
fits a fixed group of actions, this design is hard to generalize to games with
longer action text sentences and varying action choices.

####  3.2.2 Flexible

To deal with action text sentences with arbitrary length and larger
vocabulary, agent neural network models could be designed to have only one
output node to predict state-action values. This kind of model is designed to
predict values for all available actions and choose one to act based on the
policy. These models have a stronger ability to generalize to different games.
Yet, it doesn’t have other groups of weight trained for each action, which
makes the training process harder than non-flexible type models.

####  3.2.3 Template-based

Research works such as (?, ?) have text action templates based on human
gameplay experience. They only use the deep learning model to predict
appropriate words to fill in the text action templates. This model type shares
the same generalization problem with the non-flexible type, and the whole
template needs to be redesigned to generalize to different games. Also, these
models will need human gameplay experience to determine the most suitable text
templates.

###  3.3 Reinforcement Learning Algorithm

Reinforcement Learning frameworks such as policy gradient, Q-learning and
Actor-Critic are widely adopted in game solving and bioinformatics fields. (?)
firstly adopted Deep Q-Learning to solve text-based games, and some following
works adopted the same framework. Actor-critic is a widely used algorithm in
recent years and has been successfully applied to complex problems like Go
game.

####  3.3.1 Deep Q-Learning

Deep Q-learning(DQN) is a less complex yet powerful Reinforcement Learning
algorithm. DQN is widely used in solving games like Atari. Q-learning uses the
below function to update Q value and solve the game by approximating the Q
value for each state.

$Q_{(}s_{t},a_{t})=\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1})+r_{t}$ (1)

where $s_{t},a_{t},r_{t}$ demotes state,action and reward at time step $t$.

For the text-based games field, (?) firstly adopted DQN to approximate the Q
value for state-action and state-object pairs and select optimal pairs as the
input for each text game state description. (?) also used DQN, and their
result showed that, with some additional reward design, DQN could support the
text-based games to solve complicated games and generalize the performance on
unseen games.

####  3.3.2 Actor-Critic

Actor-critic is considered a more advanced algorithm than DQN, and its
capability has been proven to solve very complex games such as the GO game.
Researchers have used Advantage Actor Critic(A2C) to train text-based game
agents. A2C use below formula to calculate value $v(s_{t})$ ,return
$R(s_{t},a_{t})$ and advantage $Adv(s_{t},a_{t})$for game states and actions.

$R(s_{t},a_{t})=\gamma^{l}v(s_{t+l})+\sum_{k=1}^{l}\gamma^{k-1}r_{t+k}\ $ (2)
$Adv(s_{t},a_{t})=R(s_{t},a_{t})-v(s_{t})$ (3)

A2C is considered a reliable agent training method as it reduces variances
using advantage. Yet most TBG research work doesn’t compare performance
between applying Q-learning and A2C, as it requires neural network
architecture change.

####  3.3.3 Reward Functions

The goal of a Text-based game agent is to collect game scores from the
environment as much as possible. A game score can be directly adapted as a
reward to the agent. Researchers also designed reward functions to encourage
certain agent behaviours, such as choosing actions likely to lead the agent to
unexplored game states. For example, the exploratory reward function designed
by (?) encourages the agent to explore unseen states.

##  4 Deep Learning Architectures and modules

###  4.1 Encoder Type

As text-based game agents need to process text information given by game
environment, the architecture of the text encoder type have a large influence
on agent performance. The agent model also needs to be lite, as the agents are
trained by sampling from game environments, and the text encoder needs to be
efficient enough to support agent exploration.

####  4.1.1 MLP

Multilayer Perceptron(MLP) is a feed-forward neural network consisting of an
input layer, one or more hidden layers and an output layer. MLP is a basic
form of neural network and is widely used in domains like classification,
regression and reinforcement learning. (?) uses MLP to encode text inputs in
TBGs. For each state/action text pair, DRRN(?) feeds the state embedding
vectors and action embedding vectors into two MLPs separately, then
approximate the Q value based on the inner product of vectors from the last
hidden layers of both MLPs.

####  4.1.2 CNN

Because of its outstanding performance in deep learning, Convolutional Neural
Network(CNN) has become one of the most popular neural networks. Although CNN
is more popular in computer vision, its performance in other domains like time
series prediction and signal identification is also noticeable. The basic
architecture of CNN includes an input layer, convolutional layers, pooling
layers, fully-connected layers and an output layer. Various CNN models like
VGGNet(?), GoogLeNet(?) and ResNet(?) differ in the number of layers, number
and size of kernels, etc.. For TBG tasks, (?, ?, ?, ?) leverage CNN to encode
game description text input. All the models are modified based on (?). (?, ?)
use the similar CNN structure in (?). (?, ?) only use a lite version of CNN,
which consists of an input layer, Convolutional layer and max-pooling layer.
The output of the max-pooling layer is the encoding of text input. Besides,
(?, ?, ?) add position embedding into CNN input to assist model learning.

####  4.1.3 LSTM/GRU

A recurrent Neural Network(RNN) is widely used in processing sequential data.
It is commonly used in tasks, like machine translation, sentiment analysis,
video analysis, speech recognition, etc. Long Short-Term Memory(LSTM) and
Gated Recurrent Unit(GRU) are two typical forms of RNN. The input of both LSTM
and GRU is word embedding, and each cell generates an output and a hidden
state passed to the next cell. Research works such as(?, ?) tokenize game
state description text, map one-hot embeddings to higher dimensional word
embedding sequences, and use LSTM to encode the embedding sequence as state
encoding. Other works such as (?, ?) utilize GRU. Both output of LSTM/GRU and
the hidden state of the last cell can be used as the game state encoding.

####  4.1.4 Transformer

Since Transformer(?) was proposed, it has quickly become the hot field in
research because of its great success. Although the original transformer is
designed for NLP tasks, variants of transformers have entered other domains
like computer vision and audio processing. The transformer in (?) has an
encoder and a decoder. Both encoder and decoder take the word embedding
sequence generated from game state text, and then add positional embedding
onto the word embedding sequence. This sequence is used as the transformer
encoder input. For TBG field, in(?), (?) and (?) use transformer encoder to
encode game state description text, and use the transformer encoder output as
state or action encoding.

####  4.1.5 BERT

Bidirectional Encoder Representations from Transformers(BERT)(?) is a pre-
trained transformer-based model. It achieves many state-of-art performances in
NLP tasks. In the TBG research field, research works like(?, ?) use BERT for
state description text encoding. In these works, part of the weights of the
encoder is frozen, and only a small part of the encoder layers will be fine-
tuned during training, which makes the training more efficient. Compared to
the Transformer encoder, the pre-trained BERT model normally uses it’s own
tokenizer and produces word segment embeddings. It has a much more number of
parameters to capture information from a huge corpus. Among all the encoder
types, BERT is the heaviest model.

###  4.2 Word Embedding

Word embeddings are used to represent tokens in given texts. Word embeddings
can be pre-trained from the selected corpus and fixed for later use.
Researchers sometimes unfreeze pretrained embedding weights to let downstream
tasks update them. Most text-based game agents use embedding sequences to
represent the game state or action text.

####  4.2.1 Pretrained

(?), (?),(?) and (?) leverage pre-trained word embedding models and keep the
embeddings fixed during training the agent. Among them, (?) and (?) use
fastText(?), (?) use Glove(?) and BERT, (?) use BERT. (?) use spaCy(?) which
is a NLP library that provides static pre-trained word embeddings. In
addition, (?) uses bag-of-words(BOW), (?) compares BOW and bag-of-bigrams(BI)
representation with trained LSTM embedding. (?) uses hash values as word
embedding and shows the agent can achieve a high score even without language
semantics.

####  4.2.2 Non-pretrained

Most of the agents discussed in this paper either fine-tune the pre-trained
word embeddings or train the embeddings from scratch. Research works like (?),
(?) initialize the word embeddings as either zeros or random vectors. Some
research works like (?) and (?) initialize the embedding using pre-trained
Glove model, (?) uses word2vec(?) model instead. Based on the experiment from
(?), although fully fine-tuning BERT outperforms freezing it during training,
the former sacrifices the latter’s advantage in training speed. (?) conducts
similar experiments, and they compare fine-tuning pre-trained GPT-2(?)—a model
that only consists of decoder part of transformer—and learning a random
initialized GPT-2, the result shows pre-trained GPT-2 has better performance.

###  4.3 Pretrained Modules

Pretrained modules are widely used in text-based game agent designs. With
pretrained modules, text-based game agents are initialized with knowledge
before exploring game environments. Using pretrained modules can improve agent
performance, yet potentially could damage agent efficiency and generalization
ability towards different games.

####  4.3.1 Supervised

(?) trains a paired question encoder and answer encoder using DrQA(?) method.
The data are generated from an agent that is accessible to the shortest
solution of the game. The weights of the encoders and the embedding learned
are used to initialize the agent. Ammanabrolu et al.(?) only tested this
method on TextWorld games with a ’home’ theme in this paper. Later, they
extended the question-answering pairs to a set of Jericho games and proposed
Jericho-QA(?). In this paper, Ammanabrolu et al.(?) pre-trained ALBERT(?) with
Jericho-QA. The information extracted by ALBERT from text observation is used
to update the knowledge graph during training. (?) pre-trained a task-specific
module called Recipe Manager. It is used to predict what’s left to complete
the tasks. As mentioned, it can only be used in the cooking games generated by
TextWorld. (?) pre-trains CALM with ClubFloyd–a dataset containing state-
action pairs of TBGs— to generate a relatively small size of possible actions
at each turn of the game.

####  4.3.2 Unsupervised

(?) proposed two methods to pre-train knowledge graph updater, which are
Observation Generation (OG) and Contrastive Observation Classification (COC).
OG trains the graph updater with a decoder to reconstruct text observation.
COC instead trains the model to differentiate between true observation and a
fake one.

Figure 2: ontology ptModel Features Encoder Type MLP (?) CNN (?, ?, ?, ?)
LSTM/GRU (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) Transformer (?,
?, ?, ?) Word Embedding Pretrained (?, ?, ?, ?, ?, ?, ?, ?, ?) Non-pretrained
(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RL Frame Work DQN (?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) Actor-Critic (?, ?, ?, ?, ?, ?, ?, ?)
Action Space Flexibility Non-flexible (?, ?, ?, ?) Flexible (?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?) Template based (?, ?, ?, ?, ?) State Representation
Graph-aided (?, ?, ?, ?, ?, ?, ?) embedding only (?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?) Pretrained Module Supervised (?, ?, ?, ?) Unsupervised (?,
?) Figure 3: Ontology for TBG agent methods

##  5 Benchmark Environments

###  5.1 Text-based game platforms

####  5.1.1 Single games

At an early stage in the Text-based Games agent research field, researchers
like (?)(?) use single game environments like Evennia and Ifarchive. The games
typically offer limited difficulty settings and a few fixed game maps. For
this kind of environment, the agent only needs to learn how to solve a single
game, which Reinforcement Learning tabular methods can handle, as state and
action combinations are very limited.

####  5.1.2 Text-based game platforms

In 2018, Cote et al.(?) built the first platform to support text-based game
research. The environment supports building a set of games with more detailed
language descriptions, which enables agents to be trained to understand the
language and solve a class of games. Hausknecht(?) developed the Jericho
platform, included additional features like games state trees, and connected
it to the Textworld platform.

Urbanek et al.(?) developed a Light environment and combined the game with
dialogue. The games in this platform do not have complicated rooms and tasks
like Cook in Textworld, but it supports training agents that perform tasks and
chat at the same time.

###  5.2 Text-based game agent evaluation settings

Text-based game agent learning and evaluation settings can be classified as
follows:
1.Single game The agent learns how to play one game and achieve more game
scores possible. Some single games with very large maps and multiple subtasks
can be hard to deal with, like Zork.
2.Multiple games The agent learns to solve a set of games with similar
difficulty and settings. Each game in the set can have different objects and
room configurations, and the text description for rooms can also be different.
Still, the general task and difficulty are very similar in the game set. An
agent is evaluated with average performance among all the games in the set.
3\. Generalisation The agent learns from a set of games for training and is
evaluated by the performance on the training set and on some unseen games to
test if the learnt policy generalises to unseen language descriptions and
settings.

Most of the methods in this field are only evaluated in limited settings, and
we classify the evaluation methods used in related works in ontology figure 4.

ptEvaluation Types single game (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
multiple games (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) generalization (?, ?, ?, ?,
?, ?) Figure 4: Evaluation Types of related works Figure 5: ontology

###  5.3 Evaluation metrics

Two metrics are generally adopted in this text based game agent evaluation:
(1) The average game completion rate:$\sum(s_{o})/(s_{m}*b)$, where$\hskip
2.84526pts_{o}=\hskip 2.84526pt$obtained score by the agent in a game,$\hskip
2.84526pts_{m}=\hskip 2.84526pt$maximum game score, $\hskip 2.84526ptb=\hskip
2.84526pt$evaluation set size;
(2) the average steps for the agent to finish games: $\sum(s_{p})/b$,
where$\hskip 2.84526pts_{p}=\hskip 2.84526pt$step taken for the agent to
finish a game.

##  6 Ablation Test and Analysis

To evaluate and analyse methods used in the Text-based Games research field,
particularly for the methods’ performance in solving multiple games and unseen
games, we created a standardised environment and agents and performed ablation
testing for selected methods.

###  6.1 Ablation Test Experiment settings

We evaluate selected models in several Textworld(?) environments. Textworld is
the most commonly used platform in this field. As it supports large game set
generation, it enables us to train models on large game sets, perform
generalisation tests and see how the testing models perform on unseen games.
We selected three games: Coin Collector, Treasure Hunter and Cooking recipe.
The chosen games have different levels of difficulty. Coin collector is the
easiest, and it has very few rooms($\sim$20), a simple task, and a fixed
action set. The Cooking recipe game’s task is more challenging, and the agent
needs to collect multiple items and perform cooking actions to finish the
game, the action set of different states could be different. Treasure hunter’s
difficulty is between Coin Collector and Cooking recipe.

We report the models’ performance in two different settings: multiple games
and generalisation:
(1) For multiple games setting, we develop and test different games and
train/test our model on the set of games;
(2) To test model performance on unseen games, we evaluate the models in a
generalisation setting. In this setting, we train the models by using 100
generated games and test the models on 20 games that the models have never
seen during training.

The training and testing set is formed by different games that are randomly
chosen from a larger game set. For the generalisation test, we report both
training and testing results. To better visualise the data changing trend in
the figures, the values are filtered with Savgol filter(?) to reduce noise.

Feature |  State Number |  Action Voc Size
---|---|---
Coin Collector Lv5 |  308 |  10
Treasure Hunter Lv5 |  426 |  24
Cooking Recipe lv1 |  374 |  32
Table 2: Game Environment statistics

###  6.2 Ablation Test Methods

####  6.2.1 Encoder types

For encoder types, we benchmark five different kinds of text encoder
architectures popularly used by related works. The five encoders are chosen to
cover the main classes of Neural Network architectures and, at the same time,
cover different parameter number levels so we can analyze the balance of
performance and efficiency. We selected GRU and LSTM as our test encoders for
Recurrent Neural Networks, which adopt a timestep recurrence methodology.
These two models are the most commonly used RNNs. GRU has a less complex
structure than LSTM, as RL training requires better efficiency for agent
exploration. We want to compare the performance of GRU and LSTM and see if GRU
works faster with a less complex structure. The rest three
encoders(CNN,BERT,Transformer) are non-recurrent type encoders. CNN adopts a
convolution methodology, while Transformer and BERT use an attention
mechanism. BERT is the most popular model in traditional NLP tasks, yet it has
many more parameters than the others.

From ablation test results on a training set with 100 different games reported
in fig 6-8(a), the CNN encoder learns faster than other models. The
transformer encoder has comparable performance to CNN encoder, especially in
the Cook game. Both CNN and Transformer encoder converge for the Coin-
collection game and Treasure hunter game. CNN encoder learns slightly faster.
The two models are powerful enough to learn useful information in the Cook
game, which is very challenging to solve, without any additional modules.For
RNN models, we tested GRU and LSTM encoder. Two encoders have similar
performance in all settings but can only converge to optimal policy in Coin-
Collector. The two recurrent encoders failed to converge for the other two
more complex games. For the Treasure hunters, they can learn some useful
information and solve some games in the set, but in the Cook game, they can’t
stably learn from the environment. We also tested the BERT encoder, which is
an SOA language learning architecture. Surprisingly, without pretrained
weights, BERT can barely learn anything from the environment, which indicates
that model behaviour can be very different when learning in interactive
environments like Text-based games, compared to learning from the corpus for
traditional NLP tasks.

We also performed a generalisation test by evaluating trained agents on 20
unseen games. The different encoders’ performance on other games is
inconsistent Fig.6(b) shows that the Transformer agent has outstanding
performance on overlooked Coin Collector games, Fig.7(b) show that RNN
encoders perform better on unseen Treasure hunter games. In fig.8(b), except
BERT encoder, all other encoder types have similar performance. Compared to
solving a fixed set of games, agent performance on unseen games is much lower
(generally lower than 0.4), and no obvious pattern can be observed from the
evaluation scores. Solving unseen games remains a significant challenge in TBG
agent research field.

(a) Average training score

(b) Average evaluation score

Figure 6: Average scores for different encoder types in Coin Collector game

As an RL agent must explore and learn from the environment efficiently, we
also report Encoder parameters statistics. BERT has the largest parameter
size, yet it’s unable to learn effectively from all the environment settings.
The model can converge if we use a pretrained BERT and freeze most of its
layers. In this case, the model needs to be pre-trained with a prepared
corpus. We deem this a sort of word embedding and will discuss this in the
word embedding section. BERT is unsuitable for TBG tasks if additional
training corpus or pre-trained parameters are unavailable.

GRU/LSTM/Transformer encoders have similar parameters count. The Transformer
outperforms GRU and LSTM significantly. CNN has a slightly better performance
compared to Transformer, yet it has 7.7 times more parameters. The agent needs
to sample and explore the environment for complex TBG tasks to converge to the
optimal policy. Using the same training time, agents with a Transformer
encoder can explore much more states than agents with a CNN encoder. Parameter
counts are summarised in the below list.

  * •

GRU 1,255,169

  * •

LSTM 1,288,193

  * •

CNN 13,469,377

  * •

Transformer 1,741,041

  * •

BERT 109,942,017

(a) Average training score

(b) Average evaluation score

Figure 7: Average scores for different encoder types in Treasure Hunter game

(a) Average training score

(b) Average evaluation score

Figure 8: Average scores for different encoder types in Cook game

####  6.2.2 Word Embeddings

(a) Average training score

(b) Average evaluation score

Figure 9: Average scores for GRU encoder with different embedding types

(a) Average training score

(b) Average evaluation score

Figure 10: Average scores for LSTM encoder with different embedding types

(a) Average training score

(b) Average evaluation score

Figure 11: Average scores for CNN encoder with different embedding types

(a) Average training score

(b) Average evaluation score

Figure 12: Average scores for Transformer encoder with different embedding
types

Word embeddings contain external information; using word embedding means the
model is initialized with external knowledge, and it’s important to see if
this external knowledge helps model to perform better. To see which type of
work embedding is more suitable for TBG tasks, we benchmark three popular pre-
trained word embeddings and an embedding layer that is trained in the tasks.
For pre-trained embeddings, we include GLOVE, Fasttext, BERT, and see their
effect on four encoder types, including GRU, LSTM, CNN and Transformer.

In terms of fitting a large training set of games, from fig.9-12(a), we can
see that the non-pretrained task-specific embedding layer performs much better
compared to applying any pretrained word embedding. For all encoder types, the
average training score can approach close to 1.0, which means the agent can
learn to solve all games in the training set. As text-based game language is
generally less complex than traditional NLP tasks such as text classification
or translation, embedding trained from a rich corpus does not help the models
to learn faster from the simple language environment. Moreover, the non-
pretrained embedding layers learn from the rewards the agent receives, which
will be more directly related to the environment and crafted reward functions.

In the training scenario, Glove outperforms the other types among pre-trained
embeddings. For all encoder types, Glove outperforms FastText and BERT
embedding. BERT embedding is pretrained from corpus larger than Glove and
FastText, yet it performs the worst when used with GRU, LSTM and CNN encoder.
This finding suggests that language processing methods that perform well in
traditional NLP tasks could affect RL agent performance, even used in a
language environment. TBG task-specific methods need to be introduced to
improve agent performance rather than simply applying traditional NLP methods.

Different embedding also affects the TBG agent model’s generalization ability.
fig.9-12(b) presents agents performance on unseen games with different types
of word embeddings. By observing fig 9-11(b), for GRU/LSTM/CNN model, FastText
and BERT embedding outperforms other embedding types. non-pretrained and Glove
embedding perform best in training, but on unseen games, these two types
perform poorly, which indicates that they cause the agents to overfit the
training set. The different embedding types does not influence the transformer
encoder’s generalization ability a lot. In fig.12(b), all embedding types has
similar effect on evaluation score.

##  7 Discussion and Future Work

In this work we have summarized methods in Text-based games field, and
evaluated selected method on standardized agent and environment and reported
our findings. In this section we discuss on some topics which we find
important to Text-based game agent design, and offer our perspective on
challenges and future directions of Text-based game agent research field.

###  7.1 Performance versus efficiency

In our experiment, we observed that models with more parameters, like BERT,
which perform well in traditional NLP tasks such as translation and text
classification, doesn’t necessarily perform well in TBG tasks. The large
models have stronger language encoding capability, but we found that as
training the full model is inefficient and requires more complex training
techniques, and knowledge acquired from other corpus doesn’t necessarily help
the model perform better in TBG task, a middle-sized model like single layer
transformer encoder or single layer text CNN is more suitable for TBG agent.
In order to use large models like BERT, new designs and training techniques
need to be introduced in the field to empower the large models.

###  7.2 Generalization

Based on our analysis[6.2.1] and review of related works(?, ?), model’s
generalization ability remains a major challenge in this field. The models
that can easily learn to solve seen games doesn’t guarantee they perform well
in unseen game at all. For TBG agent research field, although platforms like
Textworld support game set generation, the richness and volume of games are
still not comparable to traditional NLP task datasets.For example, a model
trained on Wikipedia corpus can have 4.1 billion words with great variety in
the training set, but Textworld games training set with 100 games may only
have 50 thousands words with very limited vocabulary variations. Because of
limited vocabulary and sentence structures in the training set, better
structured state representation that generalizes to unseen conditions need to
be introduced, to improve generalization ability of the models after trained
on limited games.

###  7.3 Towards smarter Text-based game agents

Except for two issues addressed above, TBG agents are mainly trained to
evaluate language formed state-action pairs. Among a review of related
works[ref ontology] and the analysis of current methods, the agents still lack
of learning a general strategy to solve a class of games, especially
identifying key game thresholds, like finish gathering a list of different
materiel. The agents still lack of the ability to learn and extract useful
information from state and action history. The strategy-wise reward functions
introduced in the field are limited. Lots of agent designs still contain hand-
crafted rules and policies. Hand-crafted rules are also heavily used in game
state representation(e.g. graph) generation process. These obstacles remain to
be research problems in this field, in order to design a smart agent that
solves the games as efficiently, as resourceful as human.

##  8 Conclusion

In this paper, we choose some RNN, CNN, and Transformer based models to
represent popularly used models in TBG and NLP fields. The selected models
also contain different levels of total trainable parameters to represent
different levels of efficiency. We train the models using the same hyper-
parameters, with the same training games set and evaluate model game
performance both on seen and unseen games. The results show that CNN and
Transformer perform best, and Transformer has a much lower parameter count.
The models with more parameters like BERT do not deliver stronger performance.
The models do not solve all the seen games, even for easy games. For hard
games like cook, the best performing models can only solve about 50% of the
seen games. For unseen games, the models perform poorly. To reduce training
fluctuation, better training strategies need to be introduced to TBG games.
For RL algorithms like Actor-Critic, how to better apply it to TBG agent
training to improve training performance remains to be found. Current
applications of Actor-Critic in TBG agent training don’t bring significant
benefits. For harder games, better reward functions should be designed to help
the model value key thresholds to finish the games.

We standardise our environment and training procedure and remove all hand-
crafted rules, to present how the modules and word embedding types contribute
toward agent performance. The language processing modules and RL components
are also standardised. We pick games with different difficulties and form the
same training sets and generalisation evaluation sets. We see that, without
adding hand-crafted rules and other policy-related pretrained components, some
modules like the Transformer encoder and non-pretrained embedding perform
better. The modules that contain external information, like word embeddings,
do not really help the agents’ performance as expected. As applying pre-
trained modules that work in traditional NLP tasks does not necessarily
enhance the agent performance, external knowledge and pre-trained modules need
to be tuned better to fit TBG agent training.

We have chosen two evaluation types-multiple game tests and generalisation
tests to see if models can perform consistently in different evaluation types.
These two types are more meaningful for TBG agent applications. We train the
models using a set of games and evaluate them on seen and unseen game sets.
The result shows that the model’s performance in multiple and generalisation
tests has a very weak correlation. Those models that perform better on seen
games do not generalise well. The state representation is not well structured
to generalise to unseen states. Thus the state value approximation only works
for seen games. A better-structured state representation design with a smaller
feature space needs to be introduced to represent more states and improve
model generalisation performance. It’s also important to try to train the
agent to work on different game types, for example, to train the agent to
learn from Coin-collector and Treasure hunter, and examine its ability to
solve the Cooking game. The learnt knowledge should be transferable between a
larger variety of text-based games.

Through our tests analysis, we present the strengths and weaknesses of
selected deep learning components used in TBG research works. Recent
developments in the TBG agent field mainly focus on graph representation and
external modules to enhance the models’ performance. Yet the model
generalisation to unseen games and model capability remain major challenges in
this field. As a reference for future research works, new methods could be
introduced to cope with the strengths and weaknesses of the deep learning
modules analysed in our work.

Acknowledgments

The authors wish to thank Hans-Martin Adorf, Don Rosenthal, Richard Franier,
Peter Cheeseman and Monte Zweben for their assistance and advice. We also
thank Ron Musick and our anonymous reviewers for their comments. The Space
Telescope Science Institute is operated by the Association of Universities for
Research in Astronomy for NASA.

## Appendix A. Probability Distributions for N-Queens

[section ommitted]

## References

  * Adhikari et al. Adhikari, A., Yuan, X., Côté, M.-A., Zelinka, M., Rondeau, M.-A., Laroche, R., Poupart, P., Tang, J., Trischler, A., & Hamilton, W. (2020).  Learning dynamic belief graphs to generalize on text-based games.  Advances in Neural Information Processing Systems, 33.
  * Adolphs & Hofmann Adolphs, L., & Hofmann, T. (2020).  Ledeepchef deep reinforcement learning agent for families of text-based games.  In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 7342–7349.
  * Ammanabrolu & Hausknecht Ammanabrolu, P., & Hausknecht, M. (2019).  Graph constrained reinforcement learning for natural language action spaces.  In International Conference on Learning Representations.
  * Ammanabrolu & Riedl Ammanabrolu, P., & Riedl, M. (2019).  Playing text-adventure games with graph-based deep reinforcement learning.  In NAACL-HLT (1).
  * Ammanabrolu & Riedl Ammanabrolu, P., & Riedl, M. (2021).  Learning knowledge graph-based world models of textual environments.  Advances in Neural Information Processing Systems, 34, 3720–3731.
  * Ammanabrolu et al. Ammanabrolu, P., Tien, E., Hausknecht, M., & Riedl, M. O. (2020).  How to avoid being eaten by a grue: Structured exploration strategies for textual worlds.  arXiv preprint arXiv:2006.07409, 1(1).
  * Chen et al. Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017).  Reading wikipedia to answer open-domain questions..
  * Côté et al. Côté, M.-A., Kádár, Á., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M., El Asri, L., Adada, M., et al. (2018).  Textworld: A learning environment for text-based games.  In Workshop on Computer Games, pp. 41–75. Springer.
  * Hausknecht et al. Hausknecht, M., Ammanabrolu, P., Côté, M.-A., & Yuan, X. (2020).  Interactive fiction games: A colossal adventure.  In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, pp. 7903–7910.
  * Hausknecht et al. Hausknecht, M., Ammanabrolu, P., Marc-Alexandre, C., & Xingdi, Y. (2019).  Interactive fiction games: A colossal adventure.  CoRR, abs/1909.05398.
  * He et al. He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., & Ostendorf, M. (2015).  Deep reinforcement learning with an unbounded action space.  arXiv preprint arXiv:1511.04636, 5.
  * Honnibal et al. Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020).  spaCy: Industrial-strength Natural Language Processing in Python..  1(1).
  * Jack Urbanek Jack Urbanek, A. F. e. a. (2019).  Learning to speak and act in a fantasy text adventure game..
  * Jain et al. Jain, V., Fedus, W., Larochelle, H., Precup, D., & Bellemare, M. G. (2020).  Algorithmic improvements for deep reinforcement learning applied to interactive fiction..  In AAAI, pp. 4328–4336.
  * Jang et al. Jang, Y., Seo, S., Lee, J., & Kim, K.-E. (2020).  Monte-carlo planning and learning with language action value estimates.  In International Conference on Learning Representations.
  * Kenton & Toutanova Kenton, J. D. M.-W. C., & Toutanova, L. K. (2019).  Bert: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT, pp. 4171–4186.
  * Kim Kim, Y. (2014).  Convolutional neural networks for sentence classification.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746–1751, Doha, Qatar. Association for Computational Linguistics.
  * Lan et al. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019).  Albert: A lite bert for self-supervised learning of language representations.  In International Conference on Learning Representations.
  * Mikolov et al. Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013).  Efficient estimation of word representations in vector space.  arXiv preprint arXiv:1301.3781, 1(1).
  * Mikolov et al. Mikolov, T., Grave, É., Bojanowski, P., Puhrsch, C., & Joulin, A. (2018).  Advances in pre-training distributed word representations.  In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  * Murugesan et al. Murugesan, K., Atzeni, M., Kapanipathi, P., Talamadupula, K., Sachan, M., & Campbell, M. (2021).  Efficient text-based reinforcement learning by jointly leveraging state and commonsense graph representations.  In Acl-Ijcnlp 2021: The 59Th Annual Meeting Of The Association For Computational Linguistics And The 11Th International Joint Conference On Natural Language Processing, Vol 2, pp. 719–725. ASSOC COMPUTATIONAL LINGUISTICS-ACL.
  * Murugesan et al. Murugesan, K., Atzeni, M., Shukla, P., Sachan, M., Kapanipathi, P., & Talamadupula, K. (2020).  Enhancing text-based reinforcement learning agents with commonsense knowledge.  arXiv preprint arXiv:2005.00811, 1(1).
  * Nahian et al. Nahian, M. S. A., Frazier, S., Harrison, B., & Riedl, M. (2021).  Training value-aligned reinforcement learning agents using a normative prior.  arXiv preprint arXiv:2104.09469, 1(1).
  * Narasimhan et al. Narasimhan, K., Kulkarni, T. D., & Barzilay, R. (2015).  Language understanding for textbased games using deep reinforcement learning.  In In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Citeseer.
  * Pennington et al. Pennington, J., Socher, R., & Manning, C. D. (2014).  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543.
  * Radford et al. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019).  Language models are unsupervised multitask learners.  OpenAI blog, 1(8), 9.
  * Savitzky & Golay Savitzky, A., & Golay, M. J. (1964).  Smoothing and differentiation of data by simplified least squares procedures..  Analytical chemistry, 36(8), 1627–1639.
  * Simonyan & Zisserman Simonyan, K., & Zisserman, A.  Very deep convolutional networks for large-scale image recognition.  In ICLR 2015.
  * Szegedy et al. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015).  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9.
  * Vaswani et al. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017).  Attention is all you need.  In Advances in neural information processing systems, pp. 5998–6008.
  * Xu et al. Xu, Y., Fang, M., Chen, L., Du, Y., Zhou, J. T., & Zhang, C. (2020).  Deep reinforcement learning with stacked hierarchical attention for text-based games.  Advances in Neural Information Processing Systems, 33.
  * Yao et al. Yao, S., Narasimhan, K., & Hausknecht, M. (2021).  Reading and acting while blindfolded: The need for semantics in text game agents.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3097–3102.
  * Yao et al. Yao, S., Rao, R., Hausknecht, M., & Narasimhan, K. (2020).  Keep calm and explore: Language models for action generation in text-based games.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8736–8754.
  * Yin & May Yin, X., & May, J. (2019a).  Comprehensible context-driven text game playing.  In 2019 IEEE Conference on Games (CoG), pp. 1–8. IEEE.
  * Yin & May Yin, X., & May, J. (2019b).  Learn how to cook a new recipe in a new house: Using map familiarization, curriculum learning, and bandit feedback to learn families of text-based adventure games.  arXiv preprint arXiv:1908.04777, 1(1).
  * Yin et al. Yin, X., Weischedel, R., & May, J. (2020).  Learning to generalize for sequential decision making.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3046–3063.
  * Yuan et al. Yuan, X., Côté, M.-A., Fu, J., Lin, Z., Pal, C., Bengio, Y., & Trischler, A. (2019).  Interactive language learning by question answering.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2796–2813.
  * Yuan et al. Yuan, X., Côté, M.-A., Sordoni, A., Laroche, R., Combes, R. T. d., Hausknecht, M., & Trischler, A. (2018).  Counting to explore and generalize in text-based games.  In ICML 2018.
  * Zahavy et al. Zahavy, T., Haroush, M., Merlis, N., Mankowitz, D. J., & Mannor, S. (2018).  Learn what not to learn: Action elimination with deep reinforcement learning.  In Advances in Neural Information Processing Systems, pp. 3562–3573.


# Gravitational transverse-momentum distributions

Cédric Lorcé<EMAIL_ADDRESS>CPHT, CNRS, École polytechnique,
Institut Polytechnique de Paris, 91120 Palaiseau, France  Qin-Tao Song
<EMAIL_ADDRESS>CPHT, CNRS, École polytechnique, Institut Polytechnique
de Paris, 91120 Palaiseau, France  School of Physics and Microelectronics,
Zhengzhou University, Zhengzhou, Henan 450001, China

###### Abstract

We study the energy-momentum tensor of spin-$0$ and spin-$\frac{1}{2}$ hadrons
in momentum space. We parametrize this object in terms of so-called
gravitational transverse-momentum distributions, and we identify in the quark
sector the relations between the latter and the usual transverse-momentum
distributions. Focusing on particular components of the energy-momentum
tensor, we study momentum densities, flux of inertia and stress distribution
in momentum space, revealing part of the wealth of physical information that
can be gained from higher-twist transverse-momentum distributions.

##  I Introduction

The QCD energy-momentum tensor (EMT) is a key object for studying and
understanding the internal structure of hadrons Burkert:2023wzr . It is indeed
directly related to the longstanding questions of the hadron mass Ji:1994av ;
Ji:1995sv ; Yang:2018nqn ; Hatta:2018sqd ; Lorce:2017xzd ; Metz:2020vxd ;
Lorce:2021xku  and spin decompositions Jaffe:1989jz ; Ji:1996ek ;
Leader:2013jra ; Wakamatsu:2014zza ; Lorce:2021gxs . Moreover, it allows one
to investigate the mechanical properties of hadrons Polyakov:2002yz ;
Polyakov:2018zvc ; Burkert:2018bqq ; Lorce:2018egm ; Freese:2021czn . Studying
the EMT is therefore of prime importance and stands at the heart of the
physics program of the future Electron-Ion Collider in the US
AbdulKhalek:2021gbh ; AbdulKhalek:2022hcn .

Direct access to the EMT requires a gravitational probe, and is in practice
out of reach owing to the extreme weakness of gravitational interactions at
the microscopic level. Fortunately, in QCD the EMT can be probed indirectly
via electromagnetic interactions. Matrix elements of the local EMT operator
have been parametrized in terms of gravitational form factors Kobzarev:1962wt
; Pagels:1966zza ; Ji:1996ek ; Bakker:2004ib ; Cotogno:2019vjb . The latter
can then be related to generalized parton distributions (GPDs) Ji:1996ek  and
generalized distribution amplitudes Kumano:2017lhr  accessible in various
experimental processes, see e.g. Diehl:2003ny . This has been generalized to
the case of a non-local EMT operator, whose general matrix elements have been
parametrized in terms of what can be called gravitational GPDs111Strictly
speaking, the scalar functions introduced in Ref. Lorce:2015lna  correspond to
gravitational GPDs integrated over the parton longitudinal momentum, but the
general parametrization is not impacted by this integration since we
considered non-local EMT operators. Lorce:2015lna . Similar objects have later
been considered in Ref. Guo:2021aik .

While the connection between the EMT and GPDs is well established, the link
with another class of non-perturbative functions known as transverse-momentum
distributions (TMDs) Boer:1997nt  has so far been limited to the longitudinal
and transverse momentum sum rules Burkardt:2003yg ; Burkardt:2004ur ;
Lorce:2015lna ; Boer:2015vso ; Amor-Quiroz:2020qmw . The aim of the present
work is to introduce the notion of EMT distribution in momentum space and to
identify the physical information about the EMT that can be accessed via TMDs.
The paper is organized as follows. In Section II we define the transverse-
momentum dependent EMT and we parametrize the associated matrix elements in
terms of gravitational TMDs. We then discuss in Section III the connection
with the standard quark TMDs and we study in Section IV part of the physical
content that can be accessed from twist-2 and twist-3 TMDs. Finally, we
summarize our findings in Section V.

##  II Gravitational TMDs

###  II.1 TMD correlator

We start with a reminder on the TMD correlators. The fully unintegrated quark-
quark correlator for a spin-$\frac{1}{2}$ target Meissner:2009ww  is defined
in the forward limit as

$W^{[\Gamma]}(P,k,N,S;\eta)=\frac{1}{2}\int\frac{\mathrm{d}^{4}z}{(2\pi)^{4}}\,e^{ik\cdot
z}\,\langle
P,S|\overline{\psi}(-\tfrac{z}{2})\Gamma\mathcal{W}(-\tfrac{z}{2},\tfrac{z}{2}|n)\psi(\tfrac{z}{2})|P,S\rangle,$
(1)

where $\Gamma$ stands for a generic matrix in Dirac space, e.g.
$\Gamma=\gamma^{\mu},\gamma^{\mu}\gamma_{5},\cdots$. For a target of mass $M$
and four-momentum $P$, the covariant spin vector $S$ defined via
$\overline{u}(P,S)\gamma^{\mu}\gamma_{5}u(P,S)=2MS^{\mu}$ satisfies $P\cdot
S=0$ and $S^{2}=-1$. The quark average four-momentum $k$ is defined as the
Fourier conjugate variable to the space-time distance $z$ between the two
quark operators. Gauge invariance is preserved by the inclusion of a Wilson
line $\mathcal{W}$ connecting the points $-\frac{z}{2}$ and $\frac{z}{2}$ via
an infinitely long staple-shaped path along the lightlike direction $n$. Since
the same Wilson line is unchanged under the rescaling $n\mapsto\alpha n$ with
$\alpha>0$, the correlator depends in fact on the rescaling-invariant four-
vector

$N=\frac{M^{2}n}{P\cdot n}.$ (2)

The parameter $\eta=\text{sign}(n^{0})$ indicates whether the Wilson line is
future-pointing ($\eta=+1$) or past-pointing ($\eta=-1$).

For convenience, we choose the coordinate system and the rescaling factor
$\alpha$ such that

$\displaystyle P^{\mu}$
$\displaystyle=\left[P^{+},\frac{M^{2}}{2P^{+}},\bm{0}_{\perp}\right],$ (3)
$\displaystyle k^{\mu}$
$\displaystyle=\left[xP^{+},k^{-},\bm{k}_{\perp}\right],$ $\displaystyle
n^{\mu}$ $\displaystyle=\left[0,\eta,\bm{0}_{\perp}\right],$

where $v^{\mu}=[v^{+},v^{-},\bm{v}_{\perp}]$ with the light-front components
defined as $v^{\pm}=(v^{0}\pm v^{3})/\sqrt{2}$. The quark TMD correlator (see
e.g. Bacchetta:2006tn ) is then obtained by integration over the quark light-
front energy

$\displaystyle\Phi^{[\Gamma]}(P,x,\bm{k}_{\perp},N,S;\eta)$
$\displaystyle=\int\mathrm{d}k^{-}\,W^{[\Gamma]}(P,k,N,S;\eta)$ (4)
$\displaystyle=\frac{1}{2}\int\frac{\mathrm{d}z^{-}\,\mathrm{d}^{2}z_{\perp}}{(2\pi)^{3}}\,e^{ik\cdot
z}\,\langle
P,S|\overline{\psi}(-\tfrac{z}{2})\Gamma\mathcal{W}(-\tfrac{z}{2},\tfrac{z}{2}|n)\psi(\tfrac{z}{2})|P,S\rangle\Big{|}_{z^{+}=0}.$

###  II.2 Transverse-momentum dependent EMT

In QCD, the local gauge-invariant EMT operator for quarks is given by

$T^{\mu\nu}_{q}(r)=\overline{\psi}(r)\gamma^{\mu}\tfrac{i}{2}\overset{\leftrightarrow}{D}\\!\\!\\!\\!\\!\phantom{D}^{\nu}\psi(r)$
(5)

with
$\overset{\leftrightarrow}{D}\\!\\!\\!\\!\\!\phantom{D}^{\nu}=\overset{\rightarrow}{\partial}\\!\\!\\!\\!\phantom{\partial}^{\nu}-\overset{\leftarrow}{\partial}\\!\\!\\!\\!\phantom{\partial}^{\nu}-2igA^{\nu}(r)$.
In order to define the EMT for a quark with average four-momentum $k$, we need
to consider a bilocal generalization of this expression. Unfortunately, the
covariant derivative does not commute with the Wilson line, making the bilocal
generalization of Eq. (5) ambiguous Lorce:2012ce . The problem can be traced
back to the fact that $[D_{\mu},D_{\nu}]\neq 0$ whereas $[k_{\mu},k_{\nu}]=0$,
which implies that $k$ cannot be identified with the quark _kinetic_ four-
momentum. However, if we work in the gauge where the Wilson line reduces to
the identity (namely the light-front gauge with appropriate advanced of
retarded boundary conditions depending on the value of $\eta$ Belitsky:2002sm
), the four-vector $k^{\mu}$ can be represented by the partial derivatives
$i\partial^{\mu}$, and hence be interpreted as the quark _canonical_ four-
momentum. Therefore, instead of looking for the bilocal generalization of the
kinetic EMT operator (5), we should rather be looking for the bilocal
generalization of the light-front gauge-invariant canonical (gic) EMT operator
Lorce:2012rr ; Leader:2013jra ; Lorce:2015lna

$T^{\mu\nu}_{q,\text{gic}}(r)=\overline{\psi}(r)\gamma^{\mu}\tfrac{i}{2}\overset{\leftrightarrow}{D}\\!\\!\\!\\!\\!\phantom{D}^{\nu}_{\text{pure}}\psi(r),$
(6)

where $D^{\mu}_{\text{pure}}=\partial^{\mu}-igA^{\mu}_{\text{pure}}$ is known
as the pure-gauge covariant derivative Chen:2008ag ; Wakamatsu:2010cb ,
corresponding in the present context to the covariant derivative reducing in
the light-front gauge $A^{+}=0$ (with appropriate boundary conditions) to
$\partial^{\mu}$ Hatta:2011zs ; Hatta:2011ku ; Lorce:2012ce . Note that by
definition $A^{+}_{\text{pure}}(r)=A^{+}(r)$, meaning that
$T^{\mu+}_{q}(r)=T^{\mu+}_{q,\text{gic}}(r)$. Therefore, as far as the
longitudinal light-front momentum is concerned, there is no difference between
the kinetic and the gauge-invariant canonical definitions.

Following the spirit of Refs. Ji:2003ak ; Belitsky:2003nz ; Lorce:2011kd ;
Lorce:2011ni , it is natural to define the bilocal gauge-invariant canonical
(gic) EMT operator for quarks as Lorce:2012ce

$T^{\mu\nu}_{q,\text{gic}}(r,k)=k^{\nu}\int\frac{\mathrm{d}^{4}z}{(2\pi)^{4}}\,e^{ik\cdot
z}\,\overline{\psi}(r-\tfrac{z}{2})\gamma^{\mu}\mathcal{W}(r-\tfrac{z}{2},r+\tfrac{z}{2}|n)\psi(r+\tfrac{z}{2}).$
(7)

Integrating by parts, we can write

$\displaystyle T^{\mu\nu}_{q,\text{gic}}(r,k)$
$\displaystyle=\int\frac{\mathrm{d}^{4}z}{(2\pi)^{4}}\,e^{ik\cdot
z}\,i\partial^{\nu}_{z}\\!\left[\overline{\psi}(r-\tfrac{z}{2})\gamma^{\mu}\mathcal{W}(r-\tfrac{z}{2},r+\tfrac{z}{2}|n)\psi(r+\tfrac{z}{2})\right]$
(8) $\displaystyle=\int\frac{\mathrm{d}^{4}z}{(2\pi)^{4}}\,e^{ik\cdot
z}\left[\overline{\psi}(r-\tfrac{z}{2})\gamma^{\mu}\mathcal{W}(r-\tfrac{z}{2},r+\tfrac{z}{2}|n)\tfrac{i}{2}\overset{\rightarrow}{D}\\!\\!\\!\\!\\!\phantom{D}^{\nu}_{\text{pure}}(r+\tfrac{z}{2})\psi(r+\tfrac{z}{2})\right.$
$\displaystyle\qquad\qquad\qquad\quad\left.-\overline{\psi}(r-\tfrac{z}{2})\tfrac{i}{2}\overset{\leftarrow}{D}\\!\\!\\!\\!\\!\phantom{D}^{\nu}_{\text{pure}}(r-\tfrac{z}{2})\gamma^{\mu}\mathcal{W}(r-\tfrac{z}{2},r+\tfrac{z}{2}|n)\psi(r+\tfrac{z}{2})\right],$

where $A^{\mu}_{\text{pure}}$ is given by

$A^{\mu}_{\text{pure}}(r)=\mathcal{W}(r,0|n)\tfrac{i}{g}\partial^{\mu}_{r}\mathcal{W}(0,r|n).$
(9)

Since we obviously have the property

$D^{\mu}_{\text{pure}}(x)\mathcal{W}(x,y|n)=\mathcal{W}(x,y|n)D^{\mu}_{\text{pure}}(y)$
(10)

reflecting the commutativity of pure-gauge covariant derivatives, the bilocal
operator in Eq. (8) is unambiguous. Moreover, integrating over the quark four-
momentum leads to

$\int\mathrm{d}^{4}k\,T^{\mu\nu}_{q,\text{gic}}(r,k)=T^{\mu\nu}_{q,\text{gic}}(r)$
(11)

as expected.

We can now define in a natural way the fully unintegrated EMT by considering
the forward matrix element222The motivation for the factor $\frac{1}{2}$ is
the same as for the correlators in Section II.1: the light-front expectation
value of an operator $O$ is $\frac{\langle P,S|O|P,S\rangle}{2P^{+}}$ and
switching from a distribution in $k^{+}$ to a distribution in $x$ amounts to a
multiplication by the Jacobian $P^{+}$. of the operator in Eq. (8)

$\Theta^{\mu\nu}_{q}(P,k,N,S;\eta)=\frac{1}{2}\,\langle
P,S|T^{\mu\nu}_{q,\text{gic}}(0,k)|P,S\rangle,$ (12)

and the TMD EMT by further integrating over the quark light-front energy

$\displaystyle\mathcal{T}^{\mu\nu}_{q}(P,x,\bm{k}_{\perp},N,S;\eta)$
$\displaystyle=\int\mathrm{d}k^{-}\,\Theta^{\mu\nu}_{q}(P,k,N,S;\eta)$ (13)
$\displaystyle=\frac{1}{2}\int\frac{\mathrm{d}z^{-}\,\mathrm{d}^{2}z_{\perp}}{(2\pi)^{3}}\,e^{ik\cdot
z}\,i\partial_{z}^{\nu}\langle
P,S|\overline{\psi}(-\tfrac{z}{2})\gamma^{\mu}\mathcal{W}(-\tfrac{z}{2},\tfrac{z}{2}|n)\psi(\tfrac{z}{2})|P,S\rangle\Big{|}_{z^{+}=0}.$

This last object can be interpreted as the 3D distribution of the quark EMT in
momentum space.

###  II.3 Parametrization in terms of gravitational TMDs

Parity, hermiticity and time-reversal invariance imply that the fully
unintegrated EMT satisfies the relations

$\displaystyle\Theta^{\mu\nu}(k,P,N,S;\eta)$
$\displaystyle=\Theta^{\bar{\mu}\bar{\nu}}(\bar{k},\bar{P},\bar{N},-\bar{S};\eta),$
(14) $\displaystyle\Theta^{\mu\nu}(k,P,N,S;\eta)$
$\displaystyle=[\Theta^{\mu\nu}(k,P,N,S;\,\eta)]^{{\dagger}},$
$\displaystyle\Theta^{\mu\nu}(k,P,N,S;\eta)$
$\displaystyle=[\Theta^{\bar{\mu}\bar{\nu}}(\bar{k},\bar{P},\bar{N},\bar{S};-\eta)]^{*},$

with the notation $v^{\bar{\mu}}=\bar{v}^{\mu}=(v^{0},-\bm{v})$. Since the
parametrization should be the same for both quark and gluon contributions to
the EMT, we drop the label $q$ in this subsection.

For convenience, we define the transverse part of a four-vector by
$v^{\mu}_{T}=g^{\mu\nu}_{T}v_{\nu}$ using the projector onto the subspace
orthogonal to $P$ and $N$

$\displaystyle
g_{T}^{\mu\nu}=g^{\mu\nu}-\frac{P^{\mu}N^{\nu}+P^{\nu}N^{\mu}}{M^{2}}+\frac{N^{\mu}N^{\nu}}{M^{2}}.$
(15)

The covariant spin vector can then be expressed as

$\displaystyle S^{\mu}=\frac{\lambda}{M}(P^{\mu}-N^{\mu})+S_{T}^{\mu},$ (16)

where the longitudinal light-front polarization is denoted by the parameter
$\lambda$ and the transverse light-front polarization by the four-vector
$S^{\mu}_{T}=[0,0,\bm{S}_{\perp}]$. We define also the transverse Levi-Civita
pseudotensor

$\displaystyle\epsilon_{T}^{\mu\nu}=\frac{\epsilon^{\mu\nu\alpha\beta}N_{\alpha}P_{\beta}}{M^{2}}$
(17)

with the convention $\epsilon_{0123}=1$ such that $\epsilon_{T}^{12}=1$, and
we introduce the compact notation $\epsilon_{T}^{\mu
v_{T}}\equiv\epsilon_{T}^{\mu\nu}v_{T\nu}$.

A complete parametrization of the TMD EMT
$\mathcal{T}^{\mu\nu}(P,x,\bm{k}_{\perp},N,S;\eta)$ for spin-$0$ and
spin-$\frac{1}{2}$ targets can be obtained by writing down all the independent
rank-2 tensors built out of $g_{T}^{\mu\nu}$, $\epsilon_{T}^{\mu\nu}$,
$P^{\mu}$, $N^{\mu}$, and $k^{\mu}_{T}$, which are at most linear in the
polarization and which satisfy the constraints in Eq. (14). We find333Note
that other possible tensor structures have been discarded thanks to the
Schouten identity
$g^{\alpha\beta}\epsilon^{\mu\nu\rho\sigma}+g^{\alpha\mu}\epsilon^{\nu\rho\sigma\beta}+g^{\alpha\nu}\epsilon^{\rho\sigma\beta\mu}+g^{\alpha\rho}\epsilon^{\sigma\beta\mu\nu}+g^{\alpha\sigma}\epsilon^{\beta\mu\nu\rho}=0.$

$\displaystyle\mathcal{T}^{\mu\nu}=\frac{1}{P^{+}}\Big{\\{}$ $\displaystyle
P^{\mu}P^{\nu}a_{1}+N^{\mu}N^{\nu}a_{2}+k_{T}^{\mu}k_{T}^{\nu}a_{3}+P^{\mu}N^{\nu}a_{4}+N^{\mu}P^{\nu}a_{5}$
(18)
$\displaystyle+P^{\mu}k_{T}^{\nu}a_{6}+k_{T}^{\mu}P^{\nu}a_{7}+N^{\mu}k_{T}^{\nu}a_{8}+k_{T}^{\mu}N^{\nu}a_{9}+M^{2}g^{\mu\nu}_{T}a_{0}$
$\displaystyle-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\left[P^{\mu}P^{\nu}a_{1T}^{\perp}+N^{\mu}N^{\nu}a_{2T}^{\perp}+k_{T}^{\mu}k_{T}^{\nu}a_{3T}^{\perp}+P^{\mu}N^{\nu}a_{4T}^{\perp}+N^{\mu}P^{\nu}a_{5T}^{\perp}\right.$
$\displaystyle\qquad\qquad\quad\left.+P^{\mu}k_{T}^{\nu}a_{6T}^{\perp}+k_{T}^{\mu}P^{\nu}a_{7T}^{\perp}+N^{\mu}k_{T}^{\nu}a_{8T}^{\perp}+k_{T}^{\mu}N^{\nu}a_{9T}^{\perp}+M^{2}g^{\mu\nu}_{T}a_{0T}^{\perp}\right]$
$\displaystyle-M\left[P^{\mu}\epsilon_{T}^{\nu
S_{T}}a_{1T}+P^{\nu}\epsilon_{T}^{\mu S_{T}}a_{2T}+N^{\mu}\epsilon_{T}^{\nu
S_{T}}a_{3T}+N^{\nu}\epsilon_{T}^{\mu
S_{T}}a_{4T}+k_{T}^{\mu}\epsilon_{T}^{\nu
S_{T}}a_{5T}+k_{T}^{\nu}\epsilon_{T}^{\mu S_{T}}a_{6T}\right]$
$\displaystyle-\lambda\left[P^{\mu}\epsilon_{T}^{\nu
k_{T}}a_{1L}+P^{\nu}\epsilon_{T}^{\mu k_{T}}a_{2L}+N^{\mu}\epsilon_{T}^{\nu
k_{T}}a_{3L}+N^{\nu}\epsilon_{T}^{\mu
k_{T}}a_{4L}+k_{T}^{\mu}\epsilon_{T}^{\nu
k_{T}}a_{5L}+k_{T}^{\nu}\epsilon_{T}^{\mu k_{T}}a_{6L}\right]\Big{\\}},$

where the real-valued coefficients $a_{i}(x,\bm{k}_{\perp}^{2})$ will be
referred to as gravitational TMDs. There are 10 polarization-independent
gravitational TMDs (viz. $a_{0-9}$). For a spin-$0$ target, that is all we
have. For a spin-$\frac{1}{2}$ target, there are in addition 22 polarization-
dependent gravitational TMDs: 6 associated with the longitudinal polarization
(viz. $a_{1-6L}$) and 16 associated with the transverse polarization (viz.
$a_{1-6T}$ and $a_{0-9T}^{\perp}$). As a result of the discrete symmetries
(14), the polarization-independent gravitational TMDs are naive T-even (i.e.
independent of $\eta$) whereas the polarization-dependent ones are naive T-odd
(i.e. they change sign under $\eta\mapsto-\eta$). Interestingly, the same
total numbers of gravitational GPDs for spin-$0$ and spin-$\frac{1}{2}$
targets have been obtained in Ref. Lorce:2015lna . Since
$\int\mathrm{d}^{2}k_{\perp}\,\mathcal{T}^{\mu\nu}$ can not depend on
$k^{\mu}_{T}$, one may naively think by eliminating all the
$k^{\mu}_{T}$-dependent tensors in Eq. (18) that there are only 9
gravitational PDFs. Note however that the combination
$k_{T}^{\mu}\epsilon_{T}^{\nu k_{T}}-k_{T}^{\nu}\epsilon_{T}^{\mu
k_{T}}=\bm{k}_{\perp}^{2}\epsilon_{T}^{\mu\nu}$ does survive integration over
$\bm{k}_{\perp}$, meaning that there are in total 10 gravitational PDFs, in
agreement with the results in Section 4.4 of Ref. Lorce:2015lna .

##  III Relations between TMDs and gravitational TMDs

In practice, gravitational TMDs cannot be accessed directly in experiments for
the scattering amplitudes between hadrons and gravitons are extremely small.
Part of them can however be obtained indirectly through their relations with
ordinary TMDs. It is easy to see from Eqs. (1), (7) and (12) that at the level
of the fully unintegrated matrix elements we have the simple relation

$\Theta^{\mu\nu}_{q}(P,k,N,S;\eta)=k^{\nu}W^{[\gamma^{\mu}]}(P,k,N,S;\eta).$
(19)

Integrating over $k^{-}$ leads us to

$\mathcal{T}^{\mu\nu}_{q}(P,x,\bm{k}_{\perp},N,S;\eta)=k^{\nu}\Phi^{[\gamma^{\mu}]}(P,x,\bm{k}_{\perp},N,S;\eta)\qquad\text{for
}\nu\neq-.$ (20)

Let us therefore consider the quark vector TMD correlator, obtained from Eq.
(4) using $\Gamma=\gamma^{\mu}$,

$\Phi^{[\gamma^{\mu}]}(P,x,\bm{k}_{\perp},N,S;\eta)=\frac{1}{2}\int\frac{\mathrm{d}z^{-}\,\mathrm{d}^{2}z_{\perp}}{(2\pi)^{3}}\,e^{ik\cdot
z}\,\langle
P,S|\overline{\psi}(-\tfrac{z}{2})\gamma^{\mu}\mathcal{W}(-\tfrac{z}{2},\tfrac{z}{2}|n)\psi(\tfrac{z}{2})|P,S\rangle\Big{|}_{z^{+}=0}.$
(21)

Its parametrization in terms of canonical twist-2, twist-3 and twist-4 quark
TMDs reads Goeke:2005hb ; Bacchetta:2006tn

$\displaystyle\Phi^{[\gamma^{+}]}$
$\displaystyle=f_{1}-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\,f^{\perp}_{1T},$
(22) $\displaystyle\Phi^{[\gamma^{\alpha}_{T}]}$
$\displaystyle=\frac{M}{P^{+}}\left[\frac{k_{T}^{\alpha}}{M}\,f^{\perp}-\epsilon_{T}^{\alpha
S_{T}}f_{T}-\lambda\,\frac{\epsilon_{T}^{\alpha
k_{T}}}{M}\,f^{\perp}_{L}-\frac{k_{T}^{\alpha}k_{T}^{\beta}-\frac{1}{2}k_{T}^{2}g_{T}^{\alpha\beta}}{M^{2}}\,\epsilon_{T\beta
S_{T}}f^{\perp}_{T}\right],$ $\displaystyle\Phi^{[\gamma^{-}]}$
$\displaystyle=\left(\frac{M}{P^{+}}\right)^{\\!2}\left[f_{3}-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\,f^{\perp}_{3T}\right],$

reminding that $k^{2}_{T}=-\bm{k}_{\perp}^{2}$. TMDs are scale-dependent
objects444Beyond canonical twist-2, the renormalization of TMDs is troublesome
and the evolution equations are not closed, see Ref. Rodini:2022wki . that are
extracted from fits to experimental data. Their QCD evolution has been a major
focus of the past decade and is expected to play a significant role at the
future Electron-Ion Collider Angeles-Martinez:2015sea . Note that the twist-2
functions $f_{1}$ and $f_{1T}^{\perp}$ are often referred to as the
“unpolarized” TMDs in the literature.

Setting $\nu=+$ in Eq. (20), we find

$\displaystyle a_{1}$ $\displaystyle=xf_{1},$ $\displaystyle a_{1T}^{\perp}$
$\displaystyle=xf^{\perp}_{1T},$ (23) $\displaystyle\tfrac{1}{2}a_{1}+a_{5}$
$\displaystyle=xf_{3},$
$\displaystyle\qquad\tfrac{1}{2}a_{1T}^{\perp}+a_{5T}^{\perp}$
$\displaystyle=xf_{3T}^{\perp},$ $\displaystyle a_{7}$
$\displaystyle=xf^{\perp},$ $\displaystyle a_{7T}^{\perp}$
$\displaystyle=xf^{\perp}_{T},$ $\displaystyle a_{2L}$
$\displaystyle=xf^{\perp}_{L},$ $\displaystyle a_{2T}$
$\displaystyle=xf^{+}_{T},$

where $f^{\pm}_{T}=f_{T}\pm\tfrac{\bm{k}_{\perp}^{2}}{2M^{2}}\,f^{\perp}_{T}$.
Similarly, setting $\nu=i\in\\{1,2\\}$ in Eq. (20) gives

$\displaystyle a_{3}$ $\displaystyle=f^{\perp},$ $\displaystyle
a_{3T}^{\perp}$ $\displaystyle=f^{\perp}_{T},$ (24) $\displaystyle a_{6}$
$\displaystyle=f_{1},$ $\displaystyle a_{6T}^{\perp}$
$\displaystyle=f^{\perp}_{1T},$ $\displaystyle\tfrac{1}{2}a_{6}+a_{8}$
$\displaystyle=f_{3},$
$\displaystyle\tfrac{1}{2}a_{6T}^{\perp}+a_{8T}^{\perp}$
$\displaystyle=f^{\perp}_{3T},$ $\displaystyle a_{6L}$
$\displaystyle=f^{\perp}_{L},$ $\displaystyle a_{6T}$
$\displaystyle=f^{+}_{T},$ $\displaystyle a_{0}$
$\displaystyle=a_{1L}=a_{3L}=a_{5L}=0,$ $\displaystyle\qquad a_{0T}^{\perp}$
$\displaystyle=a_{1T}=a_{3T}=a_{5T}=0.$

These relations imply that the quark TMD EMT involves only 16 independent
functions

$\displaystyle\mathcal{T}^{\mu\nu}_{q}=\frac{1}{P^{+}}\Bigg{\\{}$
$\displaystyle\left[\tilde{P}^{\mu}f_{1}+k_{T}^{\mu}f^{\perp}+N^{\mu}f_{3}-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\left(\tilde{P}^{\mu}f_{1T}^{\perp}+k_{T}^{\mu}f_{T}^{\perp}+N^{\mu}f^{\perp}_{3T}\right)-M\epsilon_{T}^{\mu
S_{T}}f^{+}_{T}-\lambda\epsilon_{T}^{\mu
k_{T}}f_{L}^{\perp}\right]\tilde{k}^{\nu}$ (25)
$\displaystyle+\left[\tilde{P}^{\mu}\check{f}_{1}+k_{T}^{\mu}\check{f}^{\perp}+N^{\mu}\check{f}_{3}-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\left(\tilde{P}^{\mu}\check{f}_{1T}^{\perp}+k_{T}^{\mu}\check{f}_{T}^{\perp}+N^{\mu}\check{f}_{3T}^{\perp}\right)-M\epsilon_{T}^{\mu
S_{T}}\check{f}^{+}_{T}-\lambda\epsilon_{T}^{\mu
k_{T}}\check{f}_{L}^{\perp}\right]N^{\nu}\Bigg{\\}},$

where we introduce for convenience $\tilde{P}^{\mu}=[P^{+},0,\bm{0}_{\perp}]$
and $\tilde{k}^{\mu}=[xP^{+},0,\bm{k}_{\perp}]$. The combinations

$\displaystyle\check{f}_{1}$ $\displaystyle=a_{4}+\tfrac{x}{2}f_{1},$
$\displaystyle\check{f}_{1T}^{\perp}$
$\displaystyle=a_{4T}^{\perp}+\tfrac{x}{2}f_{1T}^{\perp},$ (26)
$\displaystyle\check{f}^{\perp}$ $\displaystyle=a_{9}+\tfrac{x}{2}f^{\perp},$
$\displaystyle\tilde{f}_{T}^{\perp}$
$\displaystyle=a_{9T}^{\perp}+\tfrac{x}{2}f_{T}^{\perp},$
$\displaystyle\check{f}_{3}$
$\displaystyle=a_{2}+\tfrac{1}{2}a_{4}+\tfrac{x}{2}f_{3},$
$\displaystyle\qquad\check{f}^{\perp}_{3T}$
$\displaystyle=a_{2T}^{\perp}+\tfrac{1}{2}a_{4T}^{\perp}+\tfrac{x}{2}f^{\perp}_{3T},$
$\displaystyle\check{f}^{\perp}_{L}$
$\displaystyle=a_{4L}+\tfrac{x}{2}f_{L}^{\perp},$
$\displaystyle\check{f}^{+}_{T}$ $\displaystyle=a_{4T}+\tfrac{x}{2}f^{+}_{T},$

parametrize the information that cannot be accessed with the ordinary quark
vector TMDs.

##  IV Mechanical properties

The interpretation of the light-front components of the EMT and the associated
distributions in impact-parameter space have been discussed in Refs.
Lorce:2018egm ; Freese:2021czn . We investigate here their momentum-space
counterparts.

###  IV.1 Densities of longitudinal and transverse momentum

Since the TMD correlator $\Phi^{[\gamma^{+}]}(P,x,\bm{k}_{\perp},N,S;\eta)$ is
interpreted as the probability density of finding a quark with three-momentum
$[xP^{+},\bm{k}_{\perp}]$, it is natural to interpret

$\displaystyle\mathcal{T}^{++}_{q}$
$\displaystyle=xP^{+}\Phi^{[\gamma^{+}]}=\left(f_{1}-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\,f^{\perp}_{1T}\right)xP^{+},$
(27) $\displaystyle\mathcal{T}^{+i}_{q}$
$\displaystyle=k^{i}_{T}\Phi^{[\gamma^{+}]}=\left(f_{1}-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\,f^{\perp}_{1T}\right)k^{i}_{T},\qquad\quad
i=1,2$

as the quark longitudinal and transverse momentum densities in momentum space.
The average quark longitudinal momentum is then obtained by integration over
the quark momentum

$\langle k^{+}\rangle_{q}=\langle
x\rangle_{q}P^{+}=\int\mathrm{d}x\,\mathrm{d}^{2}k_{\perp}\,\mathcal{T}^{++}_{q}=P^{+}\int\mathrm{d}x\,\mathrm{d}^{2}k_{\perp}\,xf_{1}.$
(28)

Similarly, the average quark transverse momentum Boer:2003cm ; Burkardt:2003yg
; Meissner:2007rx ; Amor-Quiroz:2020qmw  is given by

$\langle
k^{i}_{\perp}\rangle_{q}=\int\mathrm{d}x\,\mathrm{d}^{2}k_{\perp}\,\mathcal{T}^{+i}_{q}=\epsilon_{T}^{iS_{T}}\int\mathrm{d}x\,\mathrm{d}^{2}k_{\perp}\,\frac{\bm{k}^{2}_{\perp}}{2M}\,f_{1T}^{\perp}.$
(29)

We illustrate in Fig. 1 the two contributions to the transverse momentum
density at some fixed value of $x$ using a simple gaussian model for the
transverse momentum dependence $f(\bm{k}^{2}_{\perp})\propto
e^{-\bm{k}_{\perp}^{2}/\langle\bm{k}^{2}_{\perp}\rangle}$ with the typical
value $\langle\bm{k}^{2}_{\perp}\rangle\approx 0.6$ GeV2 for the gaussian
width Anselmino:2013lza . Since $\mathcal{T}^{+i}_{q}\propto k^{i}_{T}$, it is
natural that the transverse momentum density looks like a hedgehog. The
unpolarized contribution driven by $f_{1}$ is necessarily axially symmetric
for there is no preferred transverse direction. However, the combination of
target transverse polarization and initial/final state interactions breaks
axial symmetry. The magnitude of this effect is quantified by the Sivers
function $f^{\perp}_{1T}$ Sivers:1989cc .

Figure 1: Illustration of the contributions to the quark transverse momentum
density inside a nucleon polarized along the $x$-axis, using a simple gaussian
model for the transverse momentum dependence.

###  IV.2 Transverse flux of inertia

Because of the Galilean subgroup exhibited by the light-front coordinates, the
light-front longitudinal momentum plays the role of inertia in the transverse
plane Susskind:1967rg . The transverse flux of longitudinal momentum
$\mathcal{T}^{i+}$ can therefore be thought of as the transverse flux of
inertia, suggesting the definition of an effective quark transverse velocity
via the ratio

$v^{i}_{\perp}=\frac{\mathcal{T}^{i+}_{q}}{\mathcal{T}^{++}_{q}}.$ (30)

It is often thought that the EMT is symmetric, and hence that momentum density
$\mathcal{T}^{+i}_{q}$ equals flux of inertia $\mathcal{T}^{i+}_{q}$. In that
case, the quark transverse velocity is simply given by
$\bm{v}_{\perp}=\bm{k}_{\perp}/(xP^{+})$. In a gauge theory, velocity and
canonical momentum are however usually not parallel and we should expect in
general555Quark spin may also make the EMT asymmetric, but the antisymmetric
contribution vanishes when initial and final target momenta are the same
Leader:2013jra . $\mathcal{T}^{i+}_{q}\neq\mathcal{T}^{+i}_{q}$. Indeed, we
find that

$\mathcal{T}^{i+}_{q}=xP^{+}\Phi^{[\gamma^{i}_{T}]}=\left(xf^{\perp}-\frac{\epsilon_{T}^{k_{T}S_{T}}}{M}\,xf^{\perp}_{T}\right)k^{i}_{T}-M\epsilon_{T}^{iS_{T}}xf^{+}_{T}-\lambda\epsilon_{T}^{ik_{T}}xf^{\perp}_{L}.$
(31)

In the case of a symmetric TMD EMT, we should have

$\displaystyle xf^{\perp}$ $\displaystyle=f_{1},$ (32) $\displaystyle
xf^{\perp}_{T}$ $\displaystyle=f^{\perp}_{1T},$ $\displaystyle f^{+}_{T}$
$\displaystyle=f^{\perp}_{L}=0.$

Interestingly, the first relation was found in Ref. Lorce:2014hxa  using the
free quark equation of motion. In QCD, these relations are not expected to
hold in general and their violations are a direct measure of the interaction
between quarks and gluons.

Decomposing $\epsilon_{T}^{iS_{T}}$ onto components parallel and orthogonal to
$k^{i}_{\perp}$, we can rewrite Eq. (31) as

$\mathcal{T}^{i+}_{q}=\left(xf^{\perp}-\frac{M\epsilon_{T}^{k_{T}S_{T}}}{k^{2}_{T}}\,xf^{-}_{T}\right)k^{i}_{T}-\left(\lambda\,xf^{\perp}_{L}+\frac{M(k_{T}\cdot
S_{T})}{k^{2}_{T}}\,xf^{+}_{T}\right)\epsilon_{T}^{ik_{T}}.$ (33)

The first two terms have the same structure as $\mathcal{T}^{+i}_{q}$ in Eq.
(27) and lead to similar hedgehog distributions as in Fig. 1. The last two
terms indicate that besides modifying the magnitude of the quark velocity, QCD
interactions can also modify its direction relative to $\bm{k}_{\perp}$. The
corresponding distributions are illustrated in Fig. 2 using the same simple
gaussian model as for $\mathcal{T}^{+i}_{q}$.

Figure 2: Illustration of two contributions to the quark transverse flux of
inertia inside a nucleon polarized along the $z$-axis (left panel) or the
$x$-axis (right panel), using a simple gaussian model for the transverse
momentum dependence. The other two contributions are similar to those given in
Fig. 1.

###  IV.3 Transverse pressure and shear forces

The notions of 2D spatial distributions of pressure (or isotropic stress)
$\sigma$ and shear forces (or pressure anisotropy) $\Pi$ have been introduced
in Ref. Lorce:2018egm . Similarly, we introduce here the distributions of
transverse pressure and shear forces in momentum space

$\mathcal{T}^{ij}=-g^{ij}_{T}\,\sigma+\left(\frac{1}{2}\,g^{ij}_{T}-\frac{k^{i}_{T}k^{j}_{T}}{k^{2}_{T}}\right)\Pi+\frac{k^{i}_{T}\epsilon^{jk_{T}}+k^{j}_{T}\epsilon^{ik_{T}}}{2k^{2}_{T}}\,\Pi^{S}+\epsilon^{ij}_{T}\,\Pi^{A}.$
(34)

The first two transverse tensors are similar to those found in position space,
with transverse momentum $\bm{k}_{\perp}$ replacing impact parameter
$\bm{b}_{\perp}$. The last two transverse tensors are new, and are allowed
provided that $\Pi_{S}$ and $\Pi_{A}$ are linear in the target polarization
and naive T-odd. Using the particular structure of the quark EMT

$\displaystyle\mathcal{T}^{ij}_{q}=k^{j}_{T}\Phi^{[\gamma^{i}_{T}]}$
$\displaystyle=\tfrac{k^{j}_{T}}{xP^{+}}\,\mathcal{T}^{i+}_{q}$ (35)
$\displaystyle=\frac{1}{P^{+}}\left[\left(f^{\perp}-\frac{M\epsilon_{T}^{k_{T}S_{T}}}{k^{2}_{T}}\,f^{-}_{T}\right)k^{i}_{T}k^{j}_{T}-\left(\lambda\,f^{\perp}_{L}+\frac{M(k_{T}\cdot
S_{T})}{k^{2}_{T}}\,f^{+}_{T}\right)\epsilon_{T}^{ik_{T}}k^{j}_{T}\right],$

we find

$\displaystyle\sigma_{q}$
$\displaystyle=\tfrac{1}{2}\Pi_{q}=-\frac{1}{2P^{+}}\left[k^{2}_{T}f^{\perp}-M\epsilon^{k_{T}S_{T}}f^{-}_{T}\right],$
(36) $\displaystyle\Pi^{A}_{q}$
$\displaystyle=\tfrac{1}{2}\Pi^{S}_{q}=-\frac{1}{2P^{+}}\left[\lambda\,k^{2}_{T}f^{\perp}_{L}+M(k_{T}\cdot
S_{T})f^{+}_{T}\right].$

This is to be compared with the free quark case given by
$\mathcal{T}^{ij}_{q,\text{free}}=f_{1}\,k^{i}_{T}k^{j}_{T}/(xP^{+})$.

##  V Summary

The energy-momentum tensor is a fundamental object in any relativistic field
theory. In hadronic physics, it provides key information about quark and gluon
contributions to the nucleon mass and spin, and is therefore at the heart of
the physics program of the forthcoming Electron-Ion Collider in the US.

In this work we introduced the concept of energy-momentum tensor distribution
in momentum space. In the case of a spin-$0$ target, we found that this
distribution can be parametrized in terms of 10 independent gravitational
transverse-momentum distributions. For a spin-$\frac{1}{2}$ target, we
obtained in general 32 independent functions. Due to the particular structure
of the gauge-invariant canonical energy-momentum tensor for quarks, this
number reduces to 16, half of which can directly be expressed in terms of the
usual quark vector transverse-momentum distributions. A similar analysis can
in principle be applied to the gluon energy-momentum tensor, but is left for a
future dedicated investigation. We discussed the physical interpretation of
various components of the energy-momentum tensor and we used a simple gaussian
model for illustration. We observed in particular that the stress tensor
distribution in momentum space is expected to be asymmetric due to spin-
dependent contributions associated with initial/final-state interactions.

At the present stage, only a few gravitational transverse-momentum
distributions can be extracted from actual experiments. Our work provides
however new motivations for studying and measuring higher-twist transverse-
momentum distributions. In the meantime, it will be interesting to investigate
these gravitational transverse-momentum distributions within other approaches,
such as Lattice QCD and model calculations.

## Acknowledgements

We thank Simone Rodini for drawing our attention to recent developments
regarding the status of higher-twist transverse-momentum distributions. Qin-
Tao Song was supported by the National Natural Science Foundation of China
under Grant Number 12005191 and the China Scholarship Council for visiting
École polytechnique.

## References

  * (1) V. D. Burkert, L. Elouadrhiri, F. X. Girod, C. Lorcé, P. Schweitzer and P. E. Shanahan, [arXiv:2303.08347 [hep-ph]].
  * (2) X. D. Ji, Phys. Rev. Lett. 74, 1071-1074 (1995).
  * (3) X. D. Ji, Phys. Rev. D 52, 271-281 (1995).
  * (4) Y. B. Yang, J. Liang, Y. J. Bi, Y. Chen, T. Draper, K. F. Liu and Z. Liu, Phys. Rev. Lett. 121, no.21, 212001 (2018).
  * (5) Y. Hatta, A. Rajan and K. Tanaka, JHEP 12, 008 (2018).
  * (6) C. Lorcé, Eur. Phys. J. C 78, no.2, 120 (2018).
  * (7) A. Metz, B. Pasquini and S. Rodini, Phys. Rev. D 102, 114042 (2020).
  * (8) C. Lorcé, A. Metz, B. Pasquini and S. Rodini, JHEP 11, 121 (2021).
  * (9) R. L. Jaffe and A. Manohar, Nucl. Phys. B 337, 509-546 (1990).
  * (10) X. D. Ji, Phys. Rev. Lett. 78, 610-613 (1997).
  * (11) E. Leader and C. Lorcé, Phys. Rept. 541, no.3, 163-248 (2014).
  * (12) M. Wakamatsu, Int. J. Mod. Phys. A 29, 1430012 (2014).
  * (13) C. Lorcé, Eur. Phys. J. C 81, no.5, 413 (2021).
  * (14) M. V. Polyakov, Phys. Lett. B 555, 57-62 (2003).
  * (15) M. V. Polyakov and P. Schweitzer, Int. J. Mod. Phys. A 33, no.26, 1830025 (2018).
  * (16) V. D. Burkert, L. Elouadrhiri and F. X. Girod, Nature 557, no.7705, 396-399 (2018).
  * (17) C. Lorcé, H. Moutarde and A. P. Trawiński, Eur. Phys. J. C 79, no.1, 89 (2019).
  * (18) A. Freese and G. A. Miller, Phys. Rev. D 103, 094023 (2021).
  * (19) R. Abdul Khalek, A. Accardi, J. Adam, D. Adamiak, W. Akers, M. Albaladejo, A. Al-bataineh, M. G. Alexeev, F. Ameli and P. Antonioli, et al. Nucl. Phys. A 1026, 122447 (2022).
  * (20) R. Abdul Khalek, U. D’Alesio, M. Arratia, A. Bacchetta, M. Battaglieri, M. Begel, M. Boglione, R. Boughezal, R. Boussarie and G. Bozzi, et al. [arXiv:2203.13199 [hep-ph]].
  * (21) I. Y. Kobzarev and L. B. Okun, Zh. Eksp. Teor. Fiz. 43, 1904-1909 (1962).
  * (22) H. Pagels, Phys. Rev. 144, 1250-1260 (1966).
  * (23) B. L. G. Bakker, E. Leader and T. L. Trueman, Phys. Rev. D 70, 114001 (2004).
  * (24) S. Cotogno, C. Lorcé, P. Lowdon and M. Morales, Phys. Rev. D 101, no.5, 056016 (2020).
  * (25) S. Kumano, Q. T. Song and O. V. Teryaev, Phys. Rev. D 97, no.1, 014020 (2018).
  * (26) M. Diehl, Phys. Rept. 388, 41-277 (2003).
  * (27) C. Lorcé, JHEP 08, 045 (2015).
  * (28) Y. Guo, X. Ji and K. Shiells, Nucl. Phys. B 969, 115440 (2021).
  * (29) D. Boer and P. J. Mulders, Phys. Rev. D 57, 5780-5786 (1998).
  * (30) M. Burkardt, Phys. Rev. D 69, 057501 (2004).
  * (31) M. Burkardt, Phys. Rev. D 69, 091501 (2004).
  * (32) D. Boer, C. Lorcé, C. Pisano and J. Zhou, Adv. High Energy Phys. 2015, 371396 (2015).
  * (33) D. A. Amor-Quiroz, M. Burkardt, W. Focillon and C. Lorcé, Eur. Phys. J. C 81, no.7, 589 (2021).
  * (34) S. Meissner, A. Metz and M. Schlegel, JHEP 08, 056 (2009).
  * (35) A. Bacchetta, M. Diehl, K. Goeke, A. Metz, P. J. Mulders and M. Schlegel, JHEP 02, 093 (2007).
  * (36) C. Lorcé, Phys. Lett. B 719, 185-190 (2013).
  * (37) A. V. Belitsky, X. Ji and F. Yuan, Nucl. Phys. B 656, 165-198 (2003).
  * (38) C. Lorcé, Phys. Rev. D 87, no.3, 034031 (2013).
  * (39) X. S. Chen, X. F. Lu, W. M. Sun, F. Wang and T. Goldman, Phys. Rev. Lett. 100, 232002 (2008).
  * (40) M. Wakamatsu, Phys. Rev. D 83, 014012 (2011).
  * (41) Y. Hatta, Phys. Rev. D 84, 041701 (2011).
  * (42) Y. Hatta, Phys. Lett. B 708, 186-190 (2012).
  * (43) X. d. Ji, Phys. Rev. Lett. 91, 062001 (2003).
  * (44) A. V. Belitsky, X. d. Ji and F. Yuan, Phys. Rev. D 69, 074014 (2004).
  * (45) C. Lorcé and B. Pasquini, Phys. Rev. D 84, 014015 (2011).
  * (46) C. Lorcé, B. Pasquini, X. Xiong and F. Yuan, Phys. Rev. D 85, 114006 (2012).
  * (47) K. Goeke, A. Metz and M. Schlegel, Phys. Lett. B 618, 90-96 (2005).
  * (48) S. Rodini and A. Vladimirov, JHEP 08, 031 (2022) [erratum: JHEP 12, 048 (2022)].
  * (49) R. Angeles-Martinez, A. Bacchetta, I. I. Balitsky, D. Boer, M. Boglione, R. Boussarie, F. A. Ceccopieri, I. O. Cherednikov, P. Connor and M. G. Echevarria, et al. Acta Phys. Polon. B 46, no.12, 2501-2534 (2015).
  * (50) D. Boer, P. J. Mulders and F. Pijlman, Nucl. Phys. B 667, 201-241 (2003).
  * (51) S. Meissner, A. Metz and K. Goeke, Phys. Rev. D 76, 034002 (2007).
  * (52) M. Anselmino, M. Boglione, J. O. Gonzalez Hernandez, S. Melis and A. Prokudin, JHEP 04, 005 (2014).
  * (53) D. W. Sivers, Phys. Rev. D 41, 83 (1990).
  * (54) L. Susskind, Phys. Rev. 165, 1535-1546 (1968).
  * (55) C. Lorcé, B. Pasquini and P. Schweitzer, JHEP 01, 103 (2015).