# Learning Pareto Set for Multi-Objective Continuous Robot Control

Tianye Shu1 Ke Shang∗1,2 Cheng Gong1,3 Yang Nan1&Hisao Ishibuchi∗1 1Department
of Computer Science and Engineering, Southern University of Science and
Technology
2National Engineering Laboratory for Big Data System Computing Technology,
Shenzhen University
3Department of Computer Science, City University of Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{12150059<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

For a control problem with multiple conflicting objectives, there exists a set
of Pareto-optimal policies called the Pareto set instead of a single optimal
policy. When a multi-objective control problem is continuous and complex,
traditional multi-objective reinforcement learning (MORL) algorithms search
for many Pareto-optimal deep policies to approximate the Pareto set, which is
quite resource-consuming. In this paper, we propose a simple and resource-
efficient MORL algorithm that learns a continuous representation of the Pareto
set in a high-dimensional policy parameter space using a single hypernet. The
learned hypernet can directly generate various well-trained policy networks
for different user preferences. We compare our method with two state-of-the-
art MORL algorithms on seven multi-objective continuous robot control
problems. Experimental results show that our method achieves the best overall
performance with the least training parameters. An interesting observation is
that the Pareto set is well approximated by a curved line or surface in a
high-dimensional parameter space. This observation will provide insight for
researchers to design new MORL algorithms.

††footnotetext: *Corresponding authors.

##  1 Introduction

In many real-world control problems, we often have multiple conflicting
objectives. For example, in walking robot control, we may need to maximize
walking speed and minimize energy consumption. Instead of a single optimal
control policy, such a multi-objective control problem has a set of Pareto-
optimal control policies called the Pareto set. Each Pareto-optimal policy in
the Pareto set corresponds to a different trade-off (i.e., a different user
preference) over conflicting objectives. To find these Pareto-optimal
policies, multi-objective reinforcement learning (MORL) algorithms are widely
used Roijers et al. (2013); Hayes et al. (2022); Rame et al. (2023).

Many MORL algorithms Van Moffaert and Nowé (2014); Parisi et al. (2014); Xu et
al. (2020) search for a finite set of independent policies to approximate the
Pareto set. These algorithms have two potential drawbacks when handling
continuous multi-objective control problems. One drawback is that a finite
number of policies cannot represent the entire Pareto set. Thus, the following
undesirable situation can happen: No policy in the obtained finite policy set
is close to the user preference. The other drawback is that these methods are
memory-consuming when the control problem has many objectives. This is because
the number of required policies to approximate the Pareto set increases
exponentially as the number of objectives increases. For example, more than
1,000 Pareto-optimal deep policies (represented by deep neural networks) are
stored by an evolutionary learning algorithm for a three-objective robot
control problem Xu et al. (2020).

One idea to address these drawbacks is to train a single deep neural network
to represent the whole Pareto set Pirotta et al. (2015); Yang et al. (2019);
Basaklar et al. (2023). Based on this idea, two different types of methods
have been proposed: embedding-based method and manifold-based method. The
embedding-based methods learn a generalized policy whose output is conditioned
on the input preference Yang et al. (2019); Abels et al. (2019); Basaklar et
al. (2023). These algorithms are usually off-policy algorithms with a
specifically designed experience replay strategy. The Bellman equation is
extended into a multi-objective version by including the user preference Yang
et al. (2019); Basaklar et al. (2023). To improve the sample efficiency,
Alegre et al. Alegre et al. (2023) proposed a novel preference selection
strategy based on Generalized Policy Improvement (GPI) Barreto et al. (2020).
The manifold-based methods learn a parametric function to directly approximate
the Pareto set as a manifold in the policy parameter space. For example,
Pirotta et al. Pirotta et al. (2015) proposed a gradient-based approach called
PMGA. Parisi et al. Parisi et al. (2017) improved PMGA by using the importance
sampling technique Owen and Zhou (2000). However, these manifold-based
approaches have not yet been applied to deep policies with tens of thousands
of parameters. Two difficulties have been pointed out for these manifold-based
methods in the literature: (1) The number of required parameters to represent
the Pareto set is much more than (e.g., quadratically increases with) the
dimensionality of the parameter space Chen et al. (2019); Basaklar et al.
(2023), and (2) the Pareto set in the parameter space cannot be efficiently
represented by a single continuous policy family Xu et al. (2020).

In this paper, we propose a manifold-based approach called Hyper-MORL to learn
the Pareto sets (i.e., to search for Pareto-optimal deep policies) of multi-
objective continuous control problems. Our assumption is that the Pareto set
in a high-dimensional parameter space could be well approximated by a
continuous manifold in a low-dimensional subspace. This assumption comes from
the following property of multi-objective optimization: In general, the Pareto
set of an $m$-objective problem is an ($m-1$)-dimensional manifold in an
$n$-dimensional parameter space even if $n\gg m$ Hillermeier (2001). Based on
this assumption, we propose an on-policy MORL algorithm to learn a hypernet
that maps a user preference (represented by a direction vector) to a Pareto-
optimal policy (represented by a point) in the parameter subspace. We examine
our method on several complex multi-objective robot control problems Xu et al.
(2020) which are usually solved by deep policies with tens of thousands of
parameters. Compared with two state-of-the-art MORL algorithms Xu et al.
(2020); Basaklar et al. (2023), our method shows the best overall performance
with the least training parameters. To validate our initial assumption, we
investigate the Pareto set learned by our method. We show that the Pareto set
of continuous robot control problems can be well approximated by a single
continuous policy family.

##  2 Background

###  2.1 Multi-Objective Markov Decision Process

A multi-objective RL problem (e.g., multi-objective continuous robot control
problem) is usually modeled as a multi-objective Markov decision process
(MOMDP). An MOMDP is described by the tuple $\langle
S,A,P,\bm{R},\gamma\rangle$ where $S$, $A$, $P(s^{\prime}|s,a)$, $\bm{R}$ and
$\gamma$ represent state space, action space, probability transition function,
reward function and discount factor, respectively. Here the reward function
$\bm{R}$: $S\times A\rightarrow\mathbb{R}^{m}$ returns a reward vector
$\bm{r}=(r_{1},r_{2},...,r_{m})$ where $m$ is the number of objectives.

For an MOMDP, a policy $\pi(a|s)$ represents the probability of selecting the
action $a\in A$ at the state $s\in S$. In each time step $t$, an agent with
the policy $\pi$ receives a state $s^{(t)}$ and selects an action
$a^{(t)}\sim\pi(a|s^{(t)})$. Then, a reward vector
$\bm{r}^{(t+1)}=\bm{R}(s^{(t)},a^{(t)})$ and the next state $s^{(t+1)}\sim
P(s|s^{(t)},a^{(t)})$ are given by the MOMDP. In this manner, we can obtain a
trajectory
$\tau=(s^{(0)},a^{(0)},\bm{r}^{(1)},...,s^{(T-1)},a^{(T-1)},\bm{r}^{(T)})$
where $s^{(0)}$ is the initial state and $T$ is the time horizon. The expected
returns $\bm{J}(\pi)=(J_{1}(\pi),J_{2}(\pi),...,J_{m}(\pi))$ is defined as:

$J_{i}(\pi)=E_{\tau}[\sum_{t=1}^{T}\gamma^{t-1}r^{(t)}_{i}],\>i=1,\>2,\>...,\>m,$
(1)

where $r^{(t)}_{i}$ is the $i$-th component of the reward vector
$\bm{r}^{(t)}$.

Our target is to find a set of Pareto-optimal policies (called Pareto set) to
maximize $\bm{J}$. Here the optimality is defined by the Pareto dominance
relation as follows.

######  Definition1 ((Pareto Dominance Relation)).

A policy $\pi$ is said to be dominated by another policy $\pi^{\prime}$ (i.e.,
$\pi^{\prime}\succ\pi$) iff $\forall i\in\\{1,...,m\\}$,
$J_{i}(\pi^{\prime})\geq J_{i}(\pi)$ and $\exists j\in\\{1,...,m\\}$,
$J_{j}(\pi^{\prime})>J_{j}(\pi)$.

In this paper, we assume a policy $\pi(a|s,\theta)$ ($\pi_{\theta}$ for short)
is parameterized by ${\theta}\in{\Theta}\subseteq\mathbb{R}^{n}$ where
${\Theta}$ is the parameter space. The Pareto set $PS(\Theta)$ in the
parameter space is defined as

$PS(\Theta)=\\{\theta\in\Theta|\nexists\theta^{\prime}\in\Theta,\pi_{\theta^{\prime}}\succ\pi_{\theta}\\}.$
(2)

The image of $PS(\Theta)$ in the objective space is Pareto front.

###  2.2 Hypernets

Hypernets are a type of neural networks that take a context vector as input
and generate the parameters for the target neural networks Chauhan et al.
(2023). Compared with traditional neural networks, hypernets can be trained in
a single-model manner to generate weights for multiple neural networks for
solving related tasks von Oswald et al. (2020). Due to its parameter-
efficiency and expressiveness Galanti and Wolf (2020), hypernets have been
used in many fields such as reinforcement learning Sarafian et al. (2021). A
task-conditioned hypernet is used for generalization across tasks in meta-RL
Beck et al. (2023). Besides, hypernets have been used in continual RL Huang et
al. (2021) and zero-shot learning Rezaei-Shoshtari et al. (2023) in RL.

Recently, hypernets have been used to learn the Pareto set for many multi-
objective problems such as multi-objective combinatorial optimization Lin et
al. (2022) and multi-task learning Navon et al. (2021); Lin et al. (2019). In
this paper, we use a hypernet to learn the Pareto set $PS(\Theta)$ of an
MOMDP.

##  3 Proposed Method: Hyper-MORL

To solve a multi-objective RL problem, we propose an efficient manifold-based
algorithm called Hyper-MORL.

###  3.1 Problem Decomposition

For multi-objective optimization, decomposition is a commonly-used idea Zhang
and Li (2007). A multi-objective RL (MORL) problem can be decomposed into
several single-objective subproblems using a preference set
$\\{\bm{\omega}^{(1)},\bm{\omega}^{(2)},...,\bm{\omega}^{(k)}\\}$ and a
scalarization function $f$. Each subproblem with preference $\bm{\omega}$ is a
single-objective RL problem $SORL(\bm{\omega})$ which aims to maximize a
scalar expected return $f(\bm{J},\bm{\omega})$.

The scalarization function $f$ has two types: linear and non-linear. The
linear scalarization function
$f_{LS}(\bm{J},\bm{\omega})=\bm{\omega}^{T}\bm{J}$ is straightforward and
widely used in many MORL algorithms Parisi et al. (2014); Xu et al. (2020).
However, one limitation of $f_{LS}$ is that it can only find a convex hull,
which is a subset of the Pareto set Hayes et al. (2022). Non-linear
scalarization functions can find the whole Pareto set regardless of the Pareto
front shape. However, the use of non-linear scalarization functions in RL is
difficult since the assumed additive property in the Bellman equation does not
hold for the non-linearly scalarized returns Roijers et al. (2018); Hayes et
al. (2022).

In this paper, we consider the linear scalarization function since Lu et al.
Lu et al. (2023) theoretically revealed that a MORL problem is convex (i.e.,
the range of the value function is convex) when all stationary policies are
considered. Given a preference space
$\Omega=\\{\bm{\omega}|\bm{\omega}\in(\\{0\\}\cup\mathbb{R}^{+})^{m},\sum_{i=1}^{m}\omega_{i}=1\\}$,
the following proposition holds.

######  Proposition1 ( Lu et al. (2023)).

For a convex MORL problem, a policy $\pi\in PS(\Theta)$ iff
$\exists\bm{\omega}\in\Omega$, $\pi$ is optimal for the problem
$SORL(\bm{\omega})$ which maximizes $\bm{\omega}^{T}\bm{J}(\pi)$.

Proposition 1 says that we can find the whole Pareto set by solving the
problem $SORL(\bm{\omega})$ for each preference $\bm{\omega}\in\Omega$.

###  3.2 Pareto Set Representation via Hypernet

The basic idea of Hyper-MORL is shown in Figure 1. To represent the Pareto
set, we train a model to map each preference $\bm{\omega}\in\Omega$ to the
corresponding Pareto-optimal policy parameter $\bm{\theta}\in\Theta$ which
maximizes $\bm{\omega}^{T}\bm{J}(\bm{\theta})$. To reduce the number of the
required parameters, we consider a reduced $d$-dimensional subspace
$\Theta_{sub}$ instead of the original $n$-dimensional parameter space
$\Theta$ to represent the Pareto set ($d\ll n$). The value of $d$ is specified
as 10 in this paper (and some other values are also examined for sensitivity
analysis). A hypernet $\mathcal{H}_{\bm{\varphi}}$: $\Omega\rightarrow\Theta$
is defined as:

$\mathcal{H}_{\bm{\varphi}}(\bm{\omega})=\bm{W}f_{\bm{\mu}}(\bm{\omega})+\bm{b}.$
(3)

Here $f_{\bm{\mu}}$ maps the preference $\bm{\omega}$ to a $d$-dimensional
vector. The parameters $\bm{W}$ and $\bm{b}$ are an $n\times d$ matrix and an
$n$-dimensional vector, respectively. The row vectors of $\bm{W}$ spans a
$d$-dimensional parameter subspace $\Theta_{sub}$. The vector
$f_{\bm{\mu}}(\bm{\omega})$ is linearly transformed from $\Theta_{sub}$ to
$\Theta$. The learning parameters $\bm{\varphi}$ of the hypernet
$\mathcal{H}_{\bm{\varphi}}$ are $\\{\bm{W},\bm{\mu},\bm{b}\\}$. A detailed
illustration of $\mathcal{H}_{\bm{\varphi}}$ is included in
Appendix111https://github.com/HisaoLabSUSTC/Hyper-MORL/blob/
main/Appendix.pdf B.

Figure 1: Basic idea of Hyper-MORL to approximate the Pareto set in the
$n$-dimensional parameter space.

###  3.3 Pareto Set Learning via Policy Gradient

To efficiently learn the Pareto set, a gradient-based algorithm is proposed
for Hyper-MORL. As shown in Algorithm 1, the learning algorithm contains warm-
up stage and Pareto set learning stage. As shown in lines 1-2, $\alpha\times
100\%$ computation load is assigned to the warm-up stage where $\alpha$ is a
parameter. In the following part, we will describe these two stages.

Input : Hypernet $\mathcal{H}_{\bm{\varphi}}$: $\Omega\rightarrow\Theta$ with
parameters $\bm{\varphi}=\\{\bm{W},\bm{\mu},\bm{b}\\}$, the number of sampled
preferences $K$, the number of available environment steps $T$, parameter
$\alpha$

1 $G_{W}=\lfloor\frac{\alpha T}{T_{tra}}\rfloor$;

2 $G_{PSL}=\lfloor\frac{(1-\alpha)T}{KT_{tra}}\rfloor$; // $T_{tra}$ is the
number of consumed environment steps to collect a trajectory

3 // Warm-up Stage

4 $\bm{W}\leftarrow\bm{0}$;

5 Set $\bm{\mu}$ and $\bm{b}$ by an arbitrary initialization scheme;

6 $\bm{\omega}=(\frac{1}{m},...,\frac{1}{m})$;

7 for  _$g\leftarrow 1$ to $G_{W}$_ do

8 Run policy $\pi_{\bm{b}}$ in environment and collect a trajectory $\tau$;

9 Calculate gradients
$\bm{g}\leftarrow\nabla_{\bm{b}}[\bm{\omega}^{T}\bm{J}(\bm{b})]$;

10 Update parameters $\bm{b}\leftarrow\textbf{ADAM}(\bm{b},\bm{g})$;

11

12Reset exploration-related parameters in $\bm{b}$;

13 // Pareto Set Learning Stage

14 for  _$g\leftarrow 1$ to $G_{PSL}$_ do

15 Sample $K$ preferences $\bm{\omega}^{(1)}$, …, $\bm{\omega}^{(K)}$ from
$\Omega$;

16 for  _$i\leftarrow 1$ to $K$_ do

17 Run policy $\pi_{\mathcal{H}_{\bm{\varphi}}(\bm{\omega}^{(i)})}$ in
environment and collect a trajectory $\tau^{(i)}$;

18

19 for  _$i\leftarrow 1$ to $K$_ do

20 Calculate gradients
$\bm{g}^{(i)}\leftarrow\nabla_{\bm{\varphi}}[\bm{\omega}^{(i)T}\bm{J}(\mathcal{H}_{\bm{\varphi}}(\bm{\omega}^{(i)}))]$

21 Update parameters
$\bm{\varphi}\leftarrow\textbf{ADAM}(\bm{\varphi},\frac{1}{K}\sum_{i=1}^{K}\bm{g}^{(i)})$;

22

23

Algorithm 1 Learning Pareto Set via Policy Gradient

####  3.3.1 Warm-up Stage

The warm-up stage is to find a good initial policy close to the Pareto front.
By starting from a good initial policy instead of a random policy, the Pareto
set learning stage can be accelerated. In Algorithm 1, the warm-up stage is in
lines 3-11. In lines 4-5, we use the Bias-HyperInit method Beck et al. (2023)
to initialize the hypernet parameters. In lines 6-10, we update parameters
$\bm{b}$ to optimize all objectives simultaneously with the preference
$(\frac{1}{m},...,\frac{1}{m})$ ($m$ is the number of objectives). The multi-
objective policy gradient method Xu et al. (2020) is used as follows:

$\begin{split}\nabla_{\bm{\theta}}&\left[\bm{\omega}^{T}\bm{J}(\bm{\theta})\right]=\sum_{i=1}^{m}\left[\omega_{i}\nabla_{\bm{\theta}}J_{i}(\bm{\theta})\right]\\\
&=\mathbb{E}\left[\sum_{t=0}^{T}\left[\bm{\omega}\bm{A}(s_{t},a_{t})\nabla_{\bm{\theta}}\log{\pi_{\bm{\theta}}(a_{t}|s_{t})}\right]\right],\end{split}$
(4)

where $\bm{A}$ is the advantage function estimated from a sampled trajectory
$\tau$.

After $G_{W}$ iterations, we obtain a policy $\pi_{\bm{b}}$ from the
preference $(\frac{1}{m},...,\frac{1}{m})$. In our implementation, the policy
$\pi_{\bm{b}}$ samples an action from a normal distribution. Some values in
the $n$-dimensional vector $\bm{b}$ represent the standard deviation of the
output action. In line 11, the exploration-related parameters (i.e., the
standard deviation of the output action) are reset as one. Since other values
in $\bm{b}$ are not changed, the mean of the output action is preserved. The
reset step is to prevent the output action selection from being almost
deterministic (i.e., standard deviation from being almost zero) which severely
degrades the exploration ability in the Pareto set learning stage. After the
warm-up stage, $\bm{b}$ becomes closer to the Pareto set, and $\bm{W}$ remains
$\bm{0}$. From Eq. (3), the hypernet
$\mathcal{H}_{\bm{\varphi}}(\bm{\omega})=\bm{b}$ for
$\forall\bm{\omega}\in\Omega$. This means that the initial values of $\bm{b}$
are shared among all subproblems.

####  3.3.2 Pareto Set Learning Stage

Our task is to find the optimal parameters $\bm{\varphi}$ (i.e., $\bm{W}$,
$\bm{\mu}$ and $\bm{b}$) which maximizes
$\bm{\omega}^{T}\bm{J}(\mathcal{H}_{\bm{\varphi}}(\bm{\omega}))$ for each
$\bm{\omega}\in\Omega$. The objective function can be written as:

$\mathbb{E}_{\bm{\omega}\sim\Omega}\left[\bm{\omega}^{T}\bm{J}(\mathcal{H}_{\bm{\varphi}}(\bm{\omega}))\right].$
(5)

It is difficult to directly optimize the parameters $\bm{\varphi}$ to maximize
the exception in Eq. (5). Thus, we use Monte Carlo method to sample the
preferences, and use the policy gradient method to optimize the parameters
$\bm{\varphi}$. In Algorithm 1, lines 13-19 show the Pareto set learning
stage. In each iteration, $K$ preferences are randomly sampled from the
preference space $\Omega$. Then, we obtain $K$ policies by inputting these $K$
preferences into the hypernet. In lines 15-16, each policy interacts with the
environment and generates a trajectory. In lines 17-18, the gradient
$\bm{g}^{(i)}$ is calculated for each preference based on the corresponding
trajectory. Then, the parameters $\bm{\varphi}$ are updated based on the
average gradient over all sampled preferences (i.e.,
$\frac{1}{K}\sum_{i=1}^{K}\bm{g}^{(i)}$).

The remaining question is how to calculate the gradient $\bm{g}^{(i)}$ in line
18. By the chain rule, we have

$\nabla_{\bm{\varphi}}\left[\bm{\omega}^{T}\bm{J}(\mathcal{H}_{\bm{\varphi}}(\bm{\omega}))\right]=\nabla_{\bm{\varphi}}\left[\nabla_{\bm{\theta}}\left[\bm{\omega}^{T}\bm{J}(\bm{\theta})\right]\cdot\mathcal{H}_{\bm{\varphi}}(\bm{\omega})\right]$,

(6)

where $\bm{\theta}=\mathcal{H}_{\bm{\varphi}}(\bm{\omega})$ and $\cdot$ means
dot product. As we can see, the term
$\nabla_{\bm{\theta}}\left[\bm{\omega}^{T}\bm{J}(\bm{\theta})\right]$ can be
calculated by Eq. (4). Thus, the gradient $\bm{g}^{(i)}$ can be calculated by
Eq. (4) and Eq. (6). After the Pareto set learning stage, the whole Pareto set
$PS(\Theta)$ is obtained from the hypernet $\mathcal{H}_{\bm{\varphi}}$.

##  4 Experiments

###  4.1 Benchmark Problems

To test the performance of Hyper-MORL, we use seven problems in Table 1 from a
multi-objective robot control benchmark suite Xu et al. (2020). For more
details, please refer to Appendix A.

Problem $m$ State space Action space MO-Swimmer-v2 2
$\mathcal{S}\subseteq\mathbb{R}^{8}$ $\mathcal{A}\subseteq\mathbb{R}^{2}$ MO-
HalfCheetah-v2 2 $\mathcal{S}\subseteq\mathbb{R}^{17}$
$\mathcal{A}\subseteq\mathbb{R}^{6}$ MO-Walker2d-v2 2
$\mathcal{S}\subseteq\mathbb{R}^{17}$ $\mathcal{A}\subseteq\mathbb{R}^{6}$ MO-
Ant-v2 2 $\mathcal{S}\subseteq\mathbb{R}^{27}$
$\mathcal{A}\subseteq\mathbb{R}^{8}$ MO-Hopper-v2 2
$\mathcal{S}\subseteq\mathbb{R}^{11}$ $\mathcal{A}\subseteq\mathbb{R}^{3}$ MO-
Humanoid-v2 2 $\mathcal{S}\subseteq\mathbb{R}^{376}$
$\mathcal{A}\subseteq\mathbb{R}^{17}$ MO-Hopper-v3 3
$\mathcal{S}\subseteq\mathbb{R}^{11}$ $\mathcal{A}\subseteq\mathbb{R}^{3}$

Table 1: Seven test problems in a multi-objective robot control benchmark Xu
et al. (2020).

###  4.2 Compared MORL Algorithms

We compare the proposed Hyper-MORL algorithm with the following two state-of-
the-art MORL algorithms.

####  4.2.1 PG-MORL Xu et al. (2020)

Prediction-guided MORL (i.e., PG-MORL) algorithm is an evolutionary learning
algorithm. In each generation, PG-MORL selects a set of policies from the
current population based on a prediction model, and optimizes them with a
multi-objective policy gradient method. All Pareto-optimal policies are stored
in an unbounded external archive. We use the default settings in the original
paper for PG-MORL222https://github.com/mit-gfx/PGMORL.

####  4.2.2 PD-MORL Basaklar et al. (2023)

Preference-driven MORL (i.e., PD-MORL) algorithm learns a generalized policy
by combining the preference into the Bellman equation. We use the default
settings in the original paper for PD-
MORL333https://github.com/tbasaklar/PDMORL-Preference-Driven-Multi-Objective-
Reinforcement-Learning-Algorithm. Note that the number of environment steps
used by PD-MORL is smaller than that used by PG-MORL as shown in Table 2.
However, PD-MORL needs more computation time than PG-MORL, which we will
discuss in Section 4.3. We find it impractical for PD-MORL to use the same
number of environment steps as in PG-MORL.

Problem Hyper-MORL PG-MORL PD-MORL MO-Swimmer-v2 $1.2\times 10^{7}$ $1.2\times
10^{7}$ $1\times 10^{7}$ MO-Walker2d-v2 $3\times 10^{7}$ $3\times 10^{7}$
$1\times 10^{7}$ MO-HalfCheetah-v2 $3\times 10^{7}$ $3\times 10^{7}$ $1\times
10^{7}$ MO-Ant-v2 $4.8\times 10^{7}$ $4.8\times 10^{7}$ $1\times 10^{7}$ MO-
Hopper-v2 $4.8\times 10^{7}$ $4.8\times 10^{7}$ $1\times 10^{7}$ MO-
Humanoid-v2 $1.2\times 10^{8}$ $1.2\times 10^{8}$ $1\times 10^{7}$ MO-
Hopper-v3 $1.2\times 10^{8}$ $1.2\times 10^{8}$ $1\times 10^{7}$

Table 2: The number of environment steps used by each algorithm for each
problem. PG-MORL and PD-MORL follow the settings in their original papers Xu
et al. (2020); Basaklar et al. (2023).

####  4.2.3 Hyper-MORL (Our Algorithm)

In our algorithm, we use the same neural network architecture as in PG-MORL.
Thus, the parameter spaces of Hyper-MORL and PG-MORL are the same. For a fair
comparison, Hyper-MORL uses the same policy gradient method (i.e., PPO
Schulman et al. (2017)) as in PG-MORL. The optimizer is ADAM with learning
rate $\eta=5\times 10^{-5}$. In Hyper-MORL, the parameters $\alpha$ and $d$
are set as 0.15 (i.e., 15%) and 10, respectively. The number of sampled
preferences $K$ is set as 6 and 15 for two-objective and three-objective
problems, respectively. For more details, please refer to Appendix C. All
codes of Hyper-MORL are available from https://github.com/HisaoLabSUSTC/Hyper-
MORL.

Algorithm | Hyper-MORL (Ours) | PG-MORL | PD-MORL
---|---|---|---
MO-Swimmer-v2 | 2.88$\pm$0.44 $\times$ $10^{4}$ (2) | 2.60$\pm$0.71 $\times$ $10^{4}$ (3) | $\bm{3.13\pm 0.14\times 10^{4}(1)}$
MO-HalfCheetah-v2 | 5.53$\pm$0.10$\times$$10^{6}$ (3) | 5.75$\pm$0.02$\times$$10^{6}$ (2) | $\bm{5.89\pm 0.01\times 10^{6}(1)}$
MO-Walker2d-v2 | $\bm{5.37\pm 0.35\times 10^{6}(1)}$ | 4.41$\pm$0.74$\times$$10^{6}$ (2) | 5.08$\pm$0.23$\times$$10^{6}$ (3)
MO-Ant-v2 | $\bm{7.49\pm 0.16\times 10^{6}(1)}$ | 5.79$\pm$0.25$\times$$10^{6}$ (3) | 7.05$\pm$1.13$\times$$10^{6}$ (2)
MO-Hopper-v2 | $\bm{2.05\pm 0.06\times 10^{7}(1)}$ | 1.96$\pm$0.17$\times$$10^{7}$ (2) | 5.84$\pm$8.12$\times$$10^{6}$ (3)
MO-Humanoid-v2 | 4.32$\pm$0.42$\times$$10^{7}$ (2) | $\bm{4.69\pm 0.29\times 10^{7}(1)}$ | 1.66$\pm$0.60$\times$$10^{7}$ (3)
MO-Hopper-v3 | $\bm{3.58\pm 0.25\times 10^{10}(1)}$ | 3.31$\pm$0.23$\times$$10^{10}$ (2) | 1.14$\pm$1.24$\times$$10^{10}$ (3)
Average Rank | $\bm{1.57}$ | 2.29 | 2.14
Table 3: HV-based performance comparison of Hyper-MORL with two state-of-the-
art algorithms on a multi-objective continuous robot control benchmark Xu et
al. (2020). The reference point for HV calculation is set as (0, …, 0). The
average values and standard deviations over nine runs are reported. The rank
of each method is shown in the parenthesis, and a small value means a better
rank.

(a) MO-Swimmer-v2

(b) MO-HalfCheetah-v2

(c) MO-Walker2d-v2

(d) MO-Ant-v2

(e) MO-Hopper-v2

(f) MO-Huamnoid-v2

(g) MO-Hopper-v3

Figure 2: Visualization of all Pareto-optimal policies obtained by each MORL
algorithm on each of six two-objective problems (a)-(f) and one three-
objective problem (g). The total number of required parameters to represent
these policies is shown in the parenthesis for each algorithm. A specific run
with the median HV value among nine runs is shown for each algorithm.

###  4.3 Performance Comparison

We first compare the quality of the policy sets learned by the three
algorithms. A widely-used indicator hypervolume (HV) Zitzler et al. (2003);
Shang et al. (2021) is used for the quality assessment, which is explained in
Appendix E. For the evolutionary learning algorithm (i.e., PG-MORL), we
evaluate the final population with population sizes 200 and 420 for
2-objective and 3-objective problems, respectively. For a fair comparison, we
sample the same number of uniformly distributed preferences for each of the
single model-based algorithms (i.e., PD-MORL and Hyper-MORL), and evaluate the
output policies. Table 3 shows the HV results of each algorithm on the seven
test problems. As we can see, Hyper-MORL has the best HV performance on 4 out
of 7 problems and the worst only on MO-HalfCheetah-v2. Thus, Hyper-MORL has
the best average rank in Table 3.

To test the ability of each algorithm to generate the continuous Pareto front,
we use a single run with the median HV value for each algorithm. For PG-MORL,
we plot all Pareto-optimal policies stored in the unbounded external archive.
For Hyper-MORL and PD-MORL, we input a number of uniform preferences and plot
all Pareto-optimal policies corresponding to these preferences. The number of
input preferences is set as 2,000 and 200,100 for 2-objective and 3-objective
problems, respectively. Figure 2 shows all Pareto-optimal policies from each
algorithm on each test problem. Except for MO-HalfCheetah-v2, Hyper-MORL
generates a continuous approximation of the Pareto front with high quality for
each problem. Especially for MO-Hopper-v3 in Figure 2 (g), Hyper-MORL
generates a much more dense approximation than the other two methods. As shown
in parenthesis in Figure 2, Hyper-MORL has the smallest number of parameters
for each problem among three algorithms.

Hyper-MORL shows poor performance only on MO-HalfCheetah-v2 in Figure 2 (b).
The reason is explained in Figure 3. As we can see, many preferences are
mapped to the top left corner of the objective space by Hyper-MORL. This is
because the linear scalarization function cannot find solutions on flat and
concave regions of the Pareto front. The search for those regions by non-
linear scalarization function is a future research topic.

Figure 3: Explanation of the poor performance of Hyper-MORL on MO-
HalfCheetah-v2 in Figure 2 (b). The input preferences (Left) and their
corresponding Pareto-optimal policies (Right) are plotted in two colors. The
preferences in yellow (blue) correspond to the policies in yellow (blue). Two
similar preferences $\bm{\omega}_{A}$ and $\bm{\omega}_{B}$ lead to clearly
two different policies in the right figure.

From Table 3 and Figure 2, we can see that PD-MORL (green points in Figure 2)
has poor performance on MO-Hopper-v2, MO-Humanoid-v2, and MO-Hopper-v3, which
could be due to two reasons. One reason is that the performance of PD-MORL
highly relies on an interpolation procedure with some key solutions, as
mentioned in the original paper Basaklar et al. (2023). However, PD-MORL could
not find any good key solutions for these problems. The other reason is that
the number of environment steps (shown in Table 2) in PD-MORL may be
insufficient. Figure 4 shows the training time of each algorithm. As we can
see, PD-MORL needs the longest training time. For MO-Humanoid-v2, the training
in PD-MORL for $1\times 10^{7}$ environment steps needs about four days, while
the training in Hyper-MORL and PG-MORL for $1.2\times 10^{8}$ environment
steps needs less than one day. If we use the same number of environment steps
in all three algorithms, the training in PD-MORL will take about 40 days.

Figure 4: The average training time required for each algorithm on each test
problem. The termination condition is shown in Table 2.

##  5 Further Studies

###  5.1 Investigation of the Learned Pareto Set

(a) Preference space

(b) Parameter space

(c) Objective space

Figure 5: Visualization of the Pareto-optimal policies obtained by Hyper-MORL
for MO-Walker2d-v2 problem in the parameter space (b) and objective space (c),
and their corresponding input to the hypernet in the preference space (a).
Each policy is plotted with the same color as its corresponding input
preference. t-SNE is used for visualization in the 11,150-dimensional
parameter space.

We further investigate the relation between the input preferences and the
Pareto sets learned by Hyper-MORL. For better visualization, each preference
is plotted with a different color in Figure 5 (a). Then, each policy in the
learned Pareto set is plotted with the same color as its corresponding input
preference in Figure 5 (b) and (c) for MO-Walker2d-v2. Dominated policies and
the related preferences are not shown in Figure 5. We show the learned Pareto
set in both the parameter space and the objective space. Since the dimension
of the parameter space is high, we use t-SNE Van der Maaten and Hinton (2008)
to visualize the high-dimensional parameter space in a two-dimensional space.
Visualization results for the other problems are included in Appendix D.1. In
Figure 5, we can observe a clear relation between the preferences in (a) and
the Pareto front in (c). For example, the top-left solution in Figure 5 (c)
with the largest second objective value is obtained from the preference (0,
1).

One interesting observation from Figure 5 (b) is that the shape of the Pareto
set learned by Hyper-MORL is a simple curve after the dimensionality reduction
from the 11,150-dimensional parameter space. This observation is different
from reported results on PG-MORL Xu et al. (2020) where it was shown that a
single continuous policy family cannot represent the Pareto set of these
continuous robot control problems well. Thus, we further compare the two
policy sets learned by Hyper-MORL and PG-MORL in the parameter space. PD-MORL
is not considered since it has a different parameter space from Hyper-MORL and
PG-MORL. Figure 6 shows the comparison results on the two-objective MO-
Humanoid-v2 and three-objective MO-Hopper-v3 problems. As we can see, Hyper-
MORL learns a single continuous policy family in Figure 6 (a) and closely
related policy families in Figure 6 (b) while PG-MORL learns a set of disjoint
policy families.

(a) MO-Humanoid-v2

(b) MO-Hopper-v3

Figure 6: Comparison of the policy sets learned by Hyper-MORL and PG-MORL in
the parameter space on MO-Humanoid-v2 and MO-Hopper-v3. Figure 2 (f) and
Figure 2 (g) are the corresponding visualization in the objective space for
(a) and (b).

As discussed in Section 3.2, the Pareto set learned by Hyper-MORL is located
in a $d$-dimensional parameter subspace where $d$ is a parameter for the
hypernet used in Hyper-MORL. We examine the effects of $d$ on the hypervolume
performance of Hyper-MORL using six values of $d$: $d=$1, 2, 3, 5, 10, 20.
Figure 7 shows the results of the two-objective MO-Ant-v2 and three-objective
MO-Hopper-v3 problems. As a baseline, the hypervolume performance of PG-MORL
is shown by the blue dotted line. In Figure 7 (a), the hypervolume performance
of Hyper-MORL is better than PG-MORL even when $d=1$. That means the Pareto
set of MO-Ant-v2 is well approximated by Hyper-MORL with a curved line in the
parameter space. For the three-objective MO-Hopper-v3 problem in Figure 7 (b),
good results are obtained when $d>2$. The results for the other problems are
included in Appendix D.2. Our experimental results show that the Pareto sets
of most problems are well approximated by Hyper-MORL in low-dimensional
parameter subspace.

(a) MO-Ant-v2

(b) MO-Hopper-v3

Figure 7: Effects of the dimensionality $d$ of the reduced parameter space on
the hypervolume performance of Hyper-MORL. The red vertical line shows the
standard deviation among nine runs.

###  5.2 Effects of the Warm-Up Stage

To examine the effects of the warm-up stage, we test four specifications of
$\alpha$: $0\%$, $5\%$, $10\%$, and $20\%$. $\alpha=0\%$ means that the warm-
up stage is not performed, and the hypernet is initialized by the Bias-
HyperInit Beck et al. (2023). All the other settings except for $\alpha$ are
the same as in the previous experiments. For each specification (i.e.,
$\alpha=x\%$), we calculate the hypervolume improvement percentage (HVIP) from
the results with $\alpha=0\%$ as follows:

$\text{HVIP}=\frac{\text{HV}_{\alpha=x\%}-\text{HV}_{\alpha=0\%}}{\text{HV}_{\alpha=0\%}}\times
100.$ (7)

Here positive and negative HVIP values mean positive and negative effects of
the warm-up stage with $\alpha=x\%$. The HVIP results are shown in Figure 8.
We can see that the warm-up stage has a large positive effect on the
hypervolume performance for MO-Swimmer and MO-Humanoid-v2 (i.e., more than
$40\%$ improvement). For the other problems, the positive and negative effects
of the warm-up stage are small (less than $10\%$). Figure 8 also shows that
too small value (e.g., $5\%$) or too large value (e.g., $20\%$) for the
parameter $\alpha$ is not good. Our recommended value for $\alpha$ is $15\%$.

Figure 8: The hypervolume improvement percentage (HVIP) of Hyper-MORL by the
warm-up stage with different $\alpha$ values.

##  6 Conclusions

In this paper, we proposed an MORL algorithm called Hyper-MORL, which learns a
hypernet to approximate the Pareto set in a high-dimensional parameter space
of deep policies by using a low-dimensional reduced parameter space. We
compared Hyper-MORL with two state-of-the-art algorithms on seven multi-
objective robot control problems. Our results showed that Hyper-MORL has the
best overall performance and the least training parameters among compared
algorithms. By the investigation of the Pareto set learned by Hyper-MORL, we
found that the Pareto set of these multi-objective objective robot control
problems can be well approximated by a manifold in a low-dimensional (e.g.,
3-dimensional) subspace of the original high-dimensional (e.g.,
11,150-dimensional) parameter space. Our results provide valuable insights for
researchers to design new MORL algorithms. In the future, we try to combine a
non-linear scalarization function into our method.

## Acknowledgments

This work was supported by National Natural Science Foundation of China (Grant
No. 62250710163, 62376115), Guangdong Provincial Key Laboratory (Grant No.
2020B121201001).

## References

  * Abels et al. [2019] Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher.  Dynamic weights in multi-objective deep reinforcement learning.  In International Conference on Machine Learning, volume 97, pages 11–20, Jun 2019.
  * Alegre et al. [2023] Lucas N. Alegre, Ana L. C. Bazzan, Diederik M. Roijers, Ann Nowé, and Bruno C. da Silva.  Sample-efficient multi-objective learning via generalized policy improvement prioritization.  In Autonomous Agents and Multi-Agent Systems, page 2003–2012, 2023.
  * Barreto et al. [2020] André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup.  Fast reinforcement learning with generalized policy updates.  In Proceedings of the National Academy of Sciences, 2020.
  * Basaklar et al. [2023] Toygun Basaklar, Suat Gumussoy, and Umit Ogras.  PD-MORL: Preference-driven multi-objective reinforcement learning algorithm.  In International Conference on Learning Representations, 2023.
  * Beck et al. [2023] Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson.  Hypernetworks in meta-reinforcement learning.  In Proceedings of The 6th Conference on Robot Learning, volume 205, pages 1478–1487, 2023.
  * Chauhan et al. [2023] Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton.  A brief review of hypernetworks in deep learning.  arXiv preprint arXiv:2306.06955, 2023.
  * Chen et al. [2019] Xi Chen, Ali Ghadirzadeh, Mårten Björkman, and Patric Jensfelt.  Meta-learning for multi-objective reinforcement learning.  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 977–983. IEEE, 2019.
  * Galanti and Wolf [2020] Tomer Galanti and Lior Wolf.  On the modularity of hypernetworks.  In Advances in Neural Information Processing Systems, volume 33, pages 10409–10419, 2020.
  * Hayes et al. [2022] Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, et al.  A practical guide to multi-objective reinforcement learning and planning.  Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
  * Hillermeier [2001] Claus Hillermeier.  Nonlinear multiobjective optimization: a generalized homotopy approach, volume 135.  Springer Science & Business Media, 2001.
  * Huang et al. [2021] Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti.  Continual model-based reinforcement learning with hypernetworks.  In IEEE International Conference on Robotics and Automation (ICRA), pages 799–805, 2021.
  * Lin et al. [2019] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong.  Pareto multi-task learning.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * Lin et al. [2022] Xi Lin, Zhiyuan Yang, and Qingfu Zhang.  Pareto set learning for neural multi-objective combinatorial optimization.  In International Conference on Learning Representations, 2022.
  * Lu et al. [2023] Haoye Lu, Daniel Herman, and Yaoliang Yu.  Multi-objective reinforcement learning: Convexity, stationarity and Pareto optimality.  In International Conference on Learning Representations, 2023.
  * Navon et al. [2021] Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik.  Learning the pareto front with hypernetworks.  In International Conference on Learning Representations, 2021.
  * Owen and Zhou [2000] Art Owen and Yi Zhou.  Safe and effective importance sampling.  Journal of the American Statistical Association, 95(449):135–143, 2000.
  * Parisi et al. [2014] Simone Parisi, Matteo Pirotta, Nicola Smacchia, Luca Bascetta, and Marcello Restelli.  Policy gradient approaches for multi-objective sequential decision making.  In International Joint Conference on Neural Networks, pages 2323–2330, 2014.
  * Parisi et al. [2017] Simone Parisi, Matteo Pirotta, and Jan Peters.  Manifold-based multi-objective policy search with sample reuse.  Neurocomputing, 263:3–14, 2017.
  * Pirotta et al. [2015] Matteo Pirotta, Simone Parisi, and Marcello Restelli.  Multi-objective reinforcement learning with continuous pareto frontier approximation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  * Rame et al. [2023] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord.  Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.  In Advances in Neural Information Processing Systems, 2023.
  * Rezaei-Shoshtari et al. [2023] Sahand Rezaei-Shoshtari, Charlotte Morissette, Francois R. Hogan, Gregory Dudek, and David Meger.  Hypernetworks for zero-shot transfer in reinforcement learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9579–9587, Jun. 2023.
  * Roijers et al. [2013] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley.  A survey of multi-objective sequential decision-making.  Journal of Artificial Intelligence Research, 48:67–113, 2013.
  * Roijers et al. [2018] Diederik M Roijers, Denis Steckelmacher, and Ann Nowé.  Multi-objective reinforcement learning for the expected utility of the return.  In Proceedings of the Adaptive and Learning Agents Workshop at FAIM, 2018.
  * Sarafian et al. [2021] Elad Sarafian, Shai Keynan, and Sarit Kraus.  Recomposing the reinforcement learning building blocks with hypernetworks.  In International Conference on Machine Learning, volume 139, pages 9301–9312, 2021.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * Shang et al. [2021] Ke Shang, Hisao Ishibuchi, Linjun He, and Lie Meng Pang.  A survey on the hypervolume indicator in evolutionary multiobjective optimization.  IEEE Transactions on Evolutionary Computation, 25(1):1–20, 2021.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-SNE.  Journal of Machine Learning Research, 9(11), 2008.
  * Van Moffaert and Nowé [2014] Kristof Van Moffaert and Ann Nowé.  Multi-objective reinforcement learning using sets of pareto dominating policies.  Journal of Machine Learning Research, 15(1):3483–3512, 2014.
  * von Oswald et al. [2020] Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento.  Continual learning with hypernetworks.  In International Conference on Learning Representations, 2020.
  * Xu et al. [2020] Jie Xu, Yunsheng Tian, Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik.  Prediction-guided multi-objective reinforcement learning for continuous robot control.  In International Conference on Machine Learning, pages 10607–10616, 2020.
  * Yang et al. [2019] Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan.  A generalized algorithm for multi-objective reinforcement learning and policy adaptation.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * Zhang and Li [2007] Qingfu Zhang and Hui Li.  MOEA/D: A multiobjective evolutionary algorithm based on decomposition.  IEEE Transactions on Evolutionary Computation, 11(6):712–731, 2007.
  * Zitzler et al. [2003] Eckart Zitzler, Lothar Thiele, Marco Laumanns, Carlos M Fonseca, and Viviane Grunert Da Fonseca.  Performance assessment of multiobjective optimizers: An analysis and review.  IEEE Transactions on Evolutionary Computation, 7(2):117–132, 2003.


Recent results on top quark mass and properties and rare/anomalous top quarks
interactions in CMS

S. Wuchterl$\star$ on behalf of the CMS Collaboration

Deutsches Elektronen-Synchrotron (DESY), Hamburg, Germany

*<EMAIL_ADDRESS>
Proceedings for the XXVIII International Workshop
on Deep-Inelastic Scattering and Related Subjects,
Stony Brook University, New York, USA, 12-16 April 2021
10.21468/SciPostPhysProc.?

## Abstract

The amount of data collected by the CMS experiment at the CERN LHC in the
second data taking period provides the possibility to study absolute and
differential cross sections of top quark interaction processes with high
precision. Utilizing these results, fundamental standard model parameters such
as the top quark mass are extracted. Moreover, given standard model
measurements are interpreted in the context of Beyond the standard model
theories. Recent results on top quark properties and rare or anomalous top
quark interactions by CMS are presented in these proceedings. Furthermore, the
implications of experimental results to effective field theory are discussed.

##  1 Introduction

The standard model of particle physics (SM) describes successfully a variety
of processes in a wide energy range up to several TeV. Its properties and
parameters have been investigated and determined by different experiments such
as the CMS experiment [1] at the CERN LHC over the past decades. However, open
questions such as the origin of dark matter cannot be explained by the SM, and
it is indicated that the SM could be the low energy approximation of a more
general higher order theory. The mass of the most massive elementary particle,
the top quark, suggests that the top quark takes a special role within the SM.
Its Yukawa coupling to the Higgs boson is close to unity and hence it implies
its particular importance in the electroweak symmetry breaking. Therefore, the
precise study of processes involving top quarks can shed a light both on
beyond the SM (BSM) physics but also solve the question on the stability of
the electroweak vacuum by performing precise measurements of the top quark
mass.

At hadron colliders such as the CERN LHC, top quarks are produced with a high
production rate. They are produced predominantly in pairs ($\mathrm{t\bar{t}}$
production), but also in single top production or associated with light or
heavy bosons (e.g. $\mathrm{t\bar{t}Z}$ and $\mathrm{t\bar{t}\gamma}$). Thus,
with the extensive amount of data collected in the second data taking period
by the CMS experiment, the introduced processes can be investigated with a
precision that is compatible or more precise than recent theoretical
calculations. Because the top quark decays before performing bound states, it
provides also a unique way to study quark properties or to extract other SM
parameters.

##  2 Top quark mass measurements

The value of the top quark mass, $m_{t}$, can be measured in mainly two
different ways, which involve a different theoretical understanding and can be
classified as direct and indirect measurements. Whereas direct measurements
have reached a precision up to $0.5\,\mathrm{GeV}$, they lack a well defined
theoretical definition. They are based on the kinematic reconstruction of the
top quark decay products and thus rely on inherent features and the
probabilistic modeling of multi purpose Monte Carlo (MC) generators. On the
other hand, indirect measurements extract the top quark mass in well defined
renormalization schemes, such as the pole or modified minimal subtraction
($\mathrm{\overline{MS}}$) renormalization scheme, by measuring observables
with a direct sensitivity to the top quark mass. The relation between the top
quark MC mass, $\mathrm{m_{t}^{MC}}$, from direct measurements, is usually
related to the pole mass with an additional uncertainty of the order of
$1\,\mathrm{GeV}$ motivated by the uncertainty associated to the modeling of
MC generators [2, 3].

The top quark mass was measured by the CMS collaboration recently in the
single top t-channel using $35.9$ $\mathrm{fb^{-1}}$ of pp collision data [4].
By separating events based on the lepton charge in the analyzed leptonic decay
channel of the top quark, both the mass of the top quark and the top antiquark
are additionally measured individually by a functional fit, as it is shown in
Figure 1 left for the combined fit. By making use of multivariate analysis
techniques, the signal sensitivity is enhanced and the combined top
quark/antiquark mass is measured to be $172.13\pm 0.7\,\mathrm{GeV}$. The
extracted ratio between the quark and antiquark is $0.995\pm 0.006$, and their
difference is found to be $0.83_{-1.01}^{+0.77}$. The uncertainties are
dominated mainly by the jet energy scale, final state radiation, and color
reconnection modeling uncertainties. This is the first time that
$\mathrm{m_{t}^{MC}}$ was measured with a precision below $1\,\mathrm{GeV}$ in
the single top phase space.

Figure 1: Reconstructed data as a function of the natural logarithm of the
reconstructed top quark mass together with a functional fit (left) [4].
Measured scale dependence of the top quark mass as a function of the scale
$\mathrm{\mu_{m}}=\mathrm{m_{t\bar{t}}}/2$ (right) [5, 6].

By measuring the top quark mass in the boosted regime of decaying top quarks
in the $\mathrm{t\bar{t}}$ pair production process [7], the relation between
direct and indirect measurements can be probed [8]. The top quark mass is
extracted to be $172.6\pm 1.9\mathrm{GeV}$ by comparing the unfolded
differential cross section as a function of the jet mass at particle level to
theoretical predictions. The same amount of pp collision data as in the
aforementioned analysis was used. The reported results significantly improved
with respect to former measurements by the usage of the Xcone jet clustering
algorithm. It was used for the first time at LHC to improve the resolution of
the jet mass at particle and detector levels.

The first experimental investigation of the scale dependence (running) of the
top quark mass [6] is performed by analyzing $\mathrm{t\bar{t}}$ events using
also $35.9$ $\mathrm{fb^{-1}}$ of pp collision data. The top quark mass
running in the $\mathrm{\overline{MS}}$ scheme is a fundamental effect of
quantum chromodynamics and analogous to the running of the strong coupling
constant. Through a profiled maximum likelihood fit to final state
observables, the differential cross section as a function of the invariant
mass of the $\mathrm{t\bar{t}}$ system ($\mathrm{m_{t\bar{t}}}$) is measured
at parton level. From a comparison of the measured cross section to fixed
order theory prediction, the running is probed up to a scale of
$1\,\mathrm{TeV}$ and found to agree with the SM within $1.1\mathrm{\sigma}$.

##  3 Measurements of top quark properties

Utilizing the special properties of the top quark, fundamental SM parameters
can be extracted by investigating processes involving top quarks. In the
presented measurement, the top quark Yukawa coupling $\mathrm{Y_{t}}$ is
determined by a likelihood fit to final state distributions of
$\mathrm{t\bar{t}}$ events using $137$$\mathrm{fb^{-1}}$ of pp collision data
[9]. Higher order electroweak effects affecting the Yukawa coupling are
modeled using HATHOR [10] predictions, and proxy variables are fitted to
mitigate the dependence of the missing transverse momentum resolution in the
leptonic decay channel. The best fit value for $\mathrm{Y_{t}}$ yields
$\mathrm{Y_{t}}=1.16^{+0.24}_{-0.35}$ and an upper limit of $Y_{t}<1.5$ is
extracted at 95% CL.

By separating single top t-channel signal events into multiple contributions
depending on the bottom-bottom and bottom-light quark interactions at both
$\mathrm{tWb}$ vertices, the CKM matrix elements $\mathrm{V_{tb}}$,
$\mathrm{V_{Vtd}}$, and $\mathrm{V_{ts}}$ are measured the first time
simultaneously and model independent [11]. In pp collision events
corresponding to an integrated luminosity of $35.9$ $\mathrm{fb^{-1}}$,
multivariate discriminator response observables are fitted in a likelihood fit
to determine the signal strengths of the individual single top t-channel
modes. A three-fold interpretation of the measured signal strength parameters
is provided, setting limits on the CKM matrix elements under SM unitary
assumption ($\mathrm{|V_{tb}|>0.970}$,
$\mathrm{|V_{td}|^{2}+|V_{ts}|^{2}<0.057}$), allowing for more quark families
($\mathrm{|V_{tb}|=0.988\pm 0.051}$,
$\mathrm{|V_{td}|^{2}+|V_{ts}|^{2}=0.06\pm 0.06}$), or leaving the top quark
width unconstrained ($\mathrm{|V_{tb}|=0.988\pm 0.024}$,
$\mathrm{|V_{td}|^{2}+|V_{ts}|^{2}<0.06\pm 0.06}$,
$\mathrm{\frac{\Gamma_{t}^{obs}}{\Gamma_{t}}=0.99\pm 0.42}$).

Figure 2: Comparison between measured data and SM expectation in different
bins of mass and rapidity bins for a post-fit result of $\mathrm{Y_{t}}=1.16$
(left) [12]. Distribution of the multivariate discriminator comparing data to
simulation after the fit procedure (right) [11].

Alike, the polarization of the W boson can be extracted from differential
cross sections of $\mathrm{t\bar{t}}$ and t-channel single top production
[13]. Several analyses have been published by the ATLAS and CMS collaborations
for the analyzed pp collision data at 8 TeV, and here a combination of those
results is reported. Correlations of systematic uncertainties have been
studied between the analyses, and by using the best linear unbiased estimator
method, combined results for the right handed, left handed, and longitudinal W
boson polarization fractions $\mathrm{F_{R}}$, $\mathrm{F_{L}}$, and
$\mathrm{F_{0}}$ are determined. As can be seen in Figure 3, with an overall
relative improvement of $25-30\%$, the combination yields much more precise
results than the individual measurements alone.

Figure 3: Summary plot of the individual measurements of the W polarization
fractions and their combination [13].

Because top quark pairs are produced mainly via gluon fusion at the CERN LHC,
measuring the forward-backward asymmetry ($\mathrm{A_{FB}}$) in
$\mathrm{t\bar{t}}$ production is a considerable larger challenge compared to
proton-antiproton colliders as the TEVATRON. In the results presented [14], pp
collision data corresponding to $35.9$ $\mathrm{fb^{-1}}$ are analyzed in the
boosted and resolved decay topologies. By means of a likelihood fit to three
final state observables, $\mathrm{m_{t\bar{t}}}$, the scattering angle
$\mathrm{c^{*}}$, and the longitudinal momentum fraction of the top quarks
$\mathrm{x_{F}}$, gluon and quark initiated production modes are separated and
the asymmetry is extracted. The resulting value for $\mathrm{A_{FB}}$ of
$0.048^{+0.021}_{-0.014}$ is found to be consistent with the SM and an upper
limits on the chromomagnetic and chromoeletric dipole moment (CEDM) of the top
quark are set.

In a similar approach analyzing the same set of pp collision events, the
conservation of CP symmetry in $\mathrm{t\bar{t}}$ decays is examined [12].
Theoretically, a large fraction of CP violation would be predicted from the
CEDM of the top quark. The asymmetry of the Levi-Cita tensors $\mathrm{O_{1}}$
and $\mathrm{O_{3}}$ of the four momenta of the top quarks, charged leptons,
and b quarks are directly sensitive to this effect. From their measurement,
values for the CEDM ($0.058\pm 0.098$ and $-0.01\pm 0.092$) of the top quark
are determined and are found to be consistent with the SM expectation.

##  4 Rare processes & anomalous couplings

One approach to interpret SM measurements is the framework of effective field
theory (EFT). By probing anomalous couplings of higher order operators and
their dimensionless coupling strengths represented by Wilson coefficients
(WCs), a model independent picture of BSM effects can be obtained. Two
measurements performed by CMS investigated the absolute and differential cross
section of $\mathrm{t\bar{t}}$ associated production with photons or Z bosons,
respectively. The data sets analyzed correspond to $77.5$ $\mathrm{fb^{-1}}$
for the $\mathrm{t\bar{t}\gamma}$ [15] and $137$ $\mathrm{fb^{-1}}$ for the
$\mathrm{t\bar{t}Z}$ measurement [16]. Of particular sensitivity to EFT
operators affecting $\mathrm{t\bar{t}\gamma}$ and $\mathrm{t\bar{t}Z}$
production are the momentum distribution of the Z boson and the photon, and
further the angle of the negatively charged lepton in the boson rest frame
with respect to the Z boson. Both analyses together set the tightest
constraints on the scrutinized couplings to date.

A novel approach was employed in the presented CMS analysis by parametrizing
EFT effects from multiple operators simultaneously in detector level
observables [17]. Multilepton final state events were analyzed using $137$
$\mathrm{fb^{-1}}$ pp collision data and the total yields of five associated
production modes of the top quark ($\mathrm{t\bar{t}Z}$, $\mathrm{t\bar{t}W}$,
$\mathrm{t\bar{t}H}$, $\mathrm{tZq}$, $\mathrm{tHq}$) are fitted. Using signal
categories enriched and particularly sensitive to one of the five signals
studied, the effect of 16 WCs modifying the signal strengths is probed. By
profiling them one by one or all WCs simultaneously, confidence intervals
corresponding to two standard deviations are computed. This analysis
represents the first important step to a global EFT interpretation in all
relevant processes.

Figure 4: Expected signal category yields before (left) and after the fit
procedure (right), together with the comparison to measured data [17].

##  5 Conclusion

Recent results on top quark mass and properties measurements by CMS have been
reported. Furthermore, measurements of rare and anomalous top quark
interactions have been presented with their implications for physics beyond
the standard model. Significant improvements in direct and indirect
measurements of the top quark mass have been achieved, analyzing single-top
and top pair production events. Several standard model parameters have been
extracted from studying top quark interactions, such as the top quark Yukawa
coupling, CKM matrix elements, or W boson polarization fractions. While good
agreement with the standard model is observed, interpretations of differential
cross section measurements in the effective field theory framework have been
highlighted. Together with recent analyses employing novel parametrization
techniques, they set the most stringent limits on various anomalous couplings
to date.

## References

  * [1] CMS Collaboration,  _The CMS Experiment at the CERN LHC_ ,  JINST 3, S08004 (2008),  10.1088/1748-0221/3/08/S08004.
  * [2] A. H. Hoang, S. Plätzer and D. Samitz,  _On the Cutoff Dependence of the Quark Mass Parameter in Angular Ordered Parton Showers_ ,  JHEP 10, 200 (2018),  10.1007/JHEP10(2018)200,  1807.06617.
  * [3] S. Ferrario Ravasio, T. Ježo, P. Nason and C. Oleari,  _A theoretical study of top-mass measurements at the LHC using NLO+PS generators of increasing accuracy_ ,  Eur. Phys. J. C 78(6), 458 (2018),  10.1140/epjc/s10052-019-7336-9,  [Addendum: Eur.Phys.J.C 79, 859 (2019)],  1906.09166.
  * [4] CMS Collaboration,  _Measurement of the top quark mass in events with a single reconstructed top quark at $\sqrt{s}=13\,\mathrm{TeV}$_,  CMS-PAS-TOP-19-009, https://cds.cern.ch/record/2759301 (2021).
  * [5] CMS Collaboration,  _Running of the top quark mass from proton-proton collisions at $\sqrt{s}=13\,\mathrm{TeV}$_,  http://cms-results.web.cern.ch/cms-results/public-results/publications/TOP-19-007/index.html#AddFig (2021).
  * [6] CMS Collaboration,  _Running of the top quark mass from proton-proton collisions at $\sqrt{s}=13\,\mathrm{TeV}$_,  Physics Letters B 803, 135263 (2020),  https://doi.org/10.1016/j.physletb.2020.135263.
  * [7] CMS Collaboration,  _Measurement of the jet mass distribution and top quark mass in hadronic decays of boosted top quarks in $pp$ collisions at $\sqrt{s}=13\text{ }\text{ }\mathrm{TeV}$_,  Phys. Rev. Lett. 124, 202001 (2020),  10.1103/PhysRevLett.124.202001.
  * [8] S. Fleming, A. H. Hoang, S. Mantry and I. W. Stewart,  _Jets from massive unstable particles: Top-mass determination_ ,  Phys. Rev. D 77, 074010 (2008),  10.1103/PhysRevD.77.074010.
  * [9] CMS Collaboration,  _Measurement of the top quark yukawa coupling from $t\overline{t}$ kinematic distributions in the dilepton final state in proton-proton collisions at $\sqrt{s}=13\text{ }\text{ }\mathrm{TeV}$_,  Phys. Rev. D 102, 092013 (2020),  10.1103/PhysRevD.102.092013.
  * [10] M. Aliev, H. Lacker, U. Langenfeld, S. Moch, P. Uwer and M. Wiedermann,  _HATHOR: HAdronic Top and Heavy quarks crOss section calculatoR_ ,  Comput. Phys. Commun. 182, 1034 (2011),  10.1016/j.cpc.2010.12.040,  1007.1327.
  * [11] CMS Collaboration,  _Measurement of ckm matrix elements in single top quark t-channel production in proton-proton collisions at $\sqrt{s}=13\,\mathrm{TeV}$_,  Physics Letters B 808, 135609 (2020),  https://doi.org/10.1016/j.physletb.2020.135609.
  * [12] CMS Collaboration,  _Search for CP violating anomalous top quark coupling in pp collisions with the CMS detector at $\sqrt{s}=13~{}\mathrm{TeV}$_,  CMS-PAS-TOP-18-007, https://cds.cern.ch/record/2730231 (2020).
  * [13] CMS and ATLAS Collaborations,  _Combination of the W boson polarization measurements in top quark decays using ATLAS and CMS data at $\sqrt{s}=$ 8 TeV_,  JHEP 08(08), 051 (2020),  10.1007/JHEP08(2020)051,  2005.03799.
  * [14] CMS Collaboration,  _Measurement of the top quark forward-backward production asymmetry and the anomalous chromoelectric and chromomagnetic moments in pp collisions at $\sqrt{s}$ = 13 TeV_,  JHEP 06, 146 (2020),  10.1007/JHEP06(2020)146,  1912.09540.
  * [15] CMS Collaboration,  _Measurement of the inclusive and differential $\mathrm{t}\bar{\mathrm{t}}+\gamma$ cross section and EFT interpretation in the single lepton channel at $\sqrt{s}=13~{}\mathrm{TeV}$_,  CMS-PAS-TOP-18-010, https://cds.cern.ch/record/2758332 (2021).
  * [16] CMS Collaboration,  _Measurement of top quark pair production in association with a Z boson in proton-proton collisions at $\sqrt{s}=$ 13 TeV_,  JHEP 03, 056 (2020),  10.1007/JHEP03(2020)056,  1907.11270.
  * [17] CMS Collaboration,  _Search for new physics in top quark production with additional leptons in proton-proton collisions at $\sqrt{s}=$ 13 TeV using effective field theory_,  JHEP 03, 095 (2021),  10.1007/JHEP03(2021)095,  2012.04120.