# Skill Preferences: Learning to Extract and Execute Robotic Skills from Human
Feedback

Xiaofei Wang1, Kimin Lee1, Kourosh Hakhamaneshi1 , Pieter Abbeel12 & Michael
Laskin1

###### Abstract

A promising approach to solving challenging long-horizon tasks has been to
extract behavior priors (skills) by fitting generative models to large offline
datasets of demonstrations. However, such generative models inherit the biases
of the underlying data and result in poor and unusable skills when trained on
imperfect demonstration data. To better align skill extraction with human
intent we present Skill Preferences (SkiP), an algorithm that learns a model
over human preferences and uses it to extract human-aligned skills from
offline data. After extracting human-preferred skills, SkiP also utilizes
human feedback to solve downstream tasks with RL. We show that SkiP enables a
simulated kitchen robot to solve complex multi-step manipulation tasks and
substantially outperforms prior leading RL algorithms with human preferences
as well as leading skill extraction algorithms without human preferences.

††1 University of California, Berkeley. 2 Covariant. Correspondence:
<EMAIL_ADDRESS>Video and Codes: https://sites.google.com/view/skill-pref.

> Keywords: Reinforcement Learning, Skill Extraction, Human Preferences

##  1 Introduction

Deep reinforcement learning (RL) is a framework for solving temporally
extended tasks that has resulted in a number of breakthroughs in autonomous
control including mastery of the game of Go [1, 2], learning to play video
games [3, 4, 5], and learning basic robotic control [6, 7]. However, today’s
RL systems require substantial manual human effort to engineer rewards for
each task which comes with two fundamental drawbacks. The human effort
required to design rewards is impractical to scale across numerous and diverse
task categories and the engineered rewards can often be exploited by the RL
agent to produce unintended and potentially unsafe control policies [8, 9,
10]. Moreover, it becomes increasingly difficult to design reward functions
for the kinds of complex tasks with compositional structure often encountered
real-world settings. In this work, we are interested in the following research
question - how can we learn robotic control policies that are aligned with
human intent and capable of solving complex real-world tasks?

Human-in-the-loop RL [11, 12, 13] has emerged as a promising approach to
better align RL with human intent that proposes an alternate approach to
traditional RL algorithm design. Rather than manually engineering a reward
function and then training the RL agent, human-in-the-loop RL proposes for
humans to provide feedback interactively to the agent as it is training. This
paradigm shift sidesteps reward exploitation by providing the RL algorithm
immediate feedback to align it best with human intent and, if efficient in
terms of human labels required, has the potential to scale RL training across
a diverse variety of tasks more reliably than reward engineering.

So far human-in-the-loop RL systems have been used to play Atari games [12],
solve simulated locomotion and manipulation tasks [11, 13], and better align
the output of language models [14]. While these initial results have been
promising, human-in-the-loop methods are still out of reach for the kinds of
long-horizon compositional tasks that are desired for real-world robotics. The
primary reason is that current methods do not scale efficiently with respect
to human labels for more challenging tasks. As task complexity increases, the
number of human feedback interactions required to attain a suitable policy
becomes impractical.

To address the ability of RL algorithms to scale to more complex long-horizon
tasks, a number of recent works [15, 16, 17] have proposed data-driven
extraction of behavioral priors, which we refer to as skills. In these
methods, a behavioral prior is fit to an offline dataset of demonstrations and
is then used to guide the RL policy to solve downstream tasks by regularizing
it to stay near the behavioral distribution. Such methods have been shown to
successfully solve tasks such as diverse object manipulation [16] and
operating a kitchen with a robotic arm [15]. However, they still require
engineered rewards for the downstream tasks and, more importantly, assume
access to a clean offline dataset of expert demonstrations that are
specifically relevant to the downstream tasks. In real-world scenarios, such
clean datasets are highly unlikely to exist. We desire skill extraction
methods that are robust to noisy datasets, collected by a range of policies,
with highly multi-modal structure.

In this work, we introduce Skill Preferences (SkiP), an algorithm that
integrates human-in-the-loop RL with data-driven skill extraction. Our main
insight is that human feedback can be incorporated not only for downstream RL,
as is done in prior work, but also for extracting human-aligned skills. SkiP
learns a human preference function and uses it to weigh the likelihood of
trajectories in the offline dataset based on their degree of alignment with
human intent. By incorporating human feedback during skill extraction, SkiP is
able to extract structured human-preferred skills from noisy offline data and
addresses the core limitation of prior skill extraction approaches - the
dependence on curated expert datasets. SkiP is both capable of efficiently
extracting skills and solving different downstream tasks with respect to human
labels. Similar to how prior work in human-in-the-loop RL suggested replacing
manually engineered reward functions with human feedback, our work suggests to
replace the manual effort needed to curate clean offline datasets with human
feedback. We summarize our main contributions below:

  1. 1.

We introduce Skill Preferences (SkiP), an algorithm that incorporates human
feedback to extract skills from offline data and utilize those skills to solve
downstream tasks.

  2. 2.

We show that, unlike prior leading methods for data-driven skill extraction,
SkiP is able to extract structured skills from noisy offline datasets.

  3. 3.

We show that SkiP is able to solve complex multi-step manipulation tasks in
robotic kitchen environment substantially more efficiently than prior leading
human-in-the-loop and skill extraction baselines.

Figure 1: Our method - Skill Preferences (SkiP) - consists of two phases.
During the skill extractions phase, human feedback is used to learn skills.
During the skill execution phase, human feedback is used to finetune the
skills to solve various downstream tasks. First, skills are extracted from a
noisy offline dataset with human feedback to denoise behavioral prior. Second,
skills are executed with RL in the environment with task-specific human
feedback.

##  2 Background

Reinforcement Learning: As is common with RL methods, we assume that the
control process is a Markov Decision Process (MDP) with discounted returns.
Such MDPs are defined by the tuple
$\mathcal{M}=(\mathcal{S},\mathcal{A},R,\rho_{0},\gamma)$ consisting of states
$s\in\mathcal{S}$, actions $a\in\mathcal{A}$, rewards $R=R(s,a)$, an initial
state distribution $s_{0}\sim\rho_{0}(\cdot)$, and a discount factor
$\gamma\in[0,1)$. A control policy maps states to actions within the MDP and
usually takes the form of a probability distribution – $a\sim\pi(\cdot|s)$.
The value function $V^{\pi}(s)$ and action-value function $Q^{\pi}(s,a)$
describe the value with respect to future expected returns with respect to an
initial state or state-action pair.

$V^{\pi}(s):=\mathbb{E}_{\mathcal{M},\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\mid
s_{0}=s\right],\hskip
5.0ptQ^{\pi}(s,a):=R(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim
T(\cdot|s,a)}\left[V^{\pi}(s^{\prime})\right],$

where the first expectation $\mathbb{E}_{\mathcal{M},\pi}$ denotes actions are
sampled according to $\pi$ and future states are sampled according to the MDP
dynamics. The goal in RL is the learn the optimal policy:

$\pi^{*}\in\arg\max_{\pi}\
J(\pi,\mathcal{M}):=\mathbb{E}_{s\sim\rho_{0}}\left[V^{\pi}(s)\right].$

In addition to the standard MDP setting, our method will also learn skills
$z\in\mathcal{Z}$ which consist of an encoder that maps state-action sequences
to a skill $q^{(e)}(z|s_{t},a_{t},\dots,s_{t+H-1},a_{t+H-1})$ and a decoder
that maps state-skill pairs to atomic actions
$q^{(d)}(a_{1},a_{2},...,a_{H}|s,z)$.

##  3 Method

The two primary contributions of SkiP are (i) introducing human feedback
during the skill extraction process to learn structured skills from noisy data
and (ii) utilizing human preferences over skills for downstream RL training.
Our approach shown schematically in Fig. 1 and detailed in full in Algo. 1.
Due to utilizing human feedback to learn the behavioral prior, unlike prior
approaches of skill extraction from offline data [17, 16, 15], our method is
robust to suboptimal or noisy data.

The SkiP Algorithm: We first summarize the algorithm and then proceed with its
derivation. Shown in Algo. 1, SkiP consists of two phases - (i) _skill
extraction_ and (ii) _skill execution_. A human teacher provides feedback
during both phases. During skill extraction, a human teacher labels whether a
trajectory is preferred or not (for details see Sec. 4) to train a preference
classifier. A behavioral prior is then fit to the offline data with a weighted
human preference function. During skill execution, the learned skills are
rolled out by an RL agent - a Soft Actor-Critic (SAC) [18] \- that is trained
with task-specific human preferences. As such, human feedback is used during
both phases of the algorithm. We proceed to define notation and provide a
derivation.

Preliminaries and Notation: Our method is composed of two phases - (i) the
skill extraction phase and (ii) the skill execution phase. During the skill
extraction phase, we are given an offline dataset $\mathcal{D}$ which consists
of task-agnostic, multi-modal, and potentially noisy demonstrations. We denote
trajectory sequences as $\tau_{t}=(s_{t},a_{t},\dots,s_{t+H-1},a_{t+H-1})$,
action sequences as ${\bf a}_{t}=(a_{t},\dots,a_{t+H-1})$, and skills which
decode into action sequences as $z\in\mathcal{Z}$.

Algorithm 1 SkiP: Skill Preferences

==== Skill Extraction Phase ====

INPUT: offline dataset $\tilde{\mathcal{B}}$

Initialize prior $p$, skill encoder $q_{\phi_{2}}$ and skill decoder
$p_{\phi_{1}}$. Initialize learned preference classifier $P_{\psi}$

A human provides labels $(y_{1},y_{2},...)$ for 10% of the trajectories in
$\tilde{\mathcal{B}}$ and stores them in a new buffer $\tilde{\mathcal{D}}$

for each iteration do

Update $\psi$ by maximizing
$\mathbb{E}_{(y,\tau)\sim\tilde{\mathcal{D}}}[y\cdot
logP_{\psi}(\tau)+(1-y)\cdot log(1-P_{\psi}(\tau))]$

for each iteration do

Update $p$, $q_{\phi_{2}}$, $p_{\phi_{1}}$ by optimizing $\mathcal{L}^{prior}$
(3) {Update preference weighted behavioral prior}

==== Skill Execution Phase ====

Initialize parameters of actor $\pi_{\theta 1}$, critics $Q_{\theta 2}$ and
$Q_{\bar{\theta 2}}$ and reward model $\widehat{R}_{\eta}$

Initialize a dataset of preference $\mathcal{D}\leftarrow\emptyset$ and a
dataset of transitions $\mathcal{B}\leftarrow\emptyset$

for Each iteration do

for Each environment step do

$z_{t}\sim\pi(z_{t}|s_{t})$, $s_{t+H}\sim p(s_{t+H}|s_{t},z_{t})$,
$\mathcal{B}\leftarrow\mathcal{B}\cup(s_{t},z_{t},\widehat{R}_{\eta}(s_{t},z_{t}),s_{t+H})$

if iteration $\%$ $K$ == 0 then

for step t = 1…M do

$(\tau^{(z)}_{0},\tau^{(z)}_{1})\sim\mathcal{B}$, query human for label $y$,
$\mathcal{D}\leftarrow\mathcal{D}\cup(\tau^{(z)}_{0},\tau^{(z)}_{1},y)$ {Get
preference labels}

for each gradient step of $\widehat{R}_{\eta}$ do

Sample $(\tau^{(z)}_{0},\tau^{(z)}_{1},y)\sim\mathcal{D}$, update
$\widehat{R}_{\eta}$ with $\min\mathcal{L}^{reward}$ (7) {Update preferences}

Relabel entire replay buffer $\mathcal{B}$ with $\widehat{R}_{\eta}$

for each gradient step of agent do

Sample $(s,a,s^{\prime},R)\sim\mathcal{B}$, update $\pi_{\theta 1}$ by
optimizing $\mathcal{L}^{SAC}_{actor}$ (Appendix A) {Update agent}

Update $Q_{\theta 2}$ and $Q_{\bar{\theta 2}}$ by optimizing
$\mathcal{L}^{SAC}_{critic}$ (Appendix A)

Learning Behavioral Priors with Human Feedback (Skill Extraction): Our main
insight is to use human preferences in order to fit a weighted behavioral
prior over an offline dataset of (potentially noisy) demonstrations. Our
method builds on prior work for behavioral extraction from offline data via
expected maximum likelihood latent variable models [17, 16, 15].

Specifically, prior work [17, 16, 15] considers a parameterized generative
model $p_{\alpha}({\bf a}_{t}|s_{t})$ over action sequences where ${\bf
a}_{t}=(a_{t},\dots,a_{t+H-1})$ that represents a behavioral prior and is
trained to replicate the transition statistics in the offline dataset:

$p_{\alpha}\ \in\ \arg\max_{\alpha}\
\mathbb{E}_{\tau\sim\mathcal{D}}\left[\sum_{t=0}\log\Big{(}p_{\alpha}({\bf
a}_{t}|s_{t})\Big{)}\right].$ (1)

In our approach, we consider an adaptive behavioral prior that is biased
towards trajectories that achieve higher rewards according to the human
preference function. This can be particularly useful in diverse datasets
collected with suboptimal or noisy policies or multiple policies of varying
expertise. For example, one could imagine multiple humans collecting
demonstrations or multiple robots exploring their environment. Similar to
Siegel et al. [19], we seek a behavioral prior that is biased towards the high
reward trajectories in the dataset while also staying close to the average
statistics in the dataset. However, unlike prior work on weighted behavioral
priors [19, 20, 21] the weight is determined through the human preference
function and we aim to maximize action-sequence likelihood as opposed to
single-timestep actions.

We formulate this as:

$p_{\alpha}\ \in\ \arg\max_{\alpha}\
\mathbb{E}_{\tau\sim\mathcal{D}}\left[\sum_{t=0}^{|\tau|}\omega(\tau_{t})\cdot
p_{\alpha}({\bf a}_{t}|s_{t})\right]\text{such that }\
\mathbb{E}_{\tau\sim\mathcal{D}}\left[D_{KL}\left(p_{\alpha}\|\bar{p}\right)\right]\leq\delta,$
(2)

where $\bar{p}$ denotes the empirical behavioral policy and
$\omega(s_{t},a_{t})$ is the weighting function. The non-parametric solution
to the above optimization is given by:

$p_{\alpha}({\bf a}_{t}|s_{t})\propto\bar{p}({\bf
a}_{t}|s_{t})\cdot\exp\left(\omega(\tau_{t})/T\right),$

where we have used $\propto$ to avoid specification of the normalization
factor, and $T$ represents a temperature parameter that is related to the
constraint level $\delta$. The above non-parametric policy can be projected
into the space of parametric neural network policies as [20, 19]:

$p_{\alpha}\in\ \arg\max_{\alpha}\
\mathbb{E}_{\tau\sim\mathcal{D}}\left[\sum_{t=0}^{|\tau|}\exp\left(\omega(\tau_{t})/T\right)\cdot\log\Big{(}p_{\alpha}({\bf
a}_{t}|s_{t})\Big{)}\right].$ (3)

For the choice of the weighting function, we use the learned preference
classifier $P_{\psi}(y|\tau)$ which inputs a trajectory and outputs the
likelihood of this trajectory being human-preferred with $y\in[0,1]$.
$P_{\psi}(y|\tau)$ is learned by sampling a small subset of the offline
dataset and soliciting human feedback to label preferred versus not preferred
trajectory: $\omega(\tau_{t}):=\log P_{\psi}(\tau_{t})$.

In this process, we treat the temperature $T$ as the hyper-parameter choice.
This implicitly defines the constraint threshold $\delta$, and makes the
problem specification and optimization more straightforward. For our practical
implementation, we fit a variational autoencoder similar to [17, 15] but
softly weighted to maximize the likelihood of human-preferred transitions. We
introduce a latent variable $z$ with a Guassian prior such that the ELBO loss
is given by:

$\log{p({\bf a}_{t}|s_{t})}\geq\mathbb{E}_{\tau\sim\mathcal{D},z\sim
q_{\phi_{2}}(z|\tau)}[\underbrace{\log p_{\phi_{1}}({\bf
a}_{t}|s_{t},z)}_{\mathcal{L}_{\text{rec}}}+\beta\underbrace{(\log{p(z)}-\log
q_{\phi_{2}}(z|\tau)}_{\mathcal{L}_{\text{reg}}}].$ (4)

This is the standard $\beta$-VAE loss applied to action sequence modeling
where $\beta$ is a scalar controlling the regularization strength and
$\phi_{1},\phi_{2}$ are neural network parameters that are optimized during
training. Note that $q_{\phi_{2}}$ encodes trajectories into a latent vecotr
and $p_{\phi_{1}}$ decodes latent vectors and the starting state back into
action sequences. Our training objective weighs this loss with the preference
function. Thus, our overall skill extraction objective is to maximize:

$\displaystyle\mathcal{L}=\arg\max_{\phi_{1},\phi_{2}}{E}_{\tau\sim\mathcal{D},z\sim
q_{\phi}(z|\tau)}\left[P_{\psi}(\tau)(\mathcal{L}_{\text{rec}}+\mathcal{L}_{\text{reg}})\right].$
(5)

Reward learning and human preferences over skills (Skill Execution): Unlike
traditional RL where the hand-engineered rewards are available, we consider
the preference-based RL framework [11, 12, 13, 22]: a (human) teacher provides
preferences between the agent’s behaviors and the agent uses this feedback to
perform the task. In order to incorporate human preferences into deep RL,
Christiano et al. [11] proposed a framework that learns a reward function
$\widehat{R}_{\eta}$ from preferences. In this work, we modify the preference
framework to operate not over atomic state-action transitions but rather
state-skill transitions that have substantially longer time spans.

Formally, we assume access to an offline dataset (the agent’s replay buffer)
$\mathcal{B}$ of state-action transitions and sample state-skill sequence
pairs $\tau^{(z)}_{1},\tau^{(z)}_{2}$ for which a human provides a binary
label $y\in\\{0,1\\}$, where
$\tau^{(z)}=(s_{t},z_{t},s_{t+H},z_{t+H},\dots,s_{(t+M)H},z_{(t+M)H})$ where
$H$ is the length of actions the skill decodes to and $M$ is the total number
of state-skill transitions. Note how such trajectories are $H$ times longer
than if we were to sample state-action trajectories of length $M$.

The reward function $\widehat{R}$ therefore fits a Bernoulli distribution
across sequences. In this work, we learn a parameterized reward function
$\widehat{R}_{\eta}$ as in [13] utilizing a Bradley-Terry model [23] in the
following manner:

$\displaystyle
P_{\eta}[\tau^{(z)}_{1}\succ\tau^{(z)}_{0}]=\frac{\exp\sum_{t}\widehat{R}_{\eta}(s_{t}^{1},z_{t}^{1})}{\sum_{i\in\\{0,1\\}}\exp\sum_{t}\widehat{R}_{\eta}(s_{t}^{i},z_{t}^{i})}.$
(6)

Here, the operator $A\succ B$ means that $A$ is preferred to $B$.
$\widehat{R}_{\eta}$ can therefore be interpreted as a binary preference
classifier where labels are provided through human feedback. The parameters
$\eta$ of the neural network are updated by optimizing a binary cross-entropy
loss:

$\displaystyle\mathcal{L}^{\tt
Reward}=-\mathbb{E}_{(\tau^{0},\tau^{1},y)\sim\mathcal{D}}\Big{[}$
$\displaystyle y(0)\log P_{\eta}[\tau^{(z)}_{0}\succ\tau^{(z)}_{1}]+y(1)\log
P_{\eta}[\tau^{(z)}_{1}\succ\tau^{(z)}_{0}]\Big{]}.$ (7)

##  4 Experimental Setup

Environments: For our experiments, we use the robot kitchen environment and
offline dataset from the D4RL suite [24]. This environment requires a 7-DOF
(6-DOF arm and 1-DOF gripper) robotic arm to solve complex multi-step tasks in
a kitchen. Due to the 7-DOF control and compositional long-horizon nature of
the tasks, this environment cannot be solved by standard methods such as SAC
or behavior cloning [15].

Offline dataset: We desire our method to work on suboptimal offline data and,
unlike prior skill extraction approaches [15, 16, 17] do not assume that the
offline dataset consists solely of expert demonstrations. We simulate a noisy
offline dataset by combining 601 expert trajectories and 601 noisy
trajectories generated by random policy. The expert trajectories involve
various structured kitchen interactions such as opening the microwave and
operating the stove. We solicit human feedback on 10% of the total
trajectories or equivalently 120 human labels .

Downstream tasks: We use 6 different downstream tasks shown in Fig. 2 that
vary in difficulty to evaluate our approach. The task suite consists of tasks
that require one, two, or three subtasks to be completed in a row in order to
achieve the overall goal. We note that even the tasks with one subtask is
challenging for RL methods that operate over atomic actions and do not
leverage skills, as is shown in the experimental results.

Simulated human: Similar to prior work [11, 13], we obtain feedback from
simulated human teachers instead of real humans. During skill extraction,
human provides labels whether a trajectory is noisy or structured.111Here, we
remark that limited number of human labels (10% of the total trajectories) is
utilized in our experiments for skill extraction. During skill execution, the
simulated human assigns positive labels to trajectory segments that have made
more progress toward completing the desired task. Progress is calculated by
computing $||s_{M\cdot H}-\bar{s}||_{2}-||s_{1}-\bar{s}||_{2}$, where
$\bar{s}$ is the state when the target task is completed.

Figure 2: We evaluate in the robot kitchen environment from D4RL [24], which
requires a 7-DOF robotic arm to operate a kitchen. Within this environment, we
consider a variety of manipulation tasks of varying difficulty. The simplest
tasks involve one subtask - opening a microwave or moving the kettle - while
more challenging tasks require the agent to compose multiple subtasks.
Overall, we consider 6 evaluation tasks that require chaining one, two, or
three subtasks.  Figure 3: An illustration of the skill extraction procedure
within the robot kitchen environment. Starting with a noisy offline dataset,
which consists of both expert and random actions, our method fits a behavioral
prior to the offline data using human feedback to identify human-preferred
motions which results in a set of diverse skills that can then be finetuned to
downstream tasks.

Baselines: In addition to our method, we compare to Atomic Preferences which
we based on PEBBLE [13]: a state-of-the-art human preference RL method. it
pretrains the SAC agent with behavior cloning over the optimal offline dataset
and trains the online SAC agent with human preferences over atomic transitions
instead of high-level skill transitions. We also compare to Flat Prior which
learns a single-step action prior on the atomic action space over the optimal
dataset and trains an online SAC agent regularized with the action prior over
ground-truth reward. The Oracle we compare to is SPiRL, a leading skill
extraction with access to the ground truth (expert demonstrations and ground
truth reward) in Fig. 4.

##  5 Experimental Results

For the experimental evaluation of our approach, we investigate the following
questions: (a) Can SkiP solve challenging long-horizon tasks and how does our
method compare to prior leading approaches? (b) How do SkiP compare to an
oracle baseline that extracts skills from perfect expert demonstrations and
has access to the ground truth reward? (c) Is it necessary to provide human
feedback during skill extraction or is it sufficient to fit an unweighted
behavioral prior over the offline data? (d) How should we incorporate human
feedback during the skill execution phase?

Main Results: We evaluate SkiP and related baselines on the 6 tasks shown in
Fig. 2 and display the learning curves in Fig. 4. We observe that SkiP is the
only method (except for the Oracle) that is capable of solving the majority of
tasks in the robot kitchen task suite and outperforms the baselines on all
environments. On 5 out of 6 tasks, SkiP is able to match the oracle baseline
asymptotically which means that it arrives at the optimal solution.

SkiP is also human-label efficient. During skill extraction, only 120 labels
are required to train the preference classifier. During skill execution,
300-1K labels are required to solve most tasks depending on the task’s
complexity. We hypothesize that human label efficiency is better during the
skill extraction phase because classifying structured and noisy skills from a
static offline dataset is easier than classifying task-specific preferences
from an evolving replay buffer. Further human label efficiency improvements
pose interesting research directions for future work.

Figure 4: SkiP and baselines (Sec. 4) evaluated over six tasks in the robot
kitchen environment shown in Fig. 2. SkiP outperforms both baselines across
the majority of the tasks and is the only method that is capable of matching
the Oracle on most tasks. We also compare SkiP to SkiP with 3x more human
labels and find comparable performance between the two versions. SkiP solves
most tasks given 300-1000 human labels depending on the complexity of the
task.

Ablation Studies: To further understand the properties of the SkiP algorithm,
we investigate whether human feedback is necessary during skill extraction as
well as how the human preference reward function compares to alternate
approaches to human feedback during skill execution.

Is it necessary to provide human feedback during skill extraction or is it
sufficient to fit an unweighted behavioral prior over the offline data? The
offline dataset used throughout this paper consists of suboptimal data that is
a mixture of expert and random actions. We compare fitting a human-feedback
weighted behavioral prior as opposed to an unweighted behavioral prior that
maximizes the likelihood of all action sequences equally. For the skill
execution phase, both methods have access to the same human preference reward
function. The results shown in Fig. 5 indicate that the method, which extracts
skills without human feedback, is unable to solve any of the tasks suggesting
that human feedback is essential for skill extraction from suboptimal offline
data.

Figure 5: SkiP with human feedback vs SkiP without human feedback during skill
extraction. learning curve with shaded region representing standard error
across three seeds. Both algorithms learns prior from the suboptimal dataset
and were evaluated with online RL. SkiP with human feedback outperforms SkiP
without human feedback on all 6 environments

How should we incorporate human feedback during the skill execution phase?
Instead of preferences, a simpler approach to learning from human feedback is
to provide binary feedback if a task (or subtask) has been solved and learning
a reward classifier to guide the RL agent. We implement this by providing a
positive reward of 1 for a high-level transition $(s_{t},z,s_{t+H})$ when a
subtask has been completed and 0 otherwise. Using the same number of human
queries for both approaches, we compare learning with preferences as opposed
to learning from sparse rewards. For both approaches, we use human feedback
for skill extraction. As shown in Fig. 6, RL with a reward classifier for
subtask completion is able to solve some tasks but generally performs much
worse than RL with human preferences.

Figure 6: SkiP with preferences vs SkiP with learned sparse reward. Learning
curve with shaded region representing standard error across three seeds. both
algorithms use the same prior. SkiP with preferences outperforms SkiP with
learned sparse reward on 5 out of 6 environments.

##  6 Related Work

Human-in-the-loop Reinforcement Learning: Several works have successfully
utilized feedback from real humans to train RL agents [25, 11, 12, 26, 13, 27,
28]. One of major directions is directly utilizing the human feedback as a
learning signal [29, 27, 25] but assumed unlimited access to human labels
which limited their practicality for more challenging tasks. To address this
limitation, a number of works proposed learning reward model from human
feedback [26, 28, 30, 31, 32, 33]. Recently, several works have successfully
combined human preferences with deep RL algorithms to learn basic locomotion
skills as well as playing video games from pixels using human [11, 12, 34,
13]. However, these methods are limited to short-horizon or cyclic tasks and
do not scale to more challenging compositional multi-step tasks. In this work,
we investigate how to scale human preferences to such challenging tasks by
specifying preferences over skills.

Data-driven Extraction of Behavioral Priors: Behavioral prior or skill
extraction refers to fitting a distribution over an offline dataset of
demonstrations and biasing the agent’s policy towards the most likely actions
from that distribution. Commonly used for offline RL [21, 19, 20], behavioral
priors learned through maximum likelihood latent variable models can also been
used as skills for structured exploration in RL [16], to solve complex long-
horizon tasks from sparse rewards [15, 17], and regularize offline RL policies
[21, 20, 35]. A limitation of these skill extraction methods is that the
quality of the behavioral prior is highly dependent on the demonstrations in
the offline dataset. Since a behavioral prior models maximum likelihood
transitions in the offline dataset, suboptimal, noisy, or irrelevant
transitions can degrade downstream policy learning. In this work, we introduce
human feedback into the skill extractions phase to learn a human preferred
behavioral prior which enables skill extraction methods to be robust to
suboptimal offline data.

##  7 Conclusion

We presented Skill Preferences (SkiP) an algorithm that uses human feedback
for both skill extraction as well as execution, and showed that SkiP enables
robotic agents to solve long-horizon compositional manipulation tasks. We hope
that this work excites other researchers about the potential of learning with
skills and human feedback.

##  8 Acknowledgements

We would like to thank Berkeley DeepDrive, Tencent, ONR Pecase N000141612723,
and NSF NRI 2024675 for supporting this research.

## References

  * Silver et al. [2018] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Silver et al. [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al.  Mastering the game of go without human knowledge.  _Nature_ , 550(7676):354–359, 2017.
  * Mnih et al. [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * OpenAI [2018] OpenAI.  Openai five.  https://blog.openai.com/openai-five/, 2018.
  * Vinyals et al. [2019] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Akkaya et al. [2019] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al.  Solving rubik’s cube with a robot hand.  _arXiv preprint arXiv:1910.07113_ , 2019.
  * Kalashnikov et al. [2018] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al.  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  In _Conference on Robot Learning_ , 2018.
  * Hadfield-Menell et al. [2017] D. Hadfield-Menell, S. Milli, P. Abbeel, S. Russell, and A. Dragan.  Inverse reward design.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Amodei et al. [2016] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané.  Concrete problems in ai safety.  _arXiv preprint arXiv:1606.06565_ , 2016.
  * Turner et al. [2020] A. M. Turner, N. Ratzlaff, and P. Tadepalli.  Avoiding side effects in complex environments.  _arXiv preprint arXiv:2006.06547_ , 2020.
  * Christiano et al. [2017] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.  Deep reinforcement learning from human preferences.  In _Advances in neural information processing systems_ , 2017.
  * Ibarz et al. [2018] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei.  Reward learning from human preferences and demonstrations in atari.  _Advances in neural information processing systems_ , 2018.
  * Lee et al. [2021] K. Lee, L. Smith, and P. Abbeel.  Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  _arXiv preprint arXiv:2106.05091_ , 2021.
  * Ziegler et al. [2019] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. F. Christiano, and G. Irving.  Fine-tuning language models from human preferences.  _CoRR_ , abs/1909.08593, 2019.  URL http://arxiv.org/abs/1909.08593.
  * Pertsch et al. [2020] K. Pertsch, Y. Lee, and J. J. Lim.  Accelerating reinforcement learning with learned skill priors.  In _Conference on Robot Learning (CoRL)_ , 2020.
  * Singh et al. [2021] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine.  Parrot: Data-driven behavioral priors for reinforcement learning.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=Ysuv-WOFeKR.
  * Ajay et al. [2021] A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum.  {OPAL}: Offline primitive discovery for accelerating offline reinforcement learning.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=V69LGwJ0lIN.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning_ , 2018.
  * Siegel et al. [2020] N. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, and M. A. Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  _ArXiv_ , abs/2002.08396, 2020.
  * Peng et al. [2019] X. B. Peng, A. Kumar, G. Zhang, and S. Levine.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _CoRR_ , abs/1910.00177, 2019.  URL http://arxiv.org/abs/1910.00177.
  * Wu et al. [2019] Y. Wu, G. Tucker, and O. Nachum.  Behavior regularized offline reinforcement learning.  _CoRR_ , abs/1911.11361, 2019.  URL http://arxiv.org/abs/1911.11361.
  * Leike et al. [2018] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg.  Scalable agent alignment via reward modeling: a research direction.  _arXiv preprint arXiv:1811.07871_ , 2018.
  * Bradley and Terry [1952] R. A. Bradley and M. E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345, 1952.
  * Fu et al. [2020] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Arumugam et al. [2019] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman.  Deep reinforcement learning from policy-dependent human feedback.  _arXiv preprint arXiv:1902.04257_ , 2019.
  * Knox and Stone [2009] W. B. Knox and P. Stone.  Interactively shaping agents via human reinforcement: The tamer framework.  In _International Conference on Knowledge Capture_ , 2009.
  * MacGlashan et al. [2017] J. MacGlashan, M. K. Ho, R. Loftin, B. Peng, D. Roberts, M. E. Taylor, and M. L. Littman.  Interactive learning from policy-dependent human feedback.  In _International Conference on Machine Learning_ , 2017.
  * Warnell et al. [2018] G. Warnell, N. Waytowich, V. Lawhern, and P. Stone.  Deep tamer: Interactive agent shaping in high-dimensional state spaces.  In _Conference on Artificial Intelligence_ , 2018.
  * Pilarski et al. [2011] P. M. Pilarski, M. R. Dawson, T. Degris, F. Fahimi, J. P. Carey, and R. S. Sutton.  Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning.  In _International Conference on Rehabilitation Robotics_ , 2011.
  * Pinto and Gupta [2016] L. Pinto and A. Gupta.  Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.  In _International Conference on Robotics and Automation_ , 2016.
  * Levine et al. [2018] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.  _The International Journal of Robotics Research_ , 37(4-5):421–436, 2018.
  * Fu et al. [2018] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine.  Variational inverse control with events: A general framework for data-driven reward definition.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Xie et al. [2018] A. Xie, A. Singh, S. Levine, and C. Finn.  Few-shot goal inference for visuomotor learning and planning.  In _Conference on Robot Learning_ , 2018.
  * Cao et al. [2020] Z. Cao, K. Wong, and C.-T. Lin.  Human preference scaling with demonstrations for deep reinforcement learning.  _arXiv preprint arXiv:2007.12904_ , 2020.
  * Nair et al. [2020] A. Nair, A. Gupta, M. Dalal, and S. Levine.  Awac: Accelerating online reinforcement learning with offline datasets, 2020.
  * Lillicrap et al. [2016] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra.  Continuous control with deep reinforcement learning.  In _International Conference on Learning Representations_ , 2016.
  * Ziebart [2010] B. D. Ziebart.  Modeling purposeful adaptive behavior with the principle of maximum causal entropy.  2010\.

Appendix

##  Appendix A Background

Off-policy RL with Soft Actor-Critic. The Soft Actor-Critic (SAC) [18] is a
leading off-policy RL algorithm. Like other off-policy RL methods, such as DQN
[3] or DDPG [36], SAC optimizes a Q function but does so based on the maximum
entropy framework for RL [37]. In addition to maximizing the reward function,
SAC also maximizes the policy entropy which leads to improved exploration and
helps prevent overfitting. As an actor-critic method, SAC optimizes both the
actor’s policy by maximizing a value function as well as a critic with a
Bellman loss. The actor’s parameters are updated to maximize the Q function
and policy entropy which is encapsulated by the following equation:

$\displaystyle\mathcal{L}^{\tt SAC}_{\tt
actor}=\mathbb{E}_{s_{t}\sim\mathcal{B},a_{t}\sim\pi_{\theta_{1}}}\Big{[}\alpha\log\pi_{\theta_{1}}(a_{t}|s_{t})-Q_{\theta_{2}}(s_{t},a_{t})\Big{]}.$
(8)

Here, $(s_{t},a_{t})$ are state-action pairs, $\mathcal{B}$ is a replay
buffer, ${\theta_{1}}$ is the actor’s parameters, ${\theta_{2}}$ are the
critic’s parameters, and $\alpha$ is a scalar value that control the entropy
strength. The policy $\pi_{\theta_{1}}$ is parametrized by a multi-variate
Gaussian with a diagonal covariance matrix and outputs the means and standard
deviations that are then used to sample actions from the Gaussian
distribution. To update the critic’s parameters, SAC optimizes a soft Q
function by minimizing the soft Bellman loss:

$\displaystyle\mathcal{L}^{\tt SAC}_{\tt
critic}=\mathbb{E}_{\tau_{t}}\Big{[}\left(Q_{\theta_{2}}({\bf
s}_{t},a_{t})-R_{t}-\gamma\big{[}Q_{\bar{\theta_{2}}}(s_{t},a_{t})-\alpha\log\pi_{\theta_{1}}(a_{t}|s_{t})\big{]}\right)^{2}\Big{]},$
(9)

where $\tau_{t}=(s_{t},a_{t},s_{t+1},R_{t})$ is a single timestep transition,
$\bar{\theta}$ denotes the Polyak averaging of the critic’s parameters, and
$\alpha$ is a temperature parameter.

##  Appendix B Implementation Details

###  B.1 Hyperparamters

Because we built off of SPiRL [15], we used the same set of hyperparamters for
skill extraction and online RL training. The reward model learning from human
preference has the same hyperparamters as in PEBBLE. [13].

Hyperparameters for Skill Extraction | Value
---|---
Skill Horizon | $10$
Ensemble Size | $3$
Hidden Units | $200$
Non-linearity | ReLU
Optimizer | Adam
Learning Rate | $0.001$
Weight Decay | $0.0001$
$(\beta_{1},\beta_{2})$ | $(.9,.999)$

Hyperparameters for Skill Execution | Value
---|---
Query Batch Size | $128$
Query Frequency | $100,000$
Segment Size | $5$
Sampling Scheme | Entropy Exploit

##  Appendix C Effect of segment size

Figure 7: The plot compares SkiP with different segment size over the Kettle-
Burner-Cab environment. Lines and shaded area represent mean and standard
error over three seeds, respectively.

As shown in Fig 7, unlike PEBBLE [13], we did not find segment size to affect
our method’s performance.


# Time periodic solutions of first order mean field games from the perspective
of Mather theory

Panrui Ni

###### Abstract

In this paper, the existence of non-trivial time periodic solutions of first
order mean field games is proved. It is assumed that there is a non-trivial
periodic orbit contained in the Mather set. The whole system is autonomous
with a monotonic coupling term. Moreover, the large time convergence of
solutions of first order mean field games to time periodic solutions is also
considered.

Keywords. mean field games; minimal measures; time periodic solutions

††Shanghai Center for Mathematical Sciences, Fudan University, Shanghai
200433, China;
e-mail<EMAIL_ADDRESS>_Mathematics Subject Classification (2020):_
35Q89; 37J51; 35B10

##  1 Introduction

The mean field games (MFGs) was introduced by Lasry and Lions in [22, 23, 24]
and by Caines, Huang and Malhamé in [20, 21] to analyze large population
stochastic differential games. In this paper, we consider the evolutionary
first order mean field games with a finite horizon $T>0$ and an initial-final
condition, which is a system consists of a Hamilton-Jacobi equation coupled
with a continuity equation (cf. [24, Eqs. (49)-(51)])

$\displaystyle\partial_{t}u(x,t)+H(x,Du(x,t))=F(m(t))+c,\quad(x,t)\in
M\times(0,T).$ (1.1)
$\displaystyle\partial_{t}m(x,t)+\textrm{div}(\partial_{p}H(x,Du(x,t))m(x,t))=0,\quad(x,t)\in
M\times(0,T).$ (1.2) $\displaystyle u(x,0)=\varphi(x),\quad m(x,T)=m_{T}(x).$
(1.3)

Here we assume $M$ is a compact connected and smooth Riemannian manifold
without boundary. Let $D$ stand for the spacial gradient with respect to $x\in
M$, and let $d(x,y)$ be the Riemannian distance between $x$ and $y$ in $M$. We
denote by $TM$ and $T^{*}M$ the tangent and cotangent bundle over $M$
respectively. The function $H:T^{*}M\times\mathbb{R}\to\mathbb{R}$ is called
the Hamiltonian. Let $\mathcal{P}(M)$ be the space of all Borel probability
measures on $M$. The function $m(x,t)$ is the density of the mean field term
$m(t)\in\mathcal{P}(M)$ if it is absolutely continuous with respect to the
Lebesgue measure, i.e. $dm(t)=m(x,t)dx$. In previous works, the coupling term
$F$ is usually assumed to depend on the spacial variable $x\in M$, i.e.,
$F=F(x,m)$ is from $M\times\mathcal{P}(M)$ to $\mathbb{R}$. In the present
paper, we assume the coupling term $F:\mathcal{P}(M)\to\mathbb{R}$ is a
functional independent of $x\in M$, and $c\in\mathbb{R}$ is a constant.
Although $F$ does not depend on $x$, the results of this article can still
handle the case where the coupling term is of the form $F(m)+f(x)$, because
the Hamiltonian $H$ depends on $x$.

The solution of the above system is defined as follows.

######  Definition 1.1.

(Viscosity solution) A continuous function
$u:M\times[0,T]\rightarrow\mathbb{R}$ is called a viscosity subsolution (resp.
supersolution) of (1.1) if for each test function
$\phi:M\times[0,T]\to\mathbb{R}$ of class $C^{1}$, when $u-\phi$ attains its
local maximum (resp. minimum) at $(x,t)$, then

$\partial_{t}\phi(x,t)+H(x,D\phi(x,t))\leq F(m(t))+c,\quad(\textrm{resp}.\
\geq F(m(t))+c).$

A continuous function $u$ is called a viscosity solution of (1.1) if it is
both a viscosity subsolution and a viscosity supersolution.

######  Definition 1.2.

A weak solution of the system (1.1)-(1.2) is a couple $(u(x,t),m(t))\in
C(M\times[0,T])\times L^{1}([0,T],\mathcal{P}(M))$ such that (1.1) is
satisfied in viscosity sense and (1.2) is satisfied in distributions sense,
i.e.,

$\int_{M}\varphi(x,T)dm_{T}(x)=\int_{0}^{T}\int_{M}(\partial_{t}\varphi(x,t)+\langle\partial_{p}H(x,Du(x,t)),D\varphi(x,t)\rangle)dm(t)(x),$

for each $\varphi\in C^{\infty}_{c}(M\times(0,T])$. Here
$\langle\cdot,\cdot\rangle$ represents the canonical pairing between the
tangent space and cotangent space, $C^{\infty}_{c}(M\times(0,T])$ stands for
the set of all smooth functions on $M\times(0,T]$ with compact support.

Now we recall some facts on the space of probability measures
$\mathcal{P}(M)$. A sequence
$\\{\mu_{k}\\}_{k\in\mathbb{N}}\subset\mathcal{P}(M)$ is $w^{*}$-convergent to
$\mu\in\mathcal{P}(M)$ if

$\lim_{k\rightarrow+\infty}\int_{M}f(x)d\mu_{k}=\int_{M}f(x)d\mu,\quad\forall
f\in C(M).$

We shall work with the Monge-Wasserstein distance $d_{1}$, which is defined by

$d_{1}(m_{1},m_{2})=\sup_{\phi}\int_{M}\phi d(m_{1}-m_{2}),\quad\forall
m_{1},\ m_{2}\in\mathcal{P}(M),$

where the supremum is taken over all 1-Lipschitz continuous functions on $M$.
We recall that $d_{1}$ metricizes the $w^{*}$-topology. Assume the coupling
term $F:\mathcal{P}(M)\to\mathbb{R}$ satisfies

  * (F)

Lipschitz continuity: there is $C>0$ such that $|F(m_{1})-F(m_{2})|\leq
Cd_{1}(m_{1},m_{2})$.

Consider the stationary system corresponding to (1.1)-(1.3)

$\displaystyle H(x,Du(x))=F(m)+c,\quad x\in M.$ (1.4)
$\displaystyle\textrm{div}(\partial_{p}H(x,Du(x))m(x))=0,\quad x\in M.$ (1.5)

A natural goal of the study of MFGs is to prove the convergence of solutions
of evolutionary MFGs (1.1)-(1.3) to equilibrium states (1.4)-(1.5) as the
horizon $T\to+\infty$. There are many works on the long time average of
solutions of evolutionary MFGs. Usually, we assume
$F:M\times\mathcal{P}(M)\to\mathbb{R}$ satisfies the monotone condition

$\int_{M}(F(x,m)-F(x,m^{\prime}))d(m-m^{\prime})\geq 0,\quad\forall m,\
m^{\prime}\in\mathcal{P}(M).$ (1.6)

Without the monotone condition, the solution of evolutionary MFGs may not
converges to equilibrium states, see [14, 15, 16, 19]. In this paper, we
provide a case in which there are non-trivial time periodic solutions of first
order MFGs. Since $F:\mathbb{P}(M)\to\mathbb{R}$ is independent of $x\in M$,
for all probability measures $m$ and $m^{\prime}$, we have

$\int_{M}(F(m)-F(m^{\prime}))d(m-m^{\prime})=(F(m)-F(m^{\prime}))\int_{M}d(m-m^{\prime})=0.$

Then the coupling term $F:\mathcal{P}(M)\to\mathbb{R}$ considered in the
present paper satisfies the monotone condition (1.6). The results in this
paper show that the large time behavior of MFGs is related to the minimal
measures introduced by J. Mather in [26]. For the further study, we may
consider the coupling term $F$ depending on the spacial variable $x\in M$.

Before presenting the results obtained in this paper, we first recall some
recent works on the large time behavior of mean field games. For the second
order mean field games, the large time behavior is quite well understood. One
can refer to [9, 29, 10, 11, 17]. For the first order mean field games, the
analysis is related to the weak KAM theory and the Aubry-Mather theory. There
are many works on the long time average, see [7, 5, 4, 12, 8]. In these works,
the monotone condition (1.6) plays a center role. In [2], the author proved
the uniform convergence of $u$ and the convergence of $m$ in the distance
$d_{1}$ to stationary solutions when the Hamiltonian of the form $|p|^{2}/2$.
For mean field games with non-monotone coupling term, there are also some
works on the existence of time periodic solutions. For congestion problem,
[19] discussed the traveling wave solution, which is the first example of time
periodic solutions. For second order mean field games with the Hamiltonian of
the form $|p|^{2}/2$, [15] proved the existence of time periodic solutions,
and [16] discussed the existence of oscillating solutions when there are two
populations of agents. In [14], by discussing the minimal points of an energy
functional, the authors found time periodic solutions of first order mean
field games on $\mathbb{R}^{n}$.

In [26], in order to generalize the Aubry-Mather theory for twist maps, J.
Mather introduced the concept of minimal measures, which are also called the
Mather measures. Assume the Hamiltonian $H:T^{*}M\to\mathbb{R}$ is $C^{2}$,
and satisfies the Tonelli condition

  * (H1)

Strictly convexity: the Hessian matrix $\partial^{2}_{pp}H(x,p)$ is positively
definite for all $(x,p)\in T^{*}M$.

  * (H2)

Superlinearity: $H(x,p)/|p|\to+\infty$ as $|p|\to+\infty$ for all $x\in M$.

Define the Legendre dual of $H$, which is called the Lagrangian as follows

$L(x,\dot{x}):=\sup_{p\in T^{*}_{x}M}\\{\langle\dot{x},p\rangle-H(x,p)\\}.$

Then $L:TM\to\mathbb{R}$ is also of class $C^{2}$, and satisfies the Tonelli
condition with respect to $\dot{x}$ (cf. [6, Appendix]). For each $x,y\in M$,
consider the action minimizing problem

$h_{t}(x,y):=\min\int_{0}^{t}L(\gamma(s),\dot{\gamma}(s))ds,$ (1.7)

where the minimum is taken among absolutely continuous curves
$\gamma:[0,t]\rightarrow M$ with $\gamma(0)=x$ and $\gamma(t)=y$. Here $h_{t}$
is called the minimal action. By the Tonelli Theorem, the minimum can be
achieved, see [18]. The curves achieving the minimum satisfy the Euler-
Lagrange equation

$\frac{d}{dt}\bigg{(}\frac{\partial
L}{\partial\dot{x}}(x,\dot{x})\bigg{)}=\frac{\partial L}{\partial
x}(x,\dot{x}).$

Denote by $\Phi^{L}_{t}$ the flow of the Euler-Lagrange equation. Denote by
$\Phi^{H}_{t}$ the dual flow of $\Phi^{L}_{t}$ through the Legendre
transformation, which is the flow of the Hamilton equation

$\dot{x}=\frac{\partial H}{\partial p}(x,p),\quad\dot{p}=-\frac{\partial
H}{\partial x}(x,p).$ (1.8)

In the following, we denote by $m$ the measures on $M$ and $\mu$ the measures
on $TM$. Consider the action minimizing problem

$-c_{0}=\min_{\mu}\int_{TM}L(x,\dot{x})d\mu,$

where $\mu$ varies among all Borel probability measures on $TM$ invariant
under the action of the flow $\Phi^{L}_{t}$. The constant $c_{0}$ is the
unique constant $c\in\mathbb{R}$ such that $H(x,Du)=c$ has solutions, and is
called the critical value or the effective Hamiltonian, cf. [25, Theorem 1].
We call the measures achieve this minimum the minimal measures, and denote by
$\tilde{\mathfrak{M}}$ the set of all minimal measures. Define the set of all
projected minimal measures as

$\mathfrak{M}:=\\{m\in\mathcal{P}(M):\ m=\pi_{\\#}\mu,\
\mu\in\tilde{\mathfrak{M}}\\},$

where $\pi:TM\to M$ is the canonical projection, and $\pi_{\\#}$ is the push-
forward induced by $\pi$. The Mather set is defined as

$\tilde{\mathcal{M}}=\overline{\bigcup_{\mu}\ \textrm{supp}(\mu)}\subset TM,$

where $\textrm{supp}(\mu)$ is the support of $\mu$, $\bar{A}$ is the closure
of $A\subset TM$, and the union is taken over all minimal measures
$\mu\in\tilde{\mathfrak{M}}$. Let $\mathcal{M}=\pi(\tilde{\mathcal{M}})$ be
the projected Mather set. By [26], $\tilde{\mathcal{M}}$ is a Lipschitz graph
over $\mathcal{M}$.

Now we show that Mather’s minimal measures give solutions of (1.4)-(1.5). Let
$\mu\in\tilde{\mathfrak{M}}$. Since $\mu$ is $\Phi^{L}_{t}$-invariant, by
[26], we have

$\int_{TM}\langle\dot{x},d\varphi(x)\rangle d\mu=0,$

By [18], let $u_{0}$ be a solution of

$H(x,Du)=c_{0}.$ (1.9)

Then $u_{0}$ is differentiable on $\mathcal{M}$, cf. [18, Theorem 5.2.2]. The
derivative of $u_{0}$ is given by

$Du_{0}(x)=\frac{\partial
L}{\partial\dot{x}}(x,\dot{x}),\quad(x,\dot{x})\in\tilde{\mathcal{M}},$ (1.10)

which is independent of $u_{0}$, and is Lipschitz continuous. Let
$m_{0}=\pi_{\\#}\mu\in\mathfrak{M}$ be a projected minimal measure. Then

$\int_{M}\bigg{\langle}\frac{\partial H}{\partial
p}(x,Du_{0}(x)),d\varphi(x)\bigg{\rangle}dm_{0}=\int_{\tilde{\mathcal{M}}}\langle\dot{x},d\varphi(x)\rangle
d\mu=0,$

which implies that $(u_{0},m_{0})$ is a solution of (1.4)-(1.5) with
$c=c_{0}-F(m_{0})$. Assume

  * (H3)

there is a non-trivial periodic orbit $\Gamma(t)=(\gamma(t),\dot{\gamma}(t))$
on $\tilde{\mathcal{M}}$ with period $\tau$.

Here we recall that, [28] proved that the minimal measure is unique for
generic Lagrangian. It was conjectured by R. Mañé that, the unique minimal
measure is supported on a fixed point or a periodic orbit.

######  Remark 1.1.

For the one dimensional case $M=\mathbb{T}\simeq\mathbb{R}/\mathbb{Z}$, we can
give more description on (H3) from the perspective of the Mather’s theory. Let
$\alpha:\mathbb{R}\to\mathbb{R}$ be the Mather’s $\alpha$-function on the
cohomology (see [18, Section 4.10]). Then for each $a\in\mathbb{R}$,
$\alpha(a)$ is the unique constant $c$ such that $H(x,a+Du)=c$ is solvable. By
Lemma 4.1 below, the Mather set consists of either fixed points or exactly one
periodic orbit. Assume that there is a fixed point in the Mather set
$\tilde{\mathcal{M}}$. Then the rotation number of all minimal measures is
equal to zero, Equivalently, $\alpha^{\prime}(a)=0$, one can see [26] for more
details. If (H3) holds, then $\alpha^{\prime}(a)\neq 0$. Consider

$H_{a}(x,p)=\frac{1}{2}(p+a)^{2}+V(x),$

where $V:\mathbb{T}\to\mathbb{R}$ that attains its maximum at exactly two
points $0$ and $X$ and such that $V(0)=V(X)=0$. Let
$a_{0}:=\int_{0}^{1}\sqrt{-2V(x)}dx$. Then $\alpha(a)=0$ for
$a\in[-a_{0},a_{0}]$ and $\alpha(a)>0$ for $a\notin[-a_{0},a_{0}]$ (see [30,
Chapter 4] for details). Since $\alpha:\mathbb{R}\to\mathbb{R}$ is convex (see
[26]), $\alpha^{\prime}(a)\neq 0$ holds for $a\notin[-a_{0},a_{0}]$, which
implies that (H3) holds.

Define

$\mathfrak{N}:=\\{m\notin\mathfrak{M},\
\textrm{supp}(m)\subset\\{\gamma\\}\\}\subset\mathcal{P}(M).$

######  Theorem 1.

Assume (H1)-(H3)(F) and $m_{T}\in\mathfrak{N}$, then there is
$c(m_{T})\in\mathbb{R}$ such that the system (1.1)-(1.3) has a non-trivial
time periodic solution with period $\tau$ when $c=c(m_{T})$.

######  Theorem 2.

Assume $M=\mathbb{T}$, (H1)-(H3)(F). Given
$m_{T}\in\mathcal{P}(M)\backslash\mathfrak{M}$, then there is
$c(m_{T})\in\mathbb{R}$ such that the system (1.1)-(1.3) has a non-trivial
time periodic solution $(\bar{u}(x,t),\bar{m}(t))$ when $c=c(m_{T})$. If
$m_{T}$ is an absolutely continuous measure with respect to the Lebesgue
measure on $\mathbb{T}$, then it has the density $m_{T}(x)$, and we have

  * (1)

$\bar{u}(x,t)\in C^{1}(\mathbb{T}\times[0,T])$ is unique up to a constant,
$\bar{m}(x,t)$ is unique, and $c(m_{T})$ is the unique constant
$c\in\mathbb{R}$ such that (1.1)-(1.3) has time periodic solutions.

  * (2)

The map $m_{T}\mapsto c(m_{T})$ is Lipschitz continuous in $d_{1}$.

  * (3)

If $m_{T}(x)\in C^{1}(\mathbb{T})$, then $\bar{m}(x,t)\in
C^{1}(\mathbb{T}\times[0,T])$.

  * (4)

For each $\varphi\in C(\mathbb{T})$ and $c\in\mathbb{R}$, there is a unique
weak solution $(u(x,t),m(t))\in C(\mathbb{T}\times[0,T])\times
Lip([0,T],\mathcal{P}(\mathbb{T}))$ of (1.1)-(1.3). Let $c=c(m_{T})$. If
$m_{T}(x)$ is bounded, then for each $t\in\mathbb{[}0,+\infty)$, we have

$u(x,s)-\int_{0}^{T-t}F(m(\tau))d\tau\to\
\bar{u}(x,s)-\int_{0}^{T-t}F(\bar{m}(\tau))d\tau\quad\textrm{as}\quad
T\to+\infty$

uniformly in $(x,s)\in\mathbb{T}\times[T-t,T]$ and

$d_{1}(m(s),\bar{m}(s))\to 0\quad\textrm{as}\quad T\to+\infty$

uniformly in $s\in[T-t,T]$. Here $(\bar{u}(x,t),\bar{m}(t))$ is a time
periodic solution.

######  Remark 1.2.

The main differences between the present work and [14] are

  * •

The method in the present paper is from the weak KAM theory and the Aubry-
Mather theory instead of finding minimal points of an energy functional.

  * •

In [14], the potential of the coupling term is a sort of “double-well”
potential. Then the convexity of the potential does not hold, which implies
that the coupling term is not monotone. In the present paper, as stated
before, the monotone condition (1.6) holds.

  * •

The time periodic solutions given in the present paper are weak solutions
defined in Definition 1.2, which may have more regularity. In particular, for
$M=\mathbb{T}$, the time periodic solutions can be classical solutions of
class $C^{1}$, see Items (1)(3) in Theorem 2.

  * •

Item (4) in Theorem 2 considers the uniform convergence of $u$ and the
convergence of $m$ in $d_{1}$ as the horizon $T\to+\infty$.

At last, we provide a specific example of the non-trivial smooth time periodic
solution of first order mean field games.

######  Example 1.1.

Consider $x\in\mathbb{T}^{n}\simeq[0,1]^{n}$ and

$\displaystyle\partial_{t}u(x,t)+\sum_{i=1}^{n}\bigg{(}\frac{1}{2}(\partial_{x_{i}}u(x,t))^{2}-\partial_{x_{i}}u(x,t)\bigg{)}=F(m(t)),$
(1.11)
$\displaystyle\partial_{t}m(x,t)+\sum_{i=1}^{n}\partial_{x_{i}}\big{(}(\partial_{x_{i}}u(x,t)-1)m(x,t)\big{)}=0.$
(1.12)

where

$F(m)=\int_{[0,1]^{n}}4\pi\cos\big{(}2\pi\sum_{i=1}^{n}x_{i}\big{)}m(x)dx.$

It is direct to check that

$\left\\{\begin{aligned} &\bar{u}(x,t)=\sin(2\pi t),\\\
&\bar{m}(x,t)=1+\cos\big{(}2\pi(\sum_{i=1}^{n}x_{i}+t)\big{)}.\\\
\end{aligned}\right.$

is a non-trivial time periodic solution of the above mean field games model
(1.11)-(1.12). At the same time, there is a stationary solution
$(u_{0},m_{0})=(0,1)$.

##  2 Proof of Theorem 1

Let $\Gamma(t)=(\gamma(t),\dot{\gamma}(t))$ with $t\in\mathbb{R}$ be a
periodic orbit on $\tilde{\mathcal{M}}$ with period $\tau$. Consider
$m_{T}\in\mathfrak{N}$, that is, $m_{T}\notin\mathfrak{M}$ supported on
$\\{\gamma\\}\subseteq\mathcal{M}$. For $x=\gamma(t_{0})$, define

$\Phi(t,T,x)=\gamma(t+t_{0}-T).$

Let $u_{0}(x)$ be a viscosity solution of (1.9). Define

$\left\\{\begin{aligned}
&\bar{u}(x,t)=u_{0}(x)+\int_{0}^{t}F(\bar{m}(s))ds-\frac{t}{\tau}\int_{0}^{\tau}F(\bar{m}(s))ds,\\\
&\bar{m}(t)=\Phi(t,T,\cdot)_{\\#}m_{T}.\\\ \end{aligned}\right.$ (2.1)

######  Lemma 2.1.

$\bar{m}(t)$ is periodic with period $\tau$.

###### Proof.

By the definition of push-forward, it is direct to see that for all $f\in
C(M)$, we have

$\int_{M}f(x)d\bar{m}(t)=\int_{\gamma}f(x)d\bar{m}(t)=\int_{\gamma}f(\Phi(t,T,x))dm_{T},$

and

$\displaystyle\int_{M}f(x)d\bar{m}(t+\tau)$
$\displaystyle=\int_{\gamma}f(\Phi(t+\tau,T,x))dm_{T}$
$\displaystyle=\int_{\gamma}f(\Phi(t,T,x))dm_{T}=\int_{M}f(x)d\bar{m}(t),$

which implies that $t\mapsto\bar{m}(t)$ is periodic with period $\tau$. ∎

Note that $u_{0}$ is differentiable on $\gamma$, where $Du_{0}(x)$ is
Lipschitz continuous and is given by (1.10). Since $m_{T}$ is supported on
$\gamma$ and $\gamma$ is invariant under $\Phi(t,T,\cdot)$, similar to [13,
Section 4.2], we can prove that $\bar{m}(t)$ is the unique weak solution of

$\partial_{t}m(x,t)+\textrm{div}(\partial_{p}H(x,Du_{0}(x))m(x,t))=0,\quad
m(x,T)=m_{T}(x).$

Also, one can check that $\bar{u}(x,t)$ is the unique solution of

$\partial_{t}u(x,t)+H(x,Du(x,t))=F(\bar{m}(t))+c_{0}-\frac{1}{\tau}\int_{0}^{\tau}F(\bar{m}(s))ds,\quad
u(x,0)=u_{-}(x).$

Note that $Du(x,t)=Du_{0}(x)$, we get a time periodic solution
$(\bar{u}(x,t),\bar{m}(x,t))$ of (1.1)-(1.3) with period $\tau$, where

$c(m_{T})=c_{0}-\frac{1}{\tau}\int_{0}^{\tau}F(\bar{m}(s))ds.$ (2.2)

######  Lemma 2.2.

The periodic solution $\bar{m}(t)$ is non-trivial.

###### Proof.

First we recall that $\tilde{\mathcal{M}}$ is a Lipschitz graph over
$\mathcal{M}$, then the lift $\mu_{T}$ of $m_{T}$ to $\tilde{\mathcal{M}}$ is
unique. Since $m_{T}\in\mathfrak{N}$, $\mu_{T}\notin\tilde{\mathfrak{M}}$.
According to [27], all invariant measures supported on $\tilde{\mathcal{M}}$
are minimal measures, $\mu_{T}$ is not invariant under $\Phi^{L}_{t}$. Since
$\textrm{supp}(\mu_{T})\subset\Gamma$, $\Phi^{L}_{t-T\\#}\mu_{T}$ is periodic
in $t$ with period $\tau$. For every
$(x,\dot{x})=\Gamma(t_{0})\in\textrm{supp}(\mu_{T})$, we have

$\Phi^{L}_{t-T}(\Gamma(t_{0}))=\Gamma(t+t_{0}-T).$

Let $y=\Phi(t,T,x)$, define the inverse $x=\Phi^{-1}(t,T,y)$. Then we have

$\pi^{-1}\Phi^{-1}(t,T,x)=\Gamma(t_{0}-(t-T))=(\Phi^{L}_{t-T})^{-1}\pi^{-1}(x),\quad\forall
x\in\\{\gamma\\}.$

Thus, for every $A\subseteq\\{\gamma\\}$, we have

$\displaystyle\pi_{\\#}(\Phi^{L}_{t-T\\#}\mu_{T})(A)$
$\displaystyle=\mu_{T}((\Phi^{L}_{t-T})^{-1}(\pi^{-1}(A)))$ (2.3)
$\displaystyle=\mu_{T}(\pi^{-1}(\Phi^{-1}(t,T,A)))=\Phi(t,T,\cdot)_{\\#}(\pi_{\\#}\mu_{T})(A).$

Since $\pi^{-1}$ is a bi-Lipschitz continuous map on $\gamma$ by [26], and
$\mu_{T}$ is not invariant under $\Phi^{L}_{t}$, there is
$A\subseteq\\{\gamma\\}$ and $t_{1}\in(T-\tau,T)$ such that

$\Phi^{L}_{t_{1}-T\\#}\mu_{T}(\pi^{-1}(A))\neq\mu_{T}(\pi^{-1}(A)).$

Therefore, by (2.3) we get

$\displaystyle m_{T}(A)$
$\displaystyle=\pi_{\\#}\mu_{T}(A)=\mu_{T}(\pi^{-1}(A))$
$\displaystyle\neq\Phi^{L}_{t_{1}-T\\#}\mu_{T}(\pi^{-1}(A))=\pi_{\\#}(\Phi^{L}_{t_{1}-T\\#}\mu_{T})(A)=\Phi(t_{1},T,\cdot)_{\\#}(\pi_{\\#}\mu_{T})(A)$
$\displaystyle=\Phi(t_{1},T,\cdot)_{\\#}m_{T}(A)=\bar{m}(t_{1})(A).$

Thus, $\bar{m}(t)$ is non-trivial. ∎

##  3 Basic properties of (1.1)-(1.3)

According to [18], the viscosity solution of

$\left\\{\begin{aligned} &\partial_{t}w(x,t)+H(x,Dw(x,t))=0,\quad(x,t)\in
M\times(0,+\infty).\\\ &u(x,0)=\varphi(x),\quad x\in M.\\\
\end{aligned}\right.$

has the following representation formula called the Lax-Oleinik formula

$w(x,t)=\min_{z\in
M}\bigg{\\{}\varphi(z)+h_{t}(z,x)\bigg{\\}}=\min_{\gamma}\bigg{\\{}\varphi(\gamma(0))+\int_{0}^{t}L(\gamma(s),\dot{\gamma}(s))ds\bigg{\\}},$
(3.1)

where $h_{t}$ is defined by (1.7) and $\gamma:[0,t]\to M$ is taken among
absolutely continuous curves with $\gamma(t)=x$. By the Tonelli Theorem, the
minimum in the definition of $w(x,t)$ can be achieved, and we call the curve
achieving the minimum a minimizer. We have known that $w(x,t)$ is uniformly
bounded and equi-Lipschitz continuous for all $t>1$. Moreover, $w(x,t)+c_{0}t$
uniformly converges to a solution of (1.9) as $t\to+\infty$. For these
results, one can refer to [18, Proposition 4.6.6 and Theorem 6.3.1].

Similar to [13, Section 4.1], for each $(x,t)\in M\times[0,T]$, there is a
measurable selection $\gamma(\cdot;x,t):[0,t]\to M$ of minimizers of $w(x,t)$.
Then we can define the optimal flow as

$\Psi(s,t,x)=\gamma(s;x,t),\quad s\in[0,t].$

Similar to the proof of [13, Theorem 4.18], we can prove that

######  Lemma 3.1.

If $m_{T}\in\mathcal{P}(M)$ is absolutely continuous with respect to the
Lebesgue measure, $m(t)=\Psi(t,T,\cdot)_{\\#}m_{T}$ is the unique weak
solution of

$\partial_{t}m(x,t)+\textrm{div}(\partial_{p}H(x,Dw(x,t))m(x,t))=0,\quad
m(T)=m_{T}.$ (3.2)

Outline of the proof. By [6, Theorem 6.4.7 and Theorem 6.4.8],
$Dw(\Psi(s,t,x),s)$ exists for $s\in(0,t)$, and

$Dw(\Psi(s,t,x),s)=\partial_{\dot{x}}L(\Psi(s,t,x),\partial_{s}\Psi(s,t,x)).$

Similar to [13, Lemma 4.15], $m(t)$ is a weak solution of (3.2). It remains to
prove the uniqueness of the weak solution of (3.2). The proof is based on the
fact that, if $w(x,T)$ is differentiable at $x$, then the minimizer
$\gamma(\cdot;x,T):[0,T]\to M$ of $w(x,T)$ is unique (cf. [6]). Therefore, the
optimal flow $\Psi(t,T,x)$ is uniquely given by $\gamma(t;x,T)$. Since
$x\mapsto w(x,T)$ is Lipschitz continuous, the optimal flow $\Psi(t,T,x)$ is
uniquely defined for almost all $x\in M$. Since $m_{T}$ is absolutely
continuous with respect to the Lebesgue measure, the optimal flow
$\Psi(t,T,x)$ is uniquely defined for $m_{T}$-almost all $x\in M$. Thus, the
weak solution of (3.2) is uniquely given by $\Psi(t,T,\cdot)_{\\#}m_{T}$.∎

######  Lemma 3.2.

For each $c\in\mathbb{R}$, $\varphi(x)\in C(M)$, and $m_{T}\in\mathcal{P}(M)$
which is absolutely continuous with respect to the Lebesgue measure, there is
a unique weak solution $(u(x,t),m(x,t))\in C(M\times[0,T])\times
Lip([0,T],\mathcal{P}(M))$ of (1.1)-(1.3).

###### Proof.

For all $m\in\mathcal{P}(M)$, the operator norm of $m$ is bounded. Then
$\mathcal{P}(M)$ is compact in the $w^{*}$-topology. By the
$w^{*}$-compactness of $\mathcal{P}(M)$, $F(m)$ is bounded. For each
measurable map $t\mapsto\hat{m}(t)$, $F(\hat{m}(s))\in L^{1}([0,T])$. Then
according to [1, Theorem 5.1],

$u(x,t;\hat{m})=w(x,t)+\int_{0}^{t}F(\hat{m}(s))ds+ct$

is the unique solution of

$\partial_{t}u(x,t)+H(x,Du(x,t))=F(\hat{m}(t))+c,\quad u(x,0)=\varphi(x).$

To see the uniqueness of solution of (1.1)-(1.3), let $(u(x,t),m(x,t))$ be a
solution. Note that $Du(x,t;\hat{m})=Dw(x,t)$ for all $\hat{m}$, by Lemma 3.1,
$m(t)=\Psi(t,T,\cdot)_{\\#}m_{T}$ is unique. Then the unique solution of
(1.1)-(1.3) is given by

$\left\\{\begin{aligned} &u(x,t)=w(x,t)+\int_{0}^{t}F(m(s))ds+ct,\\\
&m(t)=\Psi(t,T,\cdot)_{\\#}m_{T}.\\\ \end{aligned}\right.$ (3.3)

It remains to prove that $t\mapsto m(t)$ is Lipschitz continuous. We have

$\displaystyle d_{1}(m(t),m(s))$
$\displaystyle=\sup_{\phi}\int_{M}\phi(x)d(m(t)-m(s))$
$\displaystyle=\sup_{\phi}\int_{M}(\phi(\Psi(t,T,x))-\phi(\Psi(s,T,x)))dm_{T}$
$\displaystyle\leq\int_{M}d(\Psi(t,T,x),\Psi(s,T,x))dm_{T}.$

By [6, Theorem 6.3.3], $(\Psi(t,T,x),p(t))$ satisfies (1.8), where the dual
arc is defined by

$p(t)=\frac{\partial
L}{\partial\dot{x}}(\Psi(t,T,x),\partial_{t}\Psi(t,T,x)).$

For almost all $x\in M$ and $t\in[0,T]$, by [6, Theorem 6.4.9] we have

$H(\Psi(t,T,x),p(t))=H(x,Dw(x,T)).$

Since $\|Dw(x,T)\|_{\infty}$ is bounded for each given $T>0$, $p(t)$ is
uniformly bounded by (H2). Then

$\partial_{t}\Psi(t,T,x)=\frac{\partial H}{\partial p}(\Psi(t,T,x),p(t))$

is also bounded uniformly in $x\in M$. Therefore,

$d_{1}(m(t),m(s))\leq\|\partial_{t}\Psi(t,T,x)\|_{\infty}|t-s|,$

which implies that $m(t)\in Lip([0,T],\mathcal{P}(M))$. ∎

######  Lemma 3.3.

The family $\\{w(x,t)\\}_{t\geq 1}$ is equi-semi-concave. Moreover, let
$u_{0}:=\lim_{t\to+\infty}w(x,t)$. Then $Dw(x,t)\to Du_{0}(x)$ for almost
every $x\in M$.

###### Proof.

By [3, Theorem B.7], the map $(y,x)\mapsto h_{t}(y,x)$ is equi-semi-concave
for all $t\geq 1$. Since the argument is local, without any loss of
generality, we assume that $M$ is an open subset $U$ of $\mathbb{R}^{n}$. Let
$x$, $y\in U$, $\lambda\in[0,1]$. We denote by $z_{\lambda}$ the minimal point
in (3.1) with the value function equals $w(\lambda x+(1-\lambda)y,t)$. Then we
have

$\displaystyle\lambda w(x,t)+(1-\lambda)w(y,t)-w(\lambda x+(1-\lambda)y,t)$
$\displaystyle\leq$ $\displaystyle\lambda
h_{t}(z_{\lambda},x)+(1-\lambda)h_{t}(z_{\lambda},y)-h_{t}(z_{\lambda},\lambda
x+(1-\lambda)y)$ $\displaystyle\leq$ $\displaystyle
C\lambda(1-\lambda)|x-y|^{2},$

where $C$ is the semi-concave constant of $\\{h_{t}\\}_{t\geq 1}$. Therefore,
$\\{w(x,t)\\}_{t\geq 1}$ is equi-semi-concave. The last statement holds
according to [6, Theorem 3.3.3] or [13, Lemma 4.6]. ∎

##  4 Proof of Theorem 2

We first consider the uniqueness and smoothness of the solution of (1.9).

######  Lemma 4.1.

Let (H1)(H2) hold, and $M=\mathbb{T}$. The Mather set $\tilde{\mathcal{M}}$ of
(1.9) consists of either fixed points or a periodic orbit
$\Gamma=(\gamma,\dot{\gamma})$. For the last case, we have

  * (1)

$\mathcal{M}=\\{\gamma\\}=\mathbb{T}$. Let
$\bar{\gamma}:\mathbb{R}\to\mathbb{R}$ be the lift of
$\gamma:\mathbb{R}\to\mathbb{T}$ on the cover space $\mathbb{R}$ over
$\mathbb{T}$. Then $\bar{\gamma}:\mathbb{R}\to\mathbb{R}$ is monotone and
$\dot{\gamma}\neq 0$.

  * (2)

The viscosity solution $u_{0}$ of (1.9) is of class $C^{2}$ and unique up to a
constant.

###### Proof.

Since $M=\mathbb{T}$, by the Poincaré-Bendixon Theorem for planar dynamical
systems and the recurrence property of the Mather set, $\tilde{\mathcal{M}}$
consists of fixed points or periodic orbits. Let
$\Gamma(t)=(\gamma(t),\dot{\gamma}(t))$ be a time periodic orbit contained in
$\tilde{\mathcal{M}}$. Assume $t\mapsto\bar{\gamma}(t)$ is not monotone. Then
there are two times $t_{1}<t_{2}$, such that $\gamma(t_{1})=\gamma(t_{2})$ and
$\dot{\gamma}_{-}(t_{1})\neq\dot{\gamma}_{-}(t_{2})$, which contradicts the
graph property. Therefore, $t\mapsto\bar{\gamma}(t)$ is monotone. Since
$\gamma$ is periodic, $\mathcal{M}=\\{\gamma\\}=\mathbb{T}^{1}$.

Let $u_{0}$ be a solution of (1.9). By $\mathcal{M}=\mathbb{T}$ and (1.10),
$u_{0}$ is differentiable on $\mathbb{T}$ with the same derivative. By the
Lipschitz graph property of $\tilde{\mathcal{M}}$, $Du_{0}(x)$ is Lipschitz
continuous. Therefore, $u_{0}$ is of class $C^{1,1}$. For each
$x_{0}\in\mathbb{T}$ and each solution $v_{0}$ of (1.9), we have

$v_{0}(x)=v_{0}(x_{0})+\int_{x_{0}}^{x}Dv_{0}(x)dx=v_{0}(x_{0})+\int_{x_{0}}^{x}Du_{0}(x)dx.$

Then $u_{0}$ is unique up to a constant.

We then show that $\dot{\gamma}\neq 0$. Since $p(x):=Du_{0}(x)$ is Lipschitz
continuous, $Dp(x)$ exists almost everywhere. Here we recall the reachable
gradient of the Lipschitz continuous function $p:\mathbb{T}\to\mathbb{R}$ (see
[6])

$D^{*}p(x):=\\{q:\ \exists x_{n}\to x,\ Dp(x_{n})\ \textrm{exists\ and}\
Dp(x_{n})\to q\\},$

which is compact. Assume there is $t_{0}$ such that $\dot{\gamma}(t_{0})=0$.
Take $x_{n}\to\gamma(t_{0})$ with $Dp(x_{n})$ exists. Since
$t\mapsto\bar{\gamma}(t)$ is monotone and $\\{\gamma\\}=\mathbb{T}$, there is
$t_{n}\to t_{0}$ with $\gamma(t_{n})=x_{n}$. Then we have

$\dot{p}(\gamma(t_{n}))=\frac{d}{dt}\bigg{|}_{t=t_{n}}Du_{0}(\gamma(t))=\frac{d}{dt}\bigg{|}_{t=t_{n}}p(\gamma(t))=Dp(\gamma(t_{n}))\cdot\dot{\gamma}(t_{n}).$

Define $p(t)=Du_{0}(\gamma(t))$, then $(\gamma(t),p(t))$ satisfies the
Hamilton equation (1.8). Note that $t\mapsto(\gamma(t),p(t))$ is $C^{1}$, then
$t\mapsto(\dot{\gamma}(t),\dot{p}(t))$ is continuous. Let $t_{n}\to t_{0}$. By
continuity, we conclude that $\dot{p}(t_{0})=0$. Then
$(\gamma(t_{0}),p(t_{0}))$ is a fixed point on $\tilde{\mathcal{M}}$, which
leads to a contradiction.

It remains to show that $u_{0}$ is $C^{2}$, that is, $Du_{0}(x)$ is $C^{1}$.
Since $\dot{\gamma}\neq 0$, there is a inverse for $t\mapsto\gamma(t)$, which
is denoted by $t(x)$. Let
$p(t)=\partial_{\dot{x}}L(\gamma(t),\dot{\gamma}(t))$ and $p(x)=p(t(x))$.
Since $\dot{\gamma}\neq 0$, we have

$\dot{\gamma}=\frac{\partial H}{\partial p}(\gamma(t),p(t))\neq 0.$ (4.1)

Then by (1.8), $(t(x),p(x))$ satisfies

$\frac{dt}{dx}=(\partial_{p}H(x,p(x)))^{-1},\quad\frac{dp}{dx}=-(\partial_{p}H(x,p(x)))^{-1}\partial_{x}H(x,p(x)).$

Since $H$ is $C^{2}$, $(t(x),p(x))$ is $C^{1}$. By (1.10), $p(x)=Du_{0}(x)$ is
$C^{1}$. ∎

Since $\\{\gamma\\}=\mathbb{T}$ by Lemma 4.1, $m_{T}\in\mathfrak{N}$ in the
present case. The existence of non-trivial time periodic solutions of
(1.1)-(1.3) is given in Theorem 1.

###  4.1 Proof of Item (1)

Now we prove the uniqueness of the time periodic solution. Let
$(\hat{u}(x,t),\hat{m}(x,t))$ be a time periodic solution of (1.1)-(1.3) with
period $\bar{\tau}$, where the constant $c\in\mathbb{R}$ may not be equal to
$c(m_{T})$. By (3.3), Lemma 3.3 and the periodicity of $\hat{u}(x,t)$, for
$n\in\mathbb{N}$, we have

$D\varphi(x)=\lim_{n\to+\infty}D\hat{u}(x,n\bar{\tau})=\lim_{n\to+\infty}Dw(x,n\bar{\tau})=Du_{0}(x)$

for almost every $x\in\mathbb{T}$. Therefore, $\varphi=u_{0}$, which implies
that

$\hat{u}(x,t)=u_{0}(x)+\int_{0}^{t}F(m(s))ds+(c-c_{0})t.$

Note that $D\hat{u}(x,t)=Du_{0}(x)$. By Lemma 3.2, $\hat{m}(t)$ is uniquely
given by $\Phi(t,T,\cdot)_{\\#}m_{T}$, which equals to $\bar{m}(t)$. Then

$c=c_{0}-\frac{1}{\tau}\int_{0}^{\tau}F(\bar{m}(s))ds$

is the unique constant such that $\hat{u}(x,t)$ is bounded. We finally
conclude that $(\hat{u}(x,t),\hat{m}(x,t))$ is the time periodic solution
given in (2.1). Since $t\mapsto F(m(t))$ is continuous, $\hat{u}(x,t)\in
C^{1}(\mathbb{T}^{1}\times[0,T])$. By Lemma 4.1, $\hat{u}(x,t)$ is unique up
to a constant.

###  4.2 Proof of Item (2)

######  Lemma 4.2.

For $x$ and $y\in\mathbb{T}$, there is $K_{1}>0$ such that

$d(\Phi(t,T,x),\Phi(t,T,y))\leq K_{1}d(x,y),\quad t\in[T-\tau,T].$

###### Proof.

Since the argument is local, we assume $\mathbb{T}$ is an open subset of
$\mathbb{R}$. Then the distance $d(x,y)=|x-y|$. By Lemma 4.1 (1), the lift
$\bar{\gamma}:\mathbb{R}\to\mathbb{R}$ is monotone. Then for $x\geq y$, we
have $\Phi(t,T,x)-\Phi(t,T,y)\geq 0$. By Lemma 4.1 (2), both
$\|Du_{0}\|_{\infty}$ and $\|D^{2}u_{0}\|_{\infty}$ is bounded. Therefore,
there is $K_{2}>0$ large enough such that

$\displaystyle\frac{d}{dt}(\Phi(t,T,x)-\Phi(t,T,y))$
$\displaystyle=\partial_{p}H(\Phi(t,T,x),Du_{0}(\Phi(t,T,x)))-\partial_{p}H(\Phi(t,T,y),Du_{0}(\Phi(t,T,y)))$
$\displaystyle\leq K_{2}(\Phi(t,T,x)-\Phi(t,T,y)),$

where $\Phi(T,T,x)-\Phi(T,T,y)=x-y$. By the comparison property of ordinary
differential equations, we have

$0\leq\Phi(t,T,x)-\Phi(t,T,y)\leq e^{\tau K_{2}}d(x,y)$

for $x\geq y$ and $t\in[T-\tau,T]$. ∎

For two different final condition $m^{1}_{T}$ and $m^{2}_{T}$, let
$\bar{m}_{i}(t)=\Phi(t,T,\cdot)_{\\#}m^{i}_{T}$, $i=1,2$. For
$t\in[T-\tau,T]$, we have

$d_{1}(\bar{m}_{1}(t),\bar{m}_{2}(t))=\sup_{\phi}\int_{\mathbb{T}}\phi(\Phi(t,T,x))d(m^{1}_{T}-m^{2}_{T})\leq
K_{1}d_{1}(m^{1}_{T},m^{2}_{T}),$

since $x\mapsto\phi(\Phi(t,T,x))$ is $K_{1}$-Lipschitz continuous by Lemma
4.2. By (2.2) and (F),

$\displaystyle|c(m^{1}_{T})-c(m^{2}_{T})|$
$\displaystyle=\frac{1}{\tau}\int_{0}^{\tau}|F(\bar{m}_{1}(s))-F(\bar{m}_{2}(s))|ds$
$\displaystyle=\frac{1}{\tau}\int_{T-\tau}^{T}|F(\bar{m}_{1}(s))-F(\bar{m}_{2}(s))|ds$
$\displaystyle\leq\frac{1}{\tau}\int_{T-\tau}^{T}Cd_{1}(\bar{m}_{1}(s),\bar{m}_{2}(s))ds\leq
CK_{1}d_{1}(m^{1}_{T},m^{2}_{T}),$

which implies that $m_{T}\mapsto c(m_{T})$ is Lipschitz continuous in $d_{1}$.

###  4.3 Proof of Item (3)

Let $y=\Phi(t,T,x)$, define the inverse $x=\Phi^{-1}(t,T,y)$.

######  Lemma 4.3.

The map $(t,T,y)\mapsto\Phi^{-1}(t,T,y)$ is of class $C^{1}$.

###### Proof.

We prove this lemma on the cover $\mathbb{R}$ of $\mathbb{T}$. Since

$\frac{dx}{dt}=\frac{\partial H}{\partial p}(x,Du_{0}(x)),$

solving the above ordinary differential equation directly, we have

$\int_{y}^{\Phi^{-1}(t,T,y)}(\partial_{p}H(x,Du_{0}(x)))^{-1}dx=T-t.$ (4.2)

Let

$G(x):=\int(\partial_{p}H(x,Du_{0}(x)))^{-1}dx.$

By Lemma 4.1, $Du_{0}$ is of class $C^{1}$. By (4.1),

$v(x):=\frac{\partial H}{\partial p}(x,Du_{0}(x))\neq 0,$

and $v(x)$ is of class $C^{1}$. Then $G^{\prime}(x)=v(x)^{-1}\neq 0$, which
implies $G$ is invertible. We also have
$G^{\prime\prime}(x)=-v^{\prime}(x)v(x)^{-2}$, which implies that $G(x)$ is of
class $C^{2}$. Then it has a $C^{2}$ inverse $G^{-1}(x)$. By (4.2) we have

$G(\Phi^{-1}(t,T,y))-G(y)=T-t,$

which implies that $\Phi^{-1}(t,T,y)=G^{-1}(G(y)+T-t)$ is of class $C^{2}$. ∎

When $m_{T}$ is $C^{1}$, by definition we have

$\displaystyle\int_{\mathbb{T}}f(x)d\bar{m}(t)$
$\displaystyle=\int_{\mathbb{T}}f(x)d\Phi(t,T,\cdot)_{\\#}m_{T}=\int_{\mathbb{T}}f(\Phi(t,T,x))m_{T}(x)dx$
$\displaystyle=\int_{\mathbb{T}}f(y)m_{T}(\Phi^{-1}(t,T,y))d\Phi^{-1}(t,T,y)$
$\displaystyle=\int_{\mathbb{T}}f(y)m_{T}(\Phi^{-1}(t,T,y))\partial_{y}\Phi^{-1}(t,T,y)dy,\quad\forall
f\in C(\mathbb{T}),$

where $y=\Phi(t,T,x)$. Then we finally get

$\bar{m}(x,t)=m_{T}(\Phi^{-1}(t,T,x))\partial_{x}\Phi^{-1}(t,T,x),$

which is of class $C^{1}$.

###  4.4 Proof of Item (4)

The existence and uniqueness of the weak solution of (1.1)-(1.3) have been
proved in Lemma 3.2. Now we consider the large time behavior of the unique
solution of (1.1)-(1.3) for each $\varphi\in C(M)$, where $c=c(m_{T})$.
According to [6, Theorem 6.4.9], for almost every $x\in\mathbb{T}$,

$(x(s),p(s)):=(\Psi(s,T,x),Dw(\Psi(s,T,x),s)),\quad s\in(0,T)$

satisfies the Hamilton equation (1.8) with $(x(T),p(T))=(x,Dw(x,T))$. By Lemma
3.3 and the continuous dependence of solutions of ODEs on the final values,
for almost all $x$ and each $t\in[0,+\infty)$, we have
$\Psi(s,T,x)\to\Phi(s,T,x)$ uniformly for $s\in[T-t,T]$ as $T\to+\infty$.
Define

$f_{T}(x)=\max_{s\in[T-t,T]}d(\Psi(s,T,x),\Phi(s,T,x)),$

then $f_{T}(x)\to 0$ a.e. on $\mathbb{T}$.

For $m(s)$, we have

$\displaystyle d_{1}(m(s),\bar{m}(s))$
$\displaystyle=\sup_{\phi}\int_{\mathbb{T}}(\phi(\Psi(s,T,x))-\phi(\Phi(s,T,x)))dm_{T}(x)$
$\displaystyle\leq\int_{\mathbb{T}}d(\Psi(s,T,x),\Phi(s,T,x))m_{T}(x)dx$
$\displaystyle\leq\int_{\mathbb{T}}f_{T}(x)m_{T}(x)dx,\quad s\in[T-\tau,T].$

Since $m_{T}(x)$ is bounded and $\mathbb{T}$ is compact, by the dominated
convergence theorem,

$d_{1}(m(s),\bar{m}(s))\to 0,\quad\textrm{as}\quad T\to+\infty,$

uniformly for $s\in[T-t,T]$.

For $u(x,s)$, we first take suitable $u_{0}$. Then we have

$\displaystyle
u(x,s)-\int_{0}^{T-t}F(m(\xi))d\xi-\bigg{(}\bar{u}(x,s)-\int_{0}^{T-t}F(\bar{m}(\xi))d\xi\bigg{)}$
$\displaystyle=w(x,s)+c_{0}s-u_{0}(x)+\int_{T-t}^{s}(F(m(\xi))-F(\bar{m}(\xi)))d\xi,$

where $w(x,s)+c_{0}s$ uniformly converges to $u_{0}$ as $T\to+\infty$. By (F)
we have

$\displaystyle\bigg{|}\int_{T-t}^{s}(F(m(\xi))-F(\bar{m}(\xi)))d\xi\bigg{|}\leq
C\int_{T-\tau}^{T}d_{1}(m(s),\bar{m}(s))ds$ $\displaystyle\leq
C\int_{T-\tau}^{T}\int_{\mathbb{T}}d(\Psi(s,T,x),\Phi(s,T,x))m_{T}(x)dxds$
$\displaystyle=C\int_{0}^{\tau}\int_{\mathbb{T}}d(\Psi(T-\tau+s,T,x),\Phi(T-\tau+s,T,x))m_{T}(x)dxds$
$\displaystyle\leq C\int_{0}^{\tau}\int_{\mathbb{T}}f_{T}(x)m_{T}(x)dxds\to
0,\quad\textrm{as}\quad T\to+\infty,$

uniformly for $s\in[T-t,T]$ by the dominated convergence theorem.

## Acknowledgements

The author would like to thank Professor J. Yan for many helpful discussions.

## References

  * [1] G. Bales. _An introduction to the theory of viscosity solutions for first-order Hamilton-Jacobi equations and applications_ , Springer-Heidelberg, 49–109, 2013.
  * [2] M. Bardi and H. Kouhkouh. _Long-time behavior of deterministic mean field games with non-monotone interactions_ , arXiv:2304.09509.
  * [3] P. Bernard. _The dynamics of pseudographs in convex Hamiltonian systems_ , J. Amer. Math. Soc., 21 (2008), 615–669.
  * [4] P. Cannarsa, W. Cheng, C. Mendico and K. Wang. _Long-time behavior of first-order mean field games on Euclidean space_. Dyn. Games Appl., 10 (2020), 361–390.
  * [5] P. Cannarsa, W. Cheng, C. Mendico and K. Wang. _Weak KAM approach to first-order mean field games with state constraints_. J. Dyn. Differ. Equ., (2021), 1–32.
  * [6] P. Cannarsa and C. Sinestrari. Semiconcave functions, Hamilton-Jacobi equations, and optimal control. Vol. 58. Springer, 2004.
  * [7] P. Cardaliaguet. _Long time average of first order mean field games and weak KAM theory_. Dyn. Games Appl., 3 (2013), 473–488.
  * [8] P. Cardaliaguet and P. Graber. _Mean field games systems of first order_. ESAIM Control Optim. Calc. Var., 21 (2015), 690–722.
  * [9] P. Cardaliaguet, J.-M. Lasry, P.-L. Lions and A. Porretta. _Long time average of mean field games with a nonlocal coupling_. SIAM J. Control Optim. 51 (2013), 3358–3591.
  * [10] P. Cardaliaguet and A. Porretta. _Long time behavior of the master equation in mean field game theory_. Analysis & PDE, 12 (2019), 1397–1453.
  * [11] P. Cardaliaguet and M. Masoero. _Weak KAM theory for potential MFG_. J. Diff. Equ., 268 (2020), 3255–3298.
  * [12] P. Cardaliaguet and C. Mendico. _Ergodic behavior of control and mean field games problems depending on acceleration_. Nonlinear Analysis, 203 (2021), 112185.
  * [13] P. Cardaliaguet. _Notes on mean field games_ , (2013), http://www.ceremade.dauphine.fr/ cardaliaguet/MFG20130420.pdf
  * [14] A. Cesaroni and M. Cirant. _Brake orbits and heteroclinic connections for first order mean field games_. Tran. Amer. Math. Soc., 374 (2021), 5037–5070.
  * [15] M. Cirant and L. Nurbekyan. _The variational structure and time-periodic solutions for mean-field games systems_. Minimax Theory Appl., 3 (2018), 227–260.
  * [16] M. Cirant. _On the existence of oscillating solutions in non-monotone mean-field games_. J. Diff. Equ., 266 (2019), 8067–8093.
  * [17] M. Cirant and A. Porretta. _Long time behavior and turnpike solutions in mildly non-monotone mean field games_. ESAIM Control Optim. Calc. Var., 27 (2021), 86.
  * [18] A. Fathi, Weak KAM Theorem in Lagrangian Dynamics, preliminary version 10, Lyon, unpublished, 2008.
  * [19] D. Gomes and M. Sedjro. _One-dimensional, forward-forward mean-field games with congestion_. Discrete Contin. Dyn. Syst., 11 (2017), 901.
  * [20] M. Huang, R. P. Malhamé and P. E. Caines. _Large population stochastic dynamic games: closed-loop McKean-Vlasov systems and the Nash certainty equivalence principle_. Commun. Inf. Syst. 6 (2006), 221–251.
  * [21] M. Huang, P. E. Caines and R. P. Malhamé. _Large-population cost-coupled LQG problems with nonuniform agents: Individual-mass behavior and decentralized $\epsilon$-Nash equilibria_. IEEE Trans. Automat. Control. 52 (2007), 1560–1571.
  * [22] J.-M. Lasry and P.-L. Lions, _Jeux à champ moyen. I. Le cas stationnaire_ (French). C. R. Math. Acad. Sci. Paris, 343 (2006), 619–625.
  * [23] J.-M. Lasry and P.-L. Lions, _Jeux à champ moyen. II. Horizon fini et controle optimal_ (French). C. R. Math. Acad. Sci. Paris, 343 (2006), 679–684.
  * [24] J.-M. Lasry and P.-L. Lions, _Mean field games_. Jpn. J. Math. 2 (2007), 229–260.
  * [25] P.-L. Lions, G. Papanicolaou and S. Varadhan. _Homogenization of Hamilton-Jacobi equation_. unpublished preprint, 1987.
  * [26] J. Mather, _Action minimizing invariant measures for positive definite Lagrangian systems_. Math. Z., 207 (1991), 169–207.
  * [27] R. Mañé, _On the minimizing measures of Lagrangian dynamical systems_. Nonlinearity, 5 (1992), 623–38.
  * [28] R. Mañé, _Generic properties and problems of minimizing measures of Lagrangian systems_. Nonlinearity, 9 (1996), 273–310.
  * [29] M. Masoero, _On the long time convergence of potential MFG_. Nonlinear Differ. Equ. Appl., (2019) 26:15.
  * [30] M. Zavidovique, Discrete and continuous weak KAM theory: an introduction through examples and its applications to twist maps, arXiv: 2308.06356.