# Recent Advances in Modeling and Control of Epidemics using a Mean Field
Approach

Amal Roy, Chandramani Singh, and Y. Narahari
Indian Institute of Science
Bangalore, India

###### Abstract

Modeling and control of epidemics such as the novel Corona virus have assumed
paramount importance at a global level. A natural and powerful dynamical
modeling framework to use in this context is a continuous time Markov decision
process (CTMDP) that encompasses classical compartmental paradigms such as the
Susceptible-Infected-Recovered (SIR) model. However, using a CTMDP based model
suffers from the curse of dimensionality and moreover requires global state
information. The challenges with CTMDP based models motivate the need for a
more efficient approach and the mean field approach offers an effective
alternative. The mean field approach computes the collective behavior of a
dynamical system comprising numerous interacting nodes (where nodes represent
individuals in the population). This paper (a) presents an overview of the
mean field approach to epidemic modeling and control and (b) provides a state-
of-the-art update on recent advances on this topic.

When an epidemic strikes, it is important to contain and suppress the epidemic
spread to minimize the loss of lives and the suffering as well as lower the
burden on the public health care system. Non-pharmaceutical interventions are
simple yet effective measures to limit the spread of an ongoing epidemic.
However, it is observed that in spite of the threat posed by an epidemic,
individuals tend to go by their freewill rather than adhering to universally
accepted best practices. This could dangerously push the population level
behavior towards alarming consequences. Motivated by this, our discussion in
this paper proceeds along two specific threads. The first thread assumes that
the individual nodes faithfully follow a socially optimal control policy
prescribed by a regulatory authority. The second thread allows the individual
nodes to exhibit independent, strategic behavior. In this case, the strategic
interaction is modeled as a mean field game and the control is based on the
associated mean field Nash equilibria. In this paper, we start with a
discussion of modeling of epidemics using an extended compartmental model -
SIVR (Suceptible-Infected-Vaccinated-Recovered) and provide an illustrative
example. We next provide a review of relevant literature, using a mean field
approach, on optimal control of epidemics, dealing with how a regulatory
authority may optimally contain epidemic spread in a population. Following
this, we provide an update on the literature on the use of the mean field game
based approach in the study of epidemic spread and control. We conclude the
paper with relevant future research directions.

##  1 Introduction

Throughout history, epidemics have had major effects, often catastrophic, on
the lives and lifestyle of the global population. The Bubonic Plague also
called Black Death ravaged Asia and Europe in several waves during the
fourteenth century, and is estimated to have caused the death of as much as
one third of the population of Europe between 1346 and 1350. Spanish Flu, also
known as the Great Influenza epidemic, was an exceptionally deadly global
influenza pandemic which started in 1918. Two years later, nearly a third of
the global population, or an estimated 500 million people, had been infected
in four successive waves. Estimates of deaths range from 17 million to 50
million, and possibly as high as 100 million, making it one of the deadliest
pandemics in human history.

More recently, the entire world has been devastated by multiple waves of a
novel Corona virus and its variants since December 2019. Unsurprisingly, this
has led to research and innovation efforts of an unprecedented scale towards
prediction, mitigation, and management of the pandemic. Epidemiology has now
moved to the centre-stage of research and policy making in public health.

Motivation. When an epidemic or pandemic such as Corona strikes, it is
important to contain and suppress the spread of the disease to minimize the
loss of life and the suffering and also to lower the burden on the health care
system. Non-pharmaceutical interventions such as reduction of social
interactions, masking, social distancing, hand washing, hand-sanitizing, and
disinfecting surfaces are simple yet very effective measures to limit the
spread of an ongoing epidemic. However, it is observed that in spite of the
threat posed by the pandemic individuals tend to go by their freewill rather
than adhering to best practices such as stated above. There are numerous
instances, throughout the world, where individuals have refused to comply with
Covid appropriate behaviour [7]. This could dangerously push the dynamics of
the entire population towards alarming or even catastrophic consequences. This
lack of responsible behavior has forced regulatory authorities to seek
suitable measures and incentives to improve compliance to best practices. In
order for these measures to be computed scientifically, we need an appropriate
modeling and control framework.

Mean Field Approach. Set in the above backdrop, modeling and control of
epidemics have assumed paramount importance. A natural and powerful model that
has emerged in this context is a continuous time Markov decision process
(CTMDP) which encompasses the classical compartmental paradigms such as the
SIR (Susceptible-Infected-Recovered) model. The use of a CTMDP based model,
however, poses certain technical and computational challenges. First, the
state space of any CTMDP based model grows exponentially with the population
size and the computation of the optimal policy using dynamic programming
principles become intractable even for moderate population sizes [52]. Second,
the controller needs to know the global population state to execute the
policy. These two challenges motivate the need for a more efficient approach
and the mean field approach has emerged as an effective alternative [23]. The
mean field approach computes the collective behavior of a dynamical system
comprising numerous interacting nodes (individuals in the population). Our
objective in this paper is to provide a bird’s eye view of and a state-of-the-
art update on some recent advances in the use of the mean field approach for
epidemic modeling and control.

Structure of the Paper. Our discussion in this paper proceeds along two
threads. The first thread assumes that the individual nodes faithfully follow
a socially optimal control policy prescribed by a regulatory authority. Here
we follow a mean field approach to derive optimal control. The second thread
allows the nodes to exhibit individualistic, strategic behavior. In this case,
a mean field game is formulated and the individuals’ controls are governed by
the mean field Nash equilibrium. This paper is structured as follows.

  * •

Section 2: This section is devoted to a description of the model of spread of
epidemics that we will be using in the rest of the paper. First, we present an
extension of the classical SIR model (Susceptible-Infected-Recovered model)
taking into account vaccinations. We call this the SIVR (Susceptible-Infected-
Vaccinated-Recovered) model. We describe the costs incurred by each node:
lockdown cost, infection cost, and vaccination cost. We next describe the
evolution of the population, which turns out to be a time-inhomogeneous
continuous time Markov chain. We formulate the optimal control problem which
seeks to minimize the expected average cost for an individual over a finite
time horizon. We next present a stochastic game model when the individual
nodes are strategic.

  * •

Section 3: This discusses mean field modeling of epidemics. The spread of
epidemics can be modeled as the mean field limit of a sequence of dynamical
processes. We consider a general dynamical system and recall key results in
mean field analysis. We present the standard solution to the mean field
control problem. We then consider the strategic case and present the mean
field game model and a solution for the same.

  * •

Section 4: Here, we present an illustrative example. To keep things simple, we
consider an SIR model rather than an SIVR model. For this example, we
illustrate mean field control for the non-strategic case followed by a mean
field game model and control for the strategic case.

  * •

Section 5: This provides a state-of-the-art update of mean field optimal
control of epidemics when a centralized regulatory authority prescribes a
control policy and the individual nodes faithfully follow the policy. There is
rich literature on applying optimal control methods to compartmental models of
epidemics. We categorize the literature into (1) Non-pharmaceutical
interventions (2) Vaccination strategies.

  * •

Section 6: This section is devoted to a state-of-the-art update of literature
on the use of the mean field game approach in the study of epidemic spread and
control. We have categorized the relevant literature into the following
groups: (1) Non-pharmaceutical interventions (2) Vaccination strategies (3)
Control and policy design.

  * •

Section 7: We conclude the paper by providing several directions for future
research in this section.

##  2 Modeling of Epidemics

Modeling of spread of epidemics is a vast topic and there are numerous papers
which present a variety of models. We wish to point to the paper by Guan, Wei,
Zhao, and Chen [25] which provides a review of literature on modeling
techniques and dynamic models in the context of the COVID-19 pandemic. The
authors there discuss how three aspects, (a) epidemiological parameter
estimation, (b) trend prediction, and (c) control measure evaluation, which
are addressed in the literature. They conclude that dynamic models which are
extensions of basic SIR (susceptible-Infected-Recovered) and SEIR
(Susceptible-Exposed-Infected-Recovered) models provide useful insights into
these three aspects. Their conclusion is that dynamic models are useful for
exploring possibilities of interventions but can fail at making strong
predictions about long-term disease dynamics.

In this section, we present the modeling framework that we will be using in
the rest of the paper. We first describe an extension of the classical SIR
(Susceptible-Infected-Recovered model) compartmental model taking into account
vaccinations. We call this the SIVR (Susceptible-Infected-Vaccinated-
Recovered) model. We find this compartmental model congenial for illustrating
the computation of optimal control and mean field equilibrium. We describe the
costs incurred by each node: lockdown cost, infection cost, and vaccination
cost. We next describe the evolution of the population, which turns out to be
a time-inhomogeneous continuous time Markov chain. We formulate the optimal
control problem which seeks to minimize the expected average cost for an
individual over a finite time horizon. We then present a mean field game model
where the individual nodes are strategic. Important symbols used in this paper
are listed for ready reference in Table 1.

$N$ | Number of nodes (individuals or agents) in the population
---|---
$\mathcal{S}$ | State space of a Markov chain
$K$ | Cardinality of a discrete state space $\mathcal{S}$
$T$ | Time horizon
$M_{i}$ | Number of nodes in state $i$
$Y_{i}$ | State of agent $i$. $Y_{i}\in\mathcal{S}$
$X_{i}$ |  $X_{i}:=M_{i}/N$, fraction of nodes in state $i$
$p_{i}$ | Probability of an agent being in state $i$
$\Delta^{N}_{K}$ | $\\{\frac{x}{N}|x\in\mathbb{Z}^{K}_{+},\sum_{i=1}^{K}x_{i}=N\\}$
$\Delta_{K}$ | $\\{x\in\mathbb{R}^{K}_{+}|\sum_{i=1}^{K}x_{i}=1\\}$
$X^{N}$ | State of population of $N$ agents $:=(X_{1},\ldots,X_{k})\in\Delta^{N}_{K}$
$x(t)$ | State of the population in mean field $:=(x_{1},\ldots,x_{k})\in\Delta_{K}$
$Q$ |  $\in\mathbb{R}^{K\times K}$ Transition rate matrix. $Q_{ij}$ is the transition rate from state $i$ to $j$
$c_{L}(\cdot)$ | Lockdown cost per unit time per individual
$c_{I}(\cdot)$ | Infection cost per unit time per individual
$c_{V}$ | Vaccination cost per individual
$u(\cdot)$ | Control variable as a function of time. $u(t)\in U\subset\mathbb{R}^{m}$
$g(\cdot,\cdot)$ | Running cost function for optimal control problem. $g:\mathbb{R}^{K}\times U\rightarrow\mathbb{R}$
$h(\cdot)$ | Terminal cost function for optimal control problem. $h:\mathbb{R}^{K}\rightarrow\mathbb{R}$
$\pi_{t}$ | Markov decision rule $\pi_{t}:\Delta^{N}_{K}\rightarrow U$
$\pi$ |  $\pi:=(\pi_{t},t\in[0,T])$ is the policy
$J_{\pi}(x)$ | Cost for the population, given the initial state $x$ and policy $\pi$
$J_{\bar{\pi}}(i,x,\pi)$ | Cost for an agent starting in state $i$, using policy $\bar{\pi}$, with the population
| starting in state $x$ and using policy $\pi$
Table 1: List of symbols and notation.

###  2.1 Modeling an Individual Node: SIVR Model

Figure 1: State transition diagram for an individual node. Values in edges are
the transition rates between states.

Let us consider a population of $N$ nodes. Each node can be in one the four
states, susceptible ($S$), infected ($I$), vaccinated ($V$) or recovered
($R$). The state transition diagram for an individual node is shown in Figure
1 (a more detailed description of the figure appears subsequently). The nodes’
states evolve with time due to their interactions with other nodes, external
interventions or because of their own actions. For instance, a susceptible
node becomes infected on meeting an infected node or can choose to become
vaccinated. On the other hand, an infected node may recover over time either
on its own or through medical treatment. We assume that recovered nodes will
not be infected in future.

Figure 2: State transition diagram of the continuous time Markov chain
$M(t),t\geq 0$ at $M(t)=(M_{S},M_{I},M_{V},M_{R})$.

We assume that the number of unrestricted meetings between any pair of nodes
constitute a Poisson process of rate $\kappa$ and that these processes are
independent across the node pairs. Moreover, the number of meetings between
any pair of susceptible-infected nodes constitutes a Poisson process of rate
$\kappa\alpha_{S}(t)\alpha_{I}(t)$ where
$\alpha(t)\coloneqq(\alpha_{S}(t),\alpha_{I}(t))\in[\alpha_{\min},1]\times[\alpha_{\min},1]\eqqcolon
A$ quantify the efforts of susceptible and infected individuals to contact
other individuals. We also assume that the susceptible nodes’ vaccination
processes are independent Poisson processes with rate $\beta(t)\in B$ and
infected nodes’ recovery times are exponentially distributed with parameter
$\gamma(t)\in C$. We refer to $u(t)\coloneqq(\alpha(t),\beta(t),\gamma(t))\in
U\coloneqq A\times B\times C$ as the action at time $t$, and the function
$u:[0,T]\to U$ as the action function.

###  2.2 Evolution of the Population

Let $Y_{n}(t)$ denote the state of the $n^{th}$ node at time $t$ and $Y(t)$
denote the state of the entire population;
$Y(t)\coloneqq(Y_{1}(t),Y_{2}(t),\cdots,Y_{N}(t))$. Let
$M(t)\coloneqq(M_{S}(t),M_{I}(t),M_{V}(t),M_{R}(t))$ denote the numbers of
nodes in states $S,I,V$ and $R$ at time $t$, respectively;

$M_{i}(t)=\sum_{n=1}^{N}\mathbbm{1}_{\\{Y_{n}(t)=i\\}}\ \text{for}\
i\in\\{S,I,V,R\\}.$

From the above discussion, $Y(t),t\geq 0$ is a time inhomogeneous continuous
time Markov chain (CTMC). For the $n^{th}$ node, its state evolution at any
time $t$ depends on the joint population state $Y(t)$ only through $Y_{n}(t)$
and $M(t)$. Figure 1 shows the associated state transition rate diagram.
Moreover, $M(t),t\geq 0$ is also a CTMC with transition rate diagram as in
Figure 2. Observe that $M_{S}(t)+M_{I}(t)+M_{V}(t)+M_{R}(t)=N$ for all $t$,
and hence, it suffices to specify three numbers, $M_{S}(t),M_{I}(t)$ and
$M_{V}(t)$.

Further, let $X(t)\coloneqq(X_{S}(t),X_{I}(t),X_{V}(t),X_{R}(t))$ denote the
fractions of nodes in states $S,I,V$ and $R$ at time $t$;

$X(t)=\left(\frac{M_{S}(t)}{N},\frac{M_{I}(t)}{N},\frac{M_{V}(t)}{N},\frac{M_{R}(t)}{N}\right)\in\Delta^{N}_{4}$

where
$\Delta^{N}_{K}\coloneqq\\{\frac{m}{N}:m\in\mathbb{Z}^{K}_{+},\sum_{i=1}^{K}m_{i}=N\\}$.
Clearly, $X(t),t\geq 0$ is also a CTMC. Both $M(t),t\geq 0$ and $X(t),t\geq 0$
offer equivalent characterisations of the same dynamical system. From the
state transition diagram in Figure 2, we can also write the expected
conditional drift rates of $X(t)$. These are as follows:

$\displaystyle\frac{\operatorname*{{\rm
d}\\!}\mathbb{E}[X_{S}(t)|X(t)]}{\operatorname*{{\rm d}\\!}t}$
$\displaystyle=\kappa\alpha_{S}(t)\alpha_{I}(t)(NX_{S}(t))(NX_{I}(t))\left(\frac{-1}{N}\right)$
$\displaystyle\quad+\beta(t)NX_{S}(t)\left(\frac{-1}{N}\right)$
$\displaystyle=-N\kappa\alpha_{S}(t)\alpha_{I}(t)X_{S}(t)X_{I}(t)-\beta(t)X_{S}(t).$
(1a) $\displaystyle\frac{\operatorname*{{\rm
d}\\!}\mathbb{E}[X_{I}(t)|X(t)]}{\operatorname*{{\rm d}\\!}t}$
$\displaystyle=N\kappa\alpha_{S}(t)\alpha_{I}(t)X_{S}(t)X_{I}(t)-\gamma(t)X_{I}(t).$
(1b) $\displaystyle\frac{\operatorname*{{\rm
d}\\!}\mathbb{E}[X_{V}(t)|X(t)]}{\operatorname*{{\rm d}\\!}t}$
$\displaystyle=\beta(t)X_{S}(t).$ (1c)

We prefer to work with $X(t),t\geq 0$ for reasons that will be clear in the
next section.

####  2.2.1 Costs Incurred by the Nodes

We characterise the costs incurred by the nodes during the course of an
epidemic. An individual can incur the following costs, according to the action
at time $t$, $u(t)=(\alpha(t),\beta(t),\gamma(t))\in A\times B\times C$.

  1. 1.

Lockdown cost: This quantifies financial costs or discomfort caused to a
susceptible or infected individual due to social distancing and lockdowns.
These are reflected in susceptible and infected nodes’ controls of meeting
rates $\alpha_{S}(t)$ and $\alpha_{I}(t)$, respectively. We define a function
$c_{L}:[\alpha_{\min},1]\rightarrow\mathbb{R}$ where $c_{L}(\alpha_{S}(t))$
and $c_{L}(\alpha_{I}(t))$ represent the lockdown costs per unit time of
susceptible and infected nodes, respectively.

  2. 2.

Infection cost: This captures medical expenses, financial losses, etc.,
incurred by an infected individual. In general, the infection cost per unit
time and the average recovery time $1/\gamma(t)$ are correlated. For instance,
better health care incurs more cost per unit time but facilitates quicker
recovery. To capture this correlation, we define function
$c_{I}:C\rightarrow\mathbb{R}$ and $c_{I}(\gamma(t))$ denotes the infection
cost per unit time.

  3. 3.

Vaccination cost: This represents the per individual cost of vaccination and
treatment of side effects of vaccination, if any. We assume the vaccination
cost to be fixed. Let $c_{V}$ represent the vaccination cost per individual.

Now we compute the average total cost, averaged over all the individuals. We
express this cost as a function of initial population distribution $X(0)=x$
and actions $u(t)\coloneqq(\alpha(t),\beta(t),\gamma(t)),t\geq 0$.

$\displaystyle\frac{1}{N}\sum_{n=1}^{N}\operatorname*{{\mathbb{E}}}\left[\right.\int_{0}^{T}\left(c_{L}(\alpha_{S}(t))\mathbbm{1}_{Y_{I}(t)S}+(c_{L}(\alpha_{I}(t))+c_{I}(\gamma(t)))\mathbbm{1}_{Y_{I}(t)=I}\right)\operatorname*{{\rm
d}\\!}t$ $\displaystyle\qquad\qquad+c_{V}\mathbbm{1}_{Y_{I}(T)=V}\ |\
X(0)=x\left.\right]$
$\displaystyle=\operatorname*{{\mathbb{E}}}\left[\right.\int_{0}^{T}\left(c_{L}(\alpha_{S}(t))X_{S}(t)+(c_{L}(\alpha_{I}(t))+c_{I}(\gamma(t)))X_{I}(t)\right)\operatorname*{{\rm
d}\\!}t$ $\displaystyle\qquad\qquad+c_{V}X_{V}(T)\ |\ X(0)=x\left.\right]$ (2)

####  2.2.2 The Optimal Control Problem

Observe that population evolution as well as the expected average cost over
$[0,T]$ depend on actions $u(t),t\in[0,T]$. The optimal control problem seeks
to minimize this cost via appropriate choice of actions.

##### Decision rule:

A decision rule at time $t$, $\pi_{t}$, is function that takes the states
$X(s),s\in[0,T]$ as input and output the action or decision $u(t)$. A history
dependent decision rule at time $t$ relies only on $X(s),s\in[0,t]$ for its
output. Further, the output of a Markov decision rule at time $t$ depends only
on the instantaneous state $X(t)$. More precisely, a Markov decision rule
$\pi_{t}$ is a mapping $\pi_{t}:\Delta^{N}_{4}\to U$.

##### Policy:

A policy is an ordered set of decision rules for all $t\in[0,T]$:

$\pi\coloneqq(\pi_{t},t\in[0,T])$ (3)

A policy is called history dependent (respectively, Markov) policy if it
consists of history dependent (respectively, Markov) decision rules.

We have at our disposal a finite horizon continuous time Markov decision
process (CTMDP) with finite state space and compact action spaces.
Furthermore, the cost rates are bounded. Hence, following [23], there exists a
Markov policy that is optimal within the class of all deterministic history-
dependent policies. Further, observe that a Markov policy $\pi$ together with
the initial state $X(0)$ characterize the whole trajectory $X(t),t\in[0,T]$.
So, the cost in (2) also is a function of $\pi$ and $X(0)=x$, we use
$J_{\pi}(x)$ to refer to this cost,

$\displaystyle J_{\pi}(x)$
$\displaystyle\coloneqq\operatorname*{{\mathbb{E}}}\left[\right.\int_{0}^{T}\left(\right.c_{L}(\alpha_{S}(t))X_{S}(t)+(c_{L}(\alpha_{I}(t))+c_{I}(\gamma(t)))X_{I}(t)\left.\right)\operatorname*{{\rm
d}\\!}t$ $\displaystyle\qquad+c_{V}X_{V}(T)\ |\ X(0)=x\left.\right]$ (4)

where $(\alpha(t),\beta(t),\gamma(t))=\pi_{t}(X(t))$. The cost-to-go from a
starting state $x\in\Delta^{N}_{4}$ is given by (5a) and the optimal control
is given by (5b). The following pair of equations solves the Markov control
problem.

$\displaystyle J(x)$ $\displaystyle=\min_{\pi\in\Pi}J_{\pi}(x)$ (5a)
$\displaystyle\pi^{\ast}$
$\displaystyle\in\operatorname*{arg\,min}_{\pi\in\Pi}J_{\pi}(x).$ (5b)

Here $\Pi$ is the set of all Markov policies. However, this solution has two
drawbacks.

  1. 1.

The state space of the CTMDP grows exponentially with the population size.
Consequently, the algorithm to obtain the optimal policy has a prohibitive
complexity even for moderate population sizes.

  2. 2.

The controller needs to know the global population state to execute the
policy.

Both these problems are addressed with a mean-field approach. We work with the
mean-field limit of the CTMDP as described in Section 3.

###  2.3 Evolution of an Individual Node

We now focus on a particular node, say the $n$th node. Let
$p(t)\coloneqq(p_{S}(t),p_{I}(t),p_{V}(t),p_{R}(t))$ denote the conditional
probabilities of this node being in states $S$, $I$, $V$ and $R$,
respectively, at time $t$, given $Y_{n}(0)=i$ and $X(0)=x$;

$p_{i}(t)=\operatorname*{{\mathbb{E}}}\left[\left.\mathbbm{1}_{Y_{n}(t)=i}\right|Y_{n}(0)=i,X(0)=x\right]\
\text{for}\ i\in\\{S,I,V,R\\}.$

The rates of change of these probabilities are as follows.

$\displaystyle\frac{\operatorname*{{\rm d}\\!}p_{S}(t)}{\operatorname*{{\rm
d}\\!}t}$
$\displaystyle=-N\kappa\alpha_{S}(t)\alpha_{I}(t)p_{S}(t)\operatorname*{{\mathbb{E}}}(X_{I}(t))-\beta(t)p_{S}(t)$
(6a) $\displaystyle\frac{\operatorname*{{\rm
d}\\!}p_{I}(t)}{\operatorname*{{\rm d}\\!}t}$
$\displaystyle=N\kappa\alpha_{S}(t)\alpha_{I}(t)\operatorname*{{\mathbb{E}}}(X_{I}(t))-\gamma(t)p_{I}(t).$
(6b) $\displaystyle\frac{\operatorname*{{\rm
d}\\!}p_{V}(t)}{\operatorname*{{\rm d}\\!}t}$
$\displaystyle=\beta(t)p_{S}(t)+\gamma(t)p_{I}(t).$ (6c)

Suppose $Y_{n}(0)=i$ and $X(0)=x$. Further, suppose the $n$th node employs an
action function
$\bar{u}(t)=(\bar{\alpha}(t),\bar{\beta}(t),\bar{\gamma}(t)),t\geq 0$ whereas
all other nodes use $u(t)=(\alpha(t),\beta(t),\gamma(t)),t\geq 0$. Then the
expected total cost of the tagged node is given by the following expression.

$\displaystyle\mathbb{E}\left[\right.\int_{0}^{T}\left(c_{L}(\bar{\alpha}_{S}(t))\mathbbm{1}_{Y_{n}(t)=S}+(c_{L}(\bar{\alpha}_{I}(t))+c_{I}(\bar{\gamma}(t)))\mathbbm{1}_{Y_{n}(t)=I}\right)\operatorname*{{\rm
d}\\!}t$ $\displaystyle\qquad\qquad+c_{V}\mathbbm{1}_{Y_{n}(T)=V}\ |\
Y_{n}(0)=i,X(0)=x\left.\right]$
$\displaystyle=\int_{0}^{T}\left(c_{L}(\bar{\alpha}_{S}(t))p_{S}(t)+(c_{L}(\bar{\alpha}_{I}(t))+c_{I}(\bar{\gamma}(t))p^{N}_{I}(t)\right)\operatorname*{{\rm
d}\\!}t+c_{V}p_{V}(T).$

###  2.4 The Case of Strategic Agents: Stochastic Game Model

We now consider the individual nodes of the population to be strategic agents
who wish to minimize their respective costs. These autonomous nodes also
control their respective parameters giving rise to a stochastic game. We seek
a symmetric Nash equilibrium, i.e., an equilibrium in which all the nodes
employ the same policy.

We can formally describe a symmetric Nash equilibrium as follows. Suppose a
tagged node uses a policy $\bar{\pi}=(\bar{\pi}_{t},0\leq t\leq T)$ whereas
all other nodes use a policy $\pi=(\pi_{t},0\leq t\leq T)$. Then the occupancy
measure of the other $N-1$ nodes is characterised by (6). On the other hand,
the tagged node’s state evolves as (6) with $u(t),t\in[0,T]$ replaced by
$\bar{\pi}_{t}(X(t)),t\in[0,T]$, and its cost is

$\displaystyle\bar{J}_{\bar{\pi}}(i,x,\pi)$
$\displaystyle=\int_{0}^{T}\left(c_{L}(\bar{\alpha}_{S}(t))p_{S}(t)+(c_{L}(\bar{\alpha}_{I}(t))+c_{I}(\bar{\gamma}(t)))p^{N}_{I}(t)\right)\operatorname*{{\rm
d}\\!}t$ $\displaystyle\quad+c_{V}p_{S}(T)$

where $(\bar{\alpha}(t),\bar{\beta}(t),\bar{\gamma}(t))=\bar{\pi}_{t}(X(t))$.
A policy $\bar{\pi}$ is called a symmetric NE if

$\bar{\pi}\in\operatorname*{arg\,min}_{\pi^{\prime}\in\Pi}\bar{J}_{\pi^{\prime}}(i,x,\bar{\pi})$
(7)

for all $x\in\Delta^{N}_{4}$. Nash equilibrium computation and implementation
are marred with computational complexity issues, much as those faced by the
computation of an optimal policy. The computational complexity problem can be
addressed by appealing to the mean field limit as described in Section 3.

##  3 Mean Field Modeling of Epidemic Spread

This section discusses mean field modeling of epidemics. The spread of
epidemics can be modeled as the mean field limit of a sequence of dynamical
processes. To present the key results in mean field analysis, for the sake of
convenience, we move away from the SIVR model presented in the previous
section to a more general $K$-compartmental model. We present the standard
solution to the mean field control problem. We then consider the strategic
case and present the mean field game model and a solution for the same.

We start our discussion by considering a sequence of systems with increasing
population sizes. Under certain regularity conditions to be described below,
the corresponding sequences of processes $X^{N}(t),t\geq 0$ and
$p^{N}(t),t\geq 0$ converge to deterministic functions referred to as mean
field limits. Below we discuss this convergence in the context of a more
general dynamical system; spread of epidemics in a population will be a
special case.

###  3.1 Mean Field Limit in a General Dynamical System

Let us consider a population of $N$ nodes, where each node can be in one of
the $K$ states, $\\{1,\ldots,K\\}=:\mathcal{S}$. Let
$Y^{N}(t)=(Y^{N}_{1}(t),\ldots,Y^{N}_{N}(t))$ denote the states of the nodes
at time $t$ as in Section 2.2.

####  3.1.1 Evolution of Population in the Mean Field Case

Let $\pi^{N}=(\pi^{N}_{i,t},i\in\mathcal{S},t\in[0,T])$ where
$\pi^{N}_{i,t}:\Delta^{N}_{K}\to U_{i}$ for all $t\in[0,T]$ and $U_{i}$ are
compact subsets of $\mathbb{R}_{+}^{L_{i}}$, be the policy used by each of the
nodes. Further, with the occupancy measure $X^{N}(t)\in\Delta^{N}_{K},t\geq 0$
as defined in Section 2.2, the expected conditional drift rates are, for
$j=1,\ldots,K$,

$\frac{\operatorname*{{\rm
d}\\!}\mathbb{E}[X^{N}_{j}(t)|X^{N}(t)=x]}{\operatorname*{{\rm
d}\\!}t}=\sum_{i=1,i\neq
j}^{K}x_{i}Q_{ij}(x_{j},\pi^{N}_{i,t}(x),\pi^{N}_{j,t}(x))\ $

Here, for any $i,j~{}(j\neq i)$, given $X^{N}(t)=x$,
$Q_{ij}(x_{j},\pi^{N}_{i,t}(x),\pi^{N}_{j,t}(x))$ is the state transition rate
of a node from state $i$ to state $j$, and

$Q_{ii}(x,\pi^{N}_{i,t}(x),j=1,\dots,K)\coloneqq-\sum_{j\neq
i}Q_{ij}(x_{j},\pi^{N}_{i,t}(x),\pi^{N}_{j,t}(x)).$

We can also write a general form of average cost of the population,
$J^{N}_{\pi^{N}}(x)$, as follows.

$\displaystyle
J^{N}_{\pi^{N}}(x)=\operatorname*{{\mathbb{E}}}\left[\right.\int_{0}^{T}X^{N}(t)^{T}g(X^{N}(t),\pi^{N}_{i,t}(X^{N}(t)),i\in\mathcal{S})\operatorname*{{\rm
d}\\!}t$ $\displaystyle+X^{N}(T)^{T}h(X^{N}(T))\ |\ X^{N}(0)=x\left.\right]$

where
$g(x,\pi^{N}_{i,t}(x),i\in\mathcal{S})\coloneqq(g_{i}(x,\pi^{N}_{i,t}(x)),i=1,\dots,K)$
are the cost rates and $h(x)\coloneqq(h_{1}(x),\cdots,h_{K}(x))$ are the
terminal costs. The optimal control problem and its solution are as in (5).

####  3.1.2 Evolution of an Individual Node in the Mean Field Case

Consider a tagged node, say the $n$th node, which, at time $t=0$ is in state
$i\in\mathcal{S}$ and the initial population state, $X^{N}(0)=x$. Further,
suppose that this node employs a policy
$\bar{\pi}^{N}=(\bar{\pi}^{N}_{i,t},i\in\mathcal{S},t\in[0,T])$ whereas all
other nodes use
$\bar{\pi}^{N}=(\bar{\pi}^{N}_{i,t},i\in\mathcal{S},t\in[0,T])$. Let
$p^{N}(t)\coloneqq(p^{N}_{1}(t),\cdots,p^{N}_{K}(t))$ denote the conditional
probabilities of the $n$th node being in states $1,\cdots,K$, respectively, at
time $t$. The rate of change of these probabilities can be expressed as
follows.

$\displaystyle\frac{\operatorname*{{\rm
d}\\!}p^{N}_{j}(t)}{\operatorname*{{\rm d}\\!}t}=\sum_{i=1,i\neq
j}^{K}p^{N}_{i}(t)\operatorname*{{\mathbb{E}}}[Q_{ij}(X^{N}_{j}(t),\bar{\pi}^{N}_{i,t}(X^{N}(t)),\bar{\pi}^{N}_{j,t}(X^{N}(t)))]$
$\displaystyle\text{ for }j=1,\cdots,K.$

Further, the $n$th node’s cost, $\bar{J}_{\bar{\pi}^{N}}(i,x,\pi^{N})$, can be
expressed as follows.

$\displaystyle\bar{J}^{N}_{\bar{\pi}^{N}}(i,x,\pi^{N})=\int_{0}^{T}p^{N}(t)^{T}g(X^{N}(t),\pi^{N}_{j,t}(X^{N}(t)),j\in\mathcal{S})\operatorname*{{\rm
d}\\!}t$ $\displaystyle+p^{N}(T)^{T}h(X^{N}(t)).$

Finally, the Nash equilibrium can be characterised as in (7).

##### Example (Epidemics):

In the special case of evolution of epidemics as in Section 2,
$K=4,L_{1}=L_{2}=3,L_{3}=L_{4}=0$ and for $u_{1}=(\alpha_{S},\beta)$ and
$u_{2}=(\alpha_{I},\gamma)$,

$Q(x,u_{1},u_{2})=\begin{bmatrix}-N\kappa\alpha_{S}\alpha_{I}x_{I}-\beta&N\kappa\alpha_{S}\alpha_{I}x_{I}&\beta&0\\\
0&-\gamma&0&\gamma\\\ 0&0&0&0\\\ 0&0&0&0.\end{bmatrix}.$

Moreover,

$\displaystyle g(x,u_{1},u_{2})$ $\displaystyle=[c_{L}(\alpha_{S})\ \
c_{L}(\alpha_{I})+c_{I}(\gamma)\ \ 0\ \ 0]^{T}$ $\displaystyle h(x)$
$\displaystyle=[0\ \ 0\ \ 0\ \ c_{V}]^{T}.$

###  3.2 Mean Field Control

Darling [18] analyse the convergence of CTMCs to solutions of certain ODEs.
Gast et al. [23] study convergence of controlled DTMCs to continuous times
controlled deterministic dynamical systems. Following these works, we make the
following hypotheses on the dynamics and the costs.

####  3.2.1 Mean Field Convergence

##### Assumptions on dynamics

  1. 1.

Initial Conditions: The initial occupancy measure $x^{N}(0)$ converges to
$x(0)\in\Delta_{K}$ in probability, i.e.,
$\lim_{N\to\infty}P(\|X^{N}(0)-x(0)\|>\epsilon)=0$ for all $\epsilon>0$.

  2. 2.

Transition rates: For all $i,j~{}(j\neq i)$, the state transition rates
$Q_{ij}(x_{j},u_{i},u_{j})$ are $O(1)$. This implies that the expected number
of transitions per unit time is $O(N)$.

  3. 3.

Drift rates: The drifts are $O(N)$. Moreover, there exist bounded transition
rate matrices $Q(\cdot,u)\in\mathbb{R}^{K\times K}$ $\forall u\in
U\coloneqq\bigotimes_{i=1}^{K}U_{i}$, such that $x^{T}Q(x,u)$ converges to
$x^{T}Q(x,u)$ uniformly in $(x,u)$.

  4. 4.

Lipschitz continuity: $x^{T}Q(x,u)$ is Lipschitz continuous in $(x,u)$.

##### Assumptions on the costs

  1. 1.

Cost rates: The cost rates $g_{i}(\cdot,u_{i})$ and the terminal costs
$h_{i}(\cdot)$ are bounded.

  2. 2.

Lipschitz continuity: The cost rates $g_{i}(\cdot,u)$ are Lipschitz continuous
for all $u_{i}\in U_{i}$, and the terminal costs $h_{i}(\cdot)$ are also
Lipschitz continuous.

Observe that the scaling conditions are satisfied in the case of epidemics if
we choose
$\kappa^{N}=\kappa/N,\alpha_{S}^{N}=\alpha_{S},$$\alpha_{I}^{N}=\alpha_{I},\beta^{N}=\beta$
and $\gamma^{N}=\gamma$. We further assume that $c_{L}(\cdot)$ and
$c_{I}(\cdot)$ are bounded functions. It can be easily checked that the above
hypotheses are satisfied under these assumptions.

In order to state the mean field convergence results, we introduce the
following optimal control problem. Consider a continuous time dynamical system

$\frac{\operatorname*{{\rm d}\\!}x(t)}{\operatorname*{{\rm
d}\\!}t}=Q^{T}(x(t),u(t))x(t)$ (8)

with state $x(t),t\geq 0$, initial condition $x(0)$, and action function
$u(t)=(u_{i}(t),i=1,\cdots,K),t\geq 0$. Let
$J_{u}:\Delta_{K}\to\mathbb{R}_{+}$ be the associated cost function defined as
follows.

$J_{u}(x)=\int_{0}^{T}x(t)^{T}g(x(t),u(t))\operatorname*{{\rm
d}\\!}t+x(T)^{T}h(x(T))$

where $x(t),t\geq 0$ is a solution to (8) given $x(0)=x$. We seek to find an
action function $u^{\ast}(t),0\leq t\leq T$ which together with its
corresponding state trajectory $x^{\ast}(t),0\leq t\leq T$ attains the minimum
cost

$J(x)\coloneqq\min_{u}J_{u}(x).$

We then have the following convergence results.

  1. 1.

Optimal trajectory: The CTMC $X^{N}(t),0\leq t\leq T$ converges to the mean
field limit $x(t),0\leq t\leq T$. More precisely, for all $\epsilon>0$,

$\lim_{N\to\infty}\mathbb{P}\left[\sup_{0\leq t\leq
T}\|X^{N}(t)-x(t)\|>\epsilon\right]=0$

  2. 2.

Optimal cost: The optimal cost for the population of size $N$,
$J^{N}(X^{N}(0))$, converges to the optimal cost of the mean field control
problem, $J(x(0))$, in probability.

  3. 3.

Optimal policy: An optimal action function for the mean field limit, say
$u^{\ast}$, is asymptotically optimal for the system with finitely many nodes.
More precisely, $J^{N}_{\pi^{\ast}}(X^{N}(0))-J(x(0))$ converges to $0$ in
probability where $\pi^{\ast}_{i,t}\equiv u^{\ast}_{i}(t)$ for all $i$.

####  3.2.2 Solution to the Mean Field Control Problem

With a slight abuse of notation, we let $J_{t}(x)$ denote the optimal cost
over $[t,T]$ given $x(t)=x$. We can write

$\displaystyle J_{t}(x)$ $\displaystyle=\min_{u\in
U}\left\\{x^{T}g(x,u)\operatorname*{{\rm d}\\!}t+J_{t+\operatorname*{{\rm
d}\\!}t}(x(t+\operatorname*{{\rm d}\\!}t))\right\\}$ $\displaystyle=\min_{u\in
U}\left\\{x^{T}g(x,u)\operatorname*{{\rm d}\\!}t+J_{t+\operatorname*{{\rm
d}\\!}t}(x)+x^{T}Q(x,u)\nabla_{x}J_{t}(x)\operatorname*{{\rm d}\\!}t\right\\}$

On rearranging the terms, we obtain

$-\nabla_{t}J_{t}(x)=\min_{u\in
U}\left\\{x^{T}g(x,u)+x^{T}Q(x,u)\nabla_{x}J_{t}(x)\right\\}.$

The above equation, referred to as the Hamilton-Jacobi-Bellman (HJB) equation,
yields both the optimal action function $u^{\ast}:[0,T]\to U$ and the optimal
cost $J(x)\equiv J_{0}(x)$ of the mean field control problem.

##### The Minimum Principle [11] :

Let us introduce adjoint (also called costate) processes
$\lambda(t)\in\mathbb{R}^{K},t\in[0,T]$ and the Hamiltonian function
$H:\Delta^{N}_{K}\times U\times\mathbb{R}^{K}\to\mathbb{R}$ given by

$H(x,u,\lambda)=x^{T}g(x,u)+x^{T}Q(x,u)\lambda.$

Let $u^{\ast}(t),t\in[0,T]$ be an optimal action function and
$x^{\ast}(t),t\in[0,T]$ be the corresponding state trajectory, i.e.,
$x^{\ast}(t),t\in[0,T]$ is the solution to (8) with
$u(t)=u^{\ast}(t),t\in[0,T]$ and initial condition $x^{\ast}(0)=x$. The HJB
equation can be compactly written in terms of the Hamiltonian function as

$-\nabla_{t}J_{t}(x^{\ast}(t))=H(x^{\ast}(t),u^{\ast}(t),\nabla_{x}J_{t}(x^{\ast}(t))).$

The following result, referred to as the Minimum Principle, provides a
necessary condition for optimality of $u^{\ast}(t),t\in[0,T]$. Let
$\lambda(t),t\in[0,T]$ be the solution to equation

$\frac{\operatorname*{{\rm d}\\!}\lambda(t)}{\operatorname*{{\rm
d}\\!}t}=-\nabla_{x}H(x^{\ast}(t),u^{\ast}(t),\lambda(t)),$

referred to as the adjoint equation, with the boundary condition:

$\lambda(T)=\nabla(x^{\ast}(T)^{T}h(x^{\ast}(T))).$

Then, for all $t\in[0,T]$,

$u^{\ast}(t)=\operatorname*{arg\,min}_{u\in U}H(x^{\ast}(t),u,\lambda(t)).$

Furthermore, there is a constant $\theta$ such that

$H(x^{\ast}(t),u^{\ast}(t),\lambda(t))=\theta,\ \text{for all}\ t\in[0,T].$

The minimum principle can be used as the basis of a numerical solution. In the
so called two-point boundary problem method, we use the necessary condition

$u^{\ast}(t)=\operatorname*{arg\,min}_{u\in U}H(x^{\ast}(t),u,\lambda(t)).$

to express $u^{\ast}(t)$ in terms of $x^{\ast}(t)$ and $\lambda(t)$. We then
substitute the result into the system and the adjoint equations, to obtain a
set of $2K$ first order differential equations in $x^{\ast}(t)$ and
$\lambda(t)$. These equations can be solved using the split boundary
conditions:

$x^{\ast}(0)=x\ \text{and}\ \lambda(T)=\nabla(x^{\ast}(T)^{T}h(x^{\ast}(T))).$

####  3.2.3 Solution to the Finite Population Control Problem

We thus obtain the following procedure for solving the optimal control problem
for a finite population.

  * •

From the original system with $N$ nodes, write the mean field limit. In
particular, set $x(0)=X^{N}(0)$ and obtain $Q(\cdot,u)$ via appropriate
scaling of parameters.

  * •

Obtain the optimal control for the limiting problem via solving the HJB
equation or via some other method, e.g., using the minimum principle [11].

  * •

Use this control in the finite population problem. From the above discussion,
this is asymptotically optimal.

###  3.3 Mean Field Game Model of Epidemic Spread

Let us focus on a tagged node, say the $n$th node. Suppose it uses an action
function $\bar{u}:[0,T]\to U$ whereas all other nodes use an action function
$u:[0,T]\to U$. Following the discussion in Section 3.2.1, the occupancy
measures of these nodes, $X^{N}(t),t\in[0,T]$ asymptotically approach the mean
field limit $x(t),t\in[0,T]$. Hence, in the limiting system, the probabilities
of the tagged node being in various states, $p(t),t\in[0,T]$, given its
initial state, say $i$, evolve as follows.

$\frac{\operatorname*{{\rm d}\\!}p}{\operatorname*{{\rm
d}\\!}t}=Q^{T}(x(t),\bar{u}(t))p(t)$ (9)

with $p(0)=\delta_{i}$. Let $\bar{J}_{\bar{u}}(i,x,u)$ be the associated cost
defined as follows.

$\bar{J}_{\bar{u}}(i,x,u)=\int_{0}^{T}p(t)^{T}g(x(t),u(t))\operatorname*{{\rm
d}\\!}t+p(T)^{T}h(x(T))$

where $x(t),0\leq t\leq T$ and $p(t),0\leq t\leq T$ are solutions to (8) and
(9), respectively, given $x(0)=x$ and $p(0)=\delta_{i}$. We seek to find a
action function $\bar{u}:[0,T]\to U$, which, together with the corresponding
trajectory $\bar{p}(t),t\in[0,T]$, minimizes the tagged node’s cost when other
nodes are also using the same action function $\bar{u}:[0,T]\to U$, i.e.,

$\bar{u}\in\operatorname*{arg\,min}_{u^{\prime}}\bar{J}_{u^{\prime}}(i,x,\bar{u}).$

We expect the equilibrium trajectory, policy, and cost for the stochastic game
for a finite size population to be asymptotically close to those for the mean
field game. This is a conjecture and we do not yet have a proof of
convergence.

####  3.3.1 Solution to the Mean Field Game

With a slight abuse of notation we let $\bar{J}_{t}(i,x,u)$ denote the optimal
cost of the $n$th node over $[t,T]$ given $Y_{n}(t)=i,X(0)=x$. We can write

$\displaystyle\bar{J}_{t}(i,x,u)=\min_{u^{\prime}_{i}\in U_{i}}$
$\displaystyle\mathbb{E}\left[\right.g_{i}(x(t),u^{\prime}_{i})\operatorname*{{\rm
d}\\!}t+\bar{J}_{t+\operatorname*{{\rm d}\\!}t}(Y_{n}(t+\operatorname*{{\rm
d}\\!}t),x,u)$ $\displaystyle\left.\ |\ Y_{n}(t)=i,x(0)=x\right]$
$\displaystyle=\min_{u^{\prime}_{i}\in U_{i}}$
$\displaystyle\left\\{g_{i}(x(t),u^{\prime}_{i})\operatorname*{{\rm
d}\\!}t+\sum_{j\neq
i}Q_{ij}(x_{j}(t),u^{\prime}_{i},u_{j}(t))\operatorname*{{\rm
d}\\!}t\bar{J}_{t+\operatorname*{{\rm d}\\!}t}(j,x,u)\right.$
$\displaystyle\left.+\left(1-\sum_{j\neq
i}Q_{ij}(x_{j}(t),u^{\prime}_{i},u_{j}(t))\operatorname*{{\rm
d}\\!}t\right)\bar{J}_{t+\operatorname*{{\rm d}\\!}t}(i,x,u)\right\\}$

On rearranging the terms we obtain

$\displaystyle-\nabla_{t}\bar{J}_{t}(i,x,u)$
$\displaystyle=\min_{u^{\prime}_{i}\in
U_{i}}\left\\{\right.g_{i}(x(t),u^{\prime}_{i})$ $\displaystyle+\sum_{j\neq
i}Q_{ij}(x_{j}(t),u^{\prime}_{i},u_{j}(t))(\bar{J}_{t}(j,x,u)-\bar{J}_{t}(i,x,u))\left.\right\\}$
$\displaystyle=\min_{u^{\prime}_{i}\in
U}\left\\{g_{i}(x(t),u^{\prime}_{i})+\sum_{j}Q_{ij}(x_{j}(t),u^{\prime}_{i},u_{j}(t))\bar{J}_{t}(j,x,u)\right\\}.$
(10)

This is the HJB equation for the mean field game. A control $\bar{u}$ is a
Nash equilibrium of the game provided

$\bar{u}_{i}(t)\in\operatorname*{arg\,min}_{u^{\prime}_{i}\in
U_{i}}\left\\{g_{i}(x(t),u^{\prime}_{i})+\sum_{j}Q_{ij}(x_{j}(t),u^{\prime}_{i},\bar{u}_{j}(t)))\bar{J}_{t}(j,x,\bar{u})\right\\}$

for all $t\in[0,T]$. So, Nash equilibria are characterized by the system of
Kolmogorov and HJB equations, (8) and (10), respectively, together with the
boundary conditions:

$\displaystyle x(0)$ $\displaystyle=x$ $\displaystyle\text{and
}\bar{J}_{T}(i,x,\bar{u})$ $\displaystyle=h_{i}(x(T)).$

Thus we get a initial-terminal value problem (ITVP) whose fixed points yield
the solutions to the mean field game. Under regularity assumptions for the
cost functions $g_{i}(\cdot,\cdot)$ and the transition rates $Q(\cdot,\cdot)$,
it can be shown that a unique minimizer exists in (10), which is the Nash
equilibrium of the game [15, Section 7.2.2]. Following is a fixed point
iteration to obtain a Nash equilibrium given $x(0)$.

  * •

Initialize with an action function $u:[0,T]\to U$.

  * •

Solve Kolmogorov equations (8) to obtain trajectory $x(t),0\leq t\leq T$
corresponding to $u$.

  * •

For each $j\in\\{1,\cdots,K\\}$, solve HJB equations (10) to obtain the best
responses $u^{\prime}_{i}:[0,T]\to U_{i}$ to $u$ for all $i$.

  * •

If $u^{\prime}(t)=u(t)$ for all $t\in[0,T]$, set $\bar{u}=u^{\prime}$ and
exit. Otherwise, set $u=u^{\prime}$, and continue with the next iteration (go
to Step 2).

Let $f_{x}$ denote the mapping from $u$ to $\bar{u}$. When $f_{x}$ is a
_contractive_ mapping on a complete metric space, Picard-Banach fixed point
theorem states that there is a unique fixed point and that the convergence is
geometric [13].

The Master Equation: [15, Section 6.5] Let us introduce the adjoint processes
$\bar{\lambda}(t)\in\mathbb{R}^{K},t\in[0,T]$ and the Hamiltonian functions
$H_{i}:\Delta^{N}_{K}\times U\times\mathbb{R}^{K}\to\mathbb{R}$ given by

$H_{i}(x,u,\lambda)=g_{i}(x,u_{i})+\sum_{j}Q_{ij}(x_{j},u_{i},u_{j})\lambda_{j}.$

Let $\bar{u}_{i}(t),i\in\mathcal{S},t\in[0,T]$ be the best response action
functions of the tagged player. The HJB equations can be compactly written in
terms of the Hamiltonian functions as

$-\nabla_{t}\bar{J}_{t}(i,x,u)=\min_{u^{\prime}_{i}\in
U_{i}}H_{i}(x(t),(u^{\prime}_{i},u_{-i}(t)),\bar{J}_{t}(\cdot,x,u))\ \forall
i\in\mathcal{S},$

and

$\bar{u}_{i}(t)\in\operatorname*{arg\,min}_{u^{\prime}_{i}\in
U_{i}}H_{i}(x(t),(u^{\prime}_{i},\bar{u}_{-i}(t)),\bar{J}_{t}(\cdot,x,\bar{u}))\
\forall i\in\mathcal{S},$

where $\bar{J}_{t}(\cdot,x,u)\in\mathbb{R}^{K}$ has $j$th component
$\bar{J}_{t}(j,x,u)$ for all $j\in\mathcal{S}$. Finally, let ${\cal
J}_{t},t\in[0,T]$ be real valued functions, ${\cal
J}_{t}:[K]\times\mathbb{R}^{K}\to\mathbb{R}$, and let us also introduce the
following equation known as the master equation.

$\displaystyle\nabla_{t}{\cal J}_{t}(i,x)+H_{i}(x,(\nu_{i}(x,u_{-i}(t),{\cal
J}_{t}(\cdot,x)),u_{-i}(t)),{\cal J}_{t}(\cdot,x))$
$\displaystyle+\sum_{j}\left(\sum_{l}x_{l}Q_{lj}(x_{j},\nu_{i}(x,u_{-i}(t),{\cal
J}_{t}(\cdot,x)),u_{j}(t))\right)\nabla_{x_{j}}{\cal J}_{t}(i,x))=0$ (11)

where

$\nu_{i}(x,u_{-i},\lambda)\in\operatorname*{arg\,min}_{u^{\prime}_{i}\in
U_{i}}H_{i}(x,(u^{\prime}_{i},u_{-i}),\lambda)$

where

$q_{j}(x,u,z)=\sum_{l}x_{l}Q_{lj}(x_{j},u_{l}(z),u_{j}).$

Let ${\cal J}_{t},t\in[0,T]$ be a solution to the master equation with
terminal condition ${\cal J}_{T}(i,x)=h_{i}(x)$ for all $i\in\mathcal{S}$, and
$\bar{u}(t),t\in[0,t]$ be the associated optimal action function. Let
$\bar{x}(t),t\in[0,T]$ be a solution to

$\frac{\operatorname*{{\rm d}\\!}\bar{x}_{i}(t)}{\operatorname*{{\rm
d}\\!}t}=q_{t}(i,\bar{x}(t),\bar{u}(t),{\cal J}_{t}(\cdot,\bar{x}(t)))$ (12)

with initial condition $\bar{x}(0)$. Then, setting $\bar{J}_{t}(i,x,u)={\cal
J}_{t}(i,\bar{x}(t)))$,

$\displaystyle\nabla_{t}\bar{J}_{t}(i,x,u)+H_{i}(x(t),\bar{u}(t),\bar{J}_{t}(\cdot,x(t),u(t)))$
$\displaystyle=\nabla_{t}{\cal J}_{t}(i,x)+\sum_{j}q_{t}(j,x,\bar{u}(t),{\cal
J}_{t}(\cdot,x))\nabla_{x_{j}}{\cal J}_{t}(\cdot,x))$
$\displaystyle\quad+H_{i}(x,\bar{u}(t),{\cal J}_{t}(\cdot,x))$
$\displaystyle=0.$

Clearly, $\bar{J}_{t}(i,x,u)$ solves the HJB equation and is value function in
the optimization problem (10). Also, (12) can be identified with the
Kolmogorov equation (8). Consequently, $\bar{x}(t)$ is the equilibrium state
trajectory. It is therefore seen that the master equation encapsulates both
the Kolmogorov and the HJB equations in a single equation.

####  3.3.2 Solution to the Finite Population Stochastic Game

Finally, we can adopt a procedure similar to that in Section 2.2, to obtain a
solution to the stochastic game for a population of size $N$.

##  4 An Illustrative Example

We will now study how to formulate a mean field control as well as a mean
field game problem in a compartmental epidemic model. For ease of our
presentation, we simplify the SIVR model introduced in Section 2 to an SIR
model without vaccination. We assume that the transmission rate of the
disease, $\kappa$, is influenced by two factors: the disease characteristics
and the baseline contact factor within the population. The recovery rate
$\gamma$ is taken to be a constant. Individuals can adjust their level of
social interaction by choosing their _contact factor_. This is reflected by
choosing a value of
$\alpha(t)\coloneqq(\alpha_{S}(t),\alpha_{I}(t))\in[\alpha_{\min},1]\times[\alpha_{\min},1]\eqqcolon
A$. $\alpha_{min}$ represents the minimal transmission rate that an individual
who adopts maximum protection effort will encounter.

###  4.1 Illustrative Example: Mean Field Control

From a societal viewpoint, we assume that the regulating authority chooses a
policy $\pi=(\pi_{t},t\in[0,T])$ for the whole population. This can be thought
of as a combination of non-pharmaceutical interventions like lockdowns,
mandatory usage of masks, proactive testing and quarantine, etc. We look at a
finite horizon setup to identify the social optimum strategy that minimizes
the total social cost.

Once the social planner commits to a policy $\pi(t)$, analogous to equation
(8), the forward evolution of the state with the initial conditions
$x(0)=(x_{S}(0),x_{I}(0),x_{R}(0))$ is given by:

$\displaystyle\dot{x}_{S}(t)$
$\displaystyle=-\kappa\alpha_{I}(t)\alpha_{S}(t)x_{S}(t)x_{I}(t)$ (13a)
$\displaystyle\dot{x}_{I}(t)$
$\displaystyle=\kappa\alpha_{I}(t)\alpha_{S}(t)x_{S}(t)x_{I}(t)-\gamma
x_{I}(t)$ (13b) $\displaystyle\dot{x}_{R}(t)$ $\displaystyle=\gamma x_{I}(t)$
(13c)

We assume that individuals in states $i\in\\{S,I\\}$ choosing a contact factor
$\alpha_{i}(t)$ incurs a running cost
$c_{L}(\alpha_{i}(t))=c_{Q}(1-\alpha_{i}(t))$. In addition to this lockdown
cost, infected individuals pay a cost $c_{I}$ per unit time.

Starting from $x(0)=(x_{S}(0),x_{I}(0),x_{R}(0))$, policy $\pi$ induces a
social cost which can be expressed as:

$\displaystyle
J_{\pi}(x)=\int_{0}^{T}\left(c_{Q}(1-\alpha_{S}(t))x_{S}(t)+(c_{I}+c_{Q}(1-\alpha_{I}(t)))x_{I}(t)\right)\operatorname*{{\rm
d}\\!}t$

We can define the Hamiltonian corresponding to the optimal control problem as,

$\displaystyle H(x,\pi,\lambda)$
$\displaystyle=c_{Q}(1-\alpha_{S}(t))x_{S}(t)+(c_{I}+c_{Q}(1-\alpha_{I}(t)))x_{I}(t)$
$\displaystyle\ -\lambda_{2}(t)\gamma
x_{I}(t)+\kappa\alpha_{I}(t)\alpha_{S}(t)x_{S}(t)x_{I}(t)(\lambda_{2}(t)-\lambda_{1}(t))$

Defining $\lambda(t)=(\lambda_{1}(t),\lambda_{2}(t),\lambda_{3}(t))$ as the
co-state variable vector, with $\lambda_{3}(t):=0\quad\forall t\in[0,T]$,

$\displaystyle\dot{\lambda}_{1}$ $\displaystyle=-\dfrac{\partial H}{\partial
x_{S}}=-c_{Q}(1-\alpha_{S}(t))+\kappa\alpha_{I}(t)\alpha_{S}(t)x_{I}(t)(\lambda_{1}(t)-\lambda_{2}(t))$
$\displaystyle\dot{\lambda}_{2}$ $\displaystyle=-\dfrac{\partial H}{\partial
x_{I}}=-c_{I}-c_{Q}(1-\alpha_{I}(t))+\lambda_{2}(t)\gamma$
$\displaystyle\qquad\qquad+\kappa\alpha_{I}(t)\alpha_{S}(t)x_{S}(t)(\lambda_{1}(t)-\lambda_{2}(t))$
(14)

The final constraints for the co-state variables are:
$\lambda_{1}(T)=\lambda_{2}(T)=0$.

From our discussion in section 3.2.2, functions $\lambda_{1},\lambda_{2}$
exist that satisfy equation (4.1), and the optimal control $\pi^{*}$ is the
minimization of Hamiltonian, assuming that the co-state variables are set
according to optimum control.

####  4.1.1 Numerical Computation of Optimal Control

Note that in the previous section, we had to deal with two sets of ODEs. We
have the forward evolution ODEs of the compartmental model (13) along with
their initial conditions. In addition to which, we had the co-state ODE (4.1)
with terminal conditions. There are several classical procedures available to
compute the optimal control.

We use the forward-backward sweep method in [41, Chapter 4]. Figure 3 shows
time evolution of the epidemic in the population. Figure 4 shows the socially
optimal control for the set of parameters in table 2.

Figure 3: Time evolution of the state $(x_{S},x_{I},x_{R})$ under socially optimal control policy. Figure 4: Socially optimal control policy. $(S_{0},I_{0},R_{0})$ | $\gamma$ | $\kappa$ | $T$ | $c_{I}$ | $c_{Q}$ | $\alpha_{min}$
---|---|---|---|---|---|---
$(0.8,0.2,0)$ | $0.1$ | $0.2$ | $100$ | $100$ | $10$ | $0.1$
Table 2: Set of parameters for the numerical experiments

###  4.2 Illustrative Example: Mean Field Game Model

From the point of view of a strategic individual node, computing the best
response to a population strategy can be seen as a mean field game. Suppose a
tagged node uses a policy $\bar{u}=(\bar{\alpha}(t))_{t\in[0,T]}$ whereas all
other nodes use a policy $u$. The cost of an individual node is

$\displaystyle\bar{J}_{\bar{u}}(i,x,u)$
$\displaystyle=\int_{0}^{T}\\{c_{Q}(1-\bar{\alpha}_{S}(t))p_{S}(t)$
$\displaystyle\qquad+(c_{I}+c_{Q}(1-\bar{\alpha}_{I}(t)))p_{I}(t)\\}\operatorname*{{\rm
d}\\!}t.$

We seek to find a policy $\bar{u}:[0,T]\to U$ which, together with the
corresponding trajectory $\bar{p}(t),t\in[0,T]$ minimizes the tagged node’s
cost when other nodes are also using the same policy $\bar{u}$ (7).

Writing the corresponding Bellman equations (from section 3.3), with
$J_{i}(t)$ denoting the cost to go at time $t$ from state $i\in\\{S,I\\}$ we
have:

$\displaystyle-\dot{J}_{S}(t)$
$\displaystyle=\min_{\alpha_{S}(t)}c_{Q}(1-\alpha_{S}(t))+\kappa\alpha_{S}(t)\alpha_{I}(t)p_{I}(t)(J_{I}(t)-J_{S}(t))$
(15a) $\displaystyle-\dot{J}_{I}(t)$
$\displaystyle=\min_{\alpha_{I}(t)}c_{I}+c_{Q}(1-\alpha_{I}(t))-\gamma
J_{I}(t)$ (15b)

along with the terminal conditions $J_{S}(T)=J_{I}(T)=0$. The cost-to-go from
recovered state $J_{R}(t)$ is identically set to zero. From (15), we can
immediately see that for an infected individual, $\alpha_{I}(t)=1$ is the best
response action. Defining a switching function $\phi(t):=\kappa
p_{I}(t)(J_{I}(t)-J_{S}(t))-c_{Q}$, we can find the individual best response
strategy for a susceptible individual as follows: for any $t\in[0,T]$,

$\displaystyle\alpha_{S}^{*}(t)=\begin{cases}\alpha_{min}&\text{ if
}\phi(t)\geq 0\\\ 1&\text{ if }\phi(t)<0\end{cases}$

####  4.2.1 Numerical Computation of Mean Field Equilibrium

Figure 5: Mean field equilibrium control policy.

Nash equilibria correspond to a fixed point of the best response function. The
idea is to use an inductive sequence $u_{n+1}=f_{x_{0}}(u_{n})$ to reach the
fixed point. The Kolmogorov equations (8) are solved for a given control
$u_{n}$ and the best response is computed from solving the Bellman ODE (15)
backward in time, giving $u_{n+1}$. We use the iterative scheme detailed in
section 3.3.1 to compute the mean field equilibrium.

Figure 5 shows the individually optimal control for the set of parameters in
Table 2. We observe that the socially optimal control policy requires that the
population should reduce their social interaction for a longer period compared
to the mean field equilibrium policy.

###  4.3 Comparison of Mean Field Control and Mean Field Equilibrium
strategies

We can see from figure 4 that in an idealized setting where the regulating
authority has full control on contact between individuals, only the infected
needs to be isolated and this is reflected in the control actions $\alpha_{S}$
and $\alpha_{I}$ chosen for individuals in Susceptible and Infected states
respectively. However, when we look at a mean field equilibrium strategy
(figure 5), infected individuals are self interested and does not undergo
mitigation efforts. When the infection is near its peak, susceptible
individuals undergo protective measures to avoid getting the infection. This
is consistent with the free riding phenomenon which has been observed in the
context of COVID-19 pandemic.

##  5 Current State-of-the-Art in Optimal Control of Epidemics

Optimal control of epidemics concerns how a regulating authority can optimally
contain epidemic spread in a population. Depending on the underlying epidemic
model, we can classify them as (1) compartmental models (2) network based
models. Tools from optimal control were applied to compartmental models as
early as 1970s [1], [2]. Compartmental models are formulated as Markov chains
and due to their simplicity, they will be the go-to model in our discussions.
Nowzari, Preciado and Pappas [44] presents a survey on analysis and control of
epidemics in complex networks. In this section, we categorize the literature
into the following groups according to the control variable available to the
regulating authority: (1) non-pharmaceutical interventions (2) vaccination
strategies.

###  5.1 Optimal Control: Non-pharmaceutical Interventions

The first line of defence against epidemics in the absence of preventative
vaccines or in the case of vaccine hesitancy are non-pharmaceutical
interventions which reduce the mixing of infected people in the community.
Non-pharmaceutical interventions include (1) a full regional lockdown, where
only essential services are allowed to operate (2) mandatory use of personal
protective equipment (3) targeted interventions like symptomatic testing,
contact tracing and quarantine, and (4) travel restrictions.

In the initial stages of the COVID-19 pandemic, the governments all around the
world introduced strict regional lockdowns and travel bans to curb the disease
spread. However, this led to serious economic and social disruptions as
witnessed by the global economic recession following COVID-19. Early on in the
disease evolution, extensive testing, contact tracing and quarantine are
effective ways to prevent an exponential growth of the disease. This
necessitates a timely intervention from governments in terms of healthcare
infrastructure and testing capacity. Over the years, researchers have explored
how to model the cost associated with these interventions and how to frame the
problem faced by the regulator as an optimal control problem.

Abakuks [1], [2] first formulated stochastic and deterministic compartmental
models where optimal policies are computed numerically for the stochastic
model and analytically for the deterministic model. [1] considers a
compartmental SIR model of epidemic evolution. The length of infection period
is assumed to have an exponential distribution. Under the assumption that a
subset of infected population can be isolated instantaneously, the optimal
isolation policy is calculated. Note that this is a restrictive assumption
because instantaneous isolation is not practical and for many diseases,
testing is required to ascertain the state of an individual agent. Optimal
isolation policies are numerically computed for a stochastic model and
analytically computed for a deterministic model and the two policies compared.
[2] uses a similar model to analyze optimal vaccination policies.

Wickwire [57] extended the previous works of Abakuks, [1], [2] on isolation
and vaccination to analyze Kermack-McKendrick type compartmental models by
means of value function and the Bellman equation. Optimal isolation policies
for deterministic and stochastic epidemics were determined by [57]. Wickwire
et al. also relaxed the unrealistic assumption that an arbitrary number of
individuals can be isolated instantaneously, and instead assumed that there is
a hard constraint on the isolation rate. See [56] for a survey.

Behncke [10] further adapts Wickwire’s model [56] to a more general control
and cost setup and deals with both finite and infinite time horizon problems.
They use the Pontryagin’s maximum principle [11] throughout instead of Bellman
equations and study the control problem qualitatively, thus avoiding
differentiability assumptions on the value function.

In the context of the COVID-19 pandemic, optimal control has been applied to
compartmental and network based models. Tsay et al. [55] use a compartmental
SEAIR (susceptible-exposed-asymptomatic-infectious-recovered) model with an
asymptomatic compartment, provide methods for estimation of parameters, and
study the effect of isolation measures. The work by Perkins et al. [47]
performs optimal control analysis of a compartmental SEAIHV (susceptible-
exposed-asymptomatic- infected-hospitalised-vaccinated) model and concludes
that (a) heightened control early on in the pandemic is important for
achieving long-term success, (b) preventing a large wave that overwhelms the
public health system may not even be possible under some parameter
combinations and (c) prioritizing the minimization of deaths versus days under
control leads to vastly different outcomes. Kohler et al. [34] use a
compartmental model with 8 states and employ a robust model predictive control
(MPC) based feedback policy. This policy adapts the social distancing measures
cautiously and safely, thus leading to a minimum number of fatalities even if
measurements are inaccurate and the infection rates cannot be precisely
specified. Silva et al. [51] considers a social opinion biased SAIRP model to
provide forecasting mathematical models to anticipate the consequences of
political decisions.

Kantner and Koprucki [29] compute the optimal non-pharmaceutical intervention
strategy based on an extended SEIR (Susceptible-Exposed-Infected-Recovered)
model and continuous time optimal control theory. The optimal control must
satisfy the the following requirements: (1) minimize disease related deaths;
(2) establish a sufficient degree of natural immunity at the end of horizon,
to exclude a second wave; and (3) keep the socio-economic costs of
interventions minimum. This model was then calibrated to reproduce the initial
exponential growth phase of COVID-19 pandemic in Germany. The optimal
intervention strategy can be structured into 3 phases. In the first phase,
intervention begins with a strict initial lockdown to hold the effective
reproduction number, $\mathcal{R}_{eff}<1$. In the second phase, there is a
_critical period_ where the number of simultaneous cases is approximately held
constant ($\mathcal{R}_{eff}\approx 1$). During this period, the non-
pharmaceutical interventions are relaxed on a gradually increasing rate. Phase
3 commences after the critical period is over and the number of active cases
start to decay. In Phase 3, a final moderate tightening of measures is
required.

Kruse and Strack [36] derive optimal policies for social distancing in an SIR
model. They find that the optimal policy has the following features: (1) if
the death rate is not too sensitive to the number of infected, the optimal
policy has two phases. A first phase of strong interventions, followed by a
second phase with weaker interventions. (2) If the cost of reducing
transmission rate is linear, the optimal policy is always extreme (bang-bang).

Richard and co-authors [49] identify optimal age-stratified non-pharmaceutical
interventions to implement as a function of time since the onset of epidemic.
By applying optimal control theory, they arrive at a solution which minimizes
deaths and control costs. This strategy is implemented for three countries
with contrasted age distributions. They also show that this age-stratified
policy strongly outperforms a constant uniform control over the whole
population or over the younger population.

Bliman et al. [12] aims to study how partial or total containment can be
applied to an SIR epidemic model to minimize the epidemic final size
(cumulative number of infected cases during the complete course of an
epidemic). Theoretical and numerical results demonstrate that this approach
can lead to a significant decrease in epidemic final size. It is shown that
optimal intervention has to begin before the number of susceptible individuals
has crossed the herd immunity level.

Dimarco, Toscani and Zanella [19] use an SEIAR type compartmental model to
study the optimal control strategy when agents reduce their mean number of
contacts. The novelty lies in using a kinetic-type model to account for
heterogeneity in contact distribution of the population. Using a data driven
approach to determine the relevant epidemiological parameters, they show that
different types of control can lead to very different mitigation effects
according to the level of heterogeneity in contact distribution of agents.

Arruda et al. [6] incorporates multiple viral strains and reinfection to an
SEIR model and studies optimal control policies. The model is validated from
epidemiology data from COVID-19 in England and Brazil. They consider the cost
of mitigation efforts to grow exponentially with the mitigation effort and
solve an optimal control problem to to determine optimal mitigation measures.
Their results point to the importance of controlling an epidemic from outset
in hindering the emergence of new strains and avoiding the effects of a
prolonged epidemic.

Morris et al. [42] studies the role of non-pharmaceutical interventions in
reducing or delaying the peak number of infected individuals. A classical SIR
model is used to derive the theoretically optimum strategy and show that
easier to implement strategies without perfect information about current state
can perform near optimally. However, neither the optimal strategy nor the
near-optimal strategies are robust strategies. Small deviations in
intervention timing can cause large increases in the infection peak. Robust
controls should therefore aim at strong, early interventions which are
sustained in an ideal scenario.

The general conclusion which we can draw from the above literature is that for
linear cost of control, the optimal policy switches from exerting maximum
control effort until some point in time and then switches off the control
efforts after that time point. This _bang-bang_ solution, with at most one
switch, is common in similar problems. Although the bang-bang solution is
common, different types of solutions can be obtained for various formulations
of the optimal control problem [31], [33]. For an SIR model with quadratic
control cost, [33] shows that the optimal solution is not a bang-bang
controller.

###  5.2 Optimal Control with Vaccinations

Abakuks [2] started the work on optimal vaccination strategies for epidemics.
However, this model works under the assumption that the whole of susceptible
population is instantaneously vaccinated once a vaccine is available. This
work computes the optimal control for a stochastic model as well as a
deterministic model and provides a comparative study.

Bauch and Earn [9] models spread of childhood diseases and analyze voluntary
vaccination policies using a compartmental SIR model. Morton and Wickwire [43]
consider an immunisation model for susceptible individuals where the
vaccination control is bounded. This boundedness assumption on vaccination
control is needed because in practical scenarios, there will be a bound on the
maximum number of vaccines available. Kuga and co-authors [37] combine
evolutionary game theory and mathematical epidemiology to evaluate the
performance of vaccination subsidizing policies for a seasonal epidemic.
Multi-agent simulations are used to find how the topology of network structure
affects the vaccination behavior. Mean field approximations are used in this
paper to confirm the simulation results and to see the change in social
behavior when the vaccine is imperfect. The authors are able to point out
instances where vaccine subsidizing policies could be counterproductive.

Arefin and co-authors [5] build a mean-field vaccination game to analyse the
effect of an imperfect vaccine on a two-strain epidemic. The vaccination-
decision takes place at the beginning of an epidemic season and depends upon
the vaccine-effectiveness along with the cost. An additional situation where
the original strain continuously converts to a strain by mutation is also
considered.

Zaman, Kang, and Jung [59] propose an SIR epidemic model where a percentage of
susceptible population is vaccinated. They show that an optimal control exists
for the optimal vaccination problem and describe numerical simulations using a
Runge-Kutta fourth order procedure. Furthermore, a real-world example is
constructed where smoking is modelled as an epidemic, demonstrating the
efficiency of optimal control. Kar and Batabyal [30] study an optimal control
problem with vaccination coverage as a control variable on an SIR epidemic
model. With the help of the Pontryagin maximum principle and and an iterative
method, it is shown that there are two equilibria, one a disease free
equilibrium and the other an endemic equilibrium. The existence and stability
of these equilibria are studied and the optimality system is then solved
numerically.

Tchuenche and co-authors [53] analyze the dynamics of an influenza pandemic
model with vaccination and treatment using two preventive scenarios: increase
in vaccine uptake and decrease in vaccine uptake. The optimal control is
computed using Potryagin’s maximum principle and sensitivity analysis and
simulations are performed to determine the relative importance of transmission
parameters.

Acuna et al. [3] formulate an optimal control problem where vaccination
coverage (covering a certain percentage of the population in a given period)
and hospital occupancy are constrained and identifies vaccination policies
that minimizes the number of disability-adjusted years of life lost. A
compartmental model is used in the analysis and the burden of COVID-19 is
studied with respect to different scenarios such as optimal vs constant
vaccination policies, vaccine efficacy, induced vaccine immunity and natural
immunity.

#### Note

It should be noted that although this section focused on optimal control in
the context of epidemics, the same models and tools are directly applied to
general spreading processes in complex networks. Examples of optimal control
applied to these models include: malware propagation in computer networks
[31], [32], opinion dynamics in social networks [4], and adoption of a new
product in a marketplace. Nowzari et al. [44] provides a survey on analysis
and control of epidemics in complex networks.

##  6 Current State-of-the-Art in Mean Field Game Modeling of Epidemics

This section is devoted to the use of the mean field game approach in the
study of epidemic spread and control. We have categorized the relevant
literature into the following groups: (1) Analysis of epidemic spread and non-
pharmaceutical interventions (2) Study of vaccination effects (3) Control and
policy design for Epidemics.

The paper by Huang and Zhu [26] is a recent survey of the use of game
theoretic models (including mean field games) for epidemic spread and control.
The paper starts with a review of various models (such as the SIR model) for
epidemic spread. The focus of the survey is on the use of the models in
answering important questions such as what interventions, when to intervene,
etc. The review also provides a taxonomy of the literature based on (1) types
of games, such as static games, differential games, stochastic games,
evolutionary games, and mean field games (2) types of interventions, such as
social distancing, vaccination, quarantine, and antidotes; and (3) types of
decision-makers, such as individual nodes, adversaries, and central
authorities.

###  6.1 Mean Field Game Modeling and Analysis of Epidemic Spread and Non-
pharmaceutical Interventions

Elie, Hubert, and Turinici [21] formulate a model of COVID-19 spread and
control using an SIR model with an embedded mean field game. The control in
the SIR model is induced by the degree of contact among the individual nodes.
An individual node can decrease the contact rate during the epidemic and this
intervention has a social cost and an effort cost. The mean field game model
is shown to have an equilibrium in which the transmission rate of the epidemic
is reduced. The transmission rate achieved in the equilibrium is however
higher compared to a socially optimal solution. Similar results are derived in
the case of an SEIR model where an additional _exposed_ state is introduced.
The divergence between autonomous behavior and socially optimal behaviors are
shown to be more prominent immediately before and immediately after the peak
of the epidemic.

The paper by Petrakova and Krivorotko [48] presents a model for spread of an
epidemic like COVID-19 by considering three separate groups of population,
namely, suspectable (S), infected (I), removed (R) and cross-immune (C) ones.
The model is based on the mean-field control inside these three groups of
population. This model takes into account population heterogeneity and is
therefore superior to a traditional SIR model. The numerical experiments are
able to produce accurate estimates for COVID-19 spread in Novosibirsk, Russia
for two 100-day periods.

Cho [17] presents a mean field game model of individual nodes in a population
affected by an epidemic, where each node chooses a dynamic strategy of
interactions, given the benefits of the interactions as well as the risk
involved in getting infected. The mean field equilibrium that results from the
non-cooperative game model is computed and the outcome is compared to the
socially optimal outcome which maximizes the total utility of the population.
It is shown that the mean field equilibrium strategy is to make more contacts
than the level at which it would be socially optimal, in the absence of any
public policy or incentives. If incentives are offered and the cost of
incentivizing people is included, then it is shown that policies reducing
contacts of the infected should continued to be enforced even after the peak
of epidemic has passed. The paper also computes the price of anarchy to get an
idea of the conditions under which the discrepancies between the mean field
behaviour and socially optimal behaviour warrant public policy interventions.

Olmez, Aggarwal, Kim, Miehling, Basar, West, and Mehta [46] develop mean field
game models for the evolution of epidemics. The specific problem that is
modelled as a mean field game is the decision facing an individual node
regarding the degree of social activity. This is modelled as a mean field
game, taking into accounts healthcare related cost and benefits accruing from
social interactions. The authors investigated the fully observed setting as
well as a partially observed setting. The paper presents a complete analysis
of the fully observed case and some analytical results for the partially
observed case. In the fully observed case, each individual node knows its
epidemiological status perfectly. It turns out that a susceptible node will
engage in a social interaction if any only the reward outweighs the risk
whereas an infected individual will choose to quarantine. In the partially
observed case, the nodes do not know their epidemiological status perfectly
and it turns out that an infected node behaves like a susceptible node. This
could make the epidemic spread faster. The paper [45] is a follow-up of the
work in [46] and studies the behavior of self-interested agents in a large
heterogeneous population and how pre-symptomatic agents can drive the growth
in infection. A mean field type optimal control model is used to investigate
the effect of partial observation on individual decision making.

Tembine [54] considers a class of mean-field-type games with discrete-
continuous state spaces and presents Bellman systems that provide sufficiency
conditions for the existence of mean-field-type equilibria in state-and-mean-
field-type feedback form. The author derives unnormalized master adjoint
systems (MASS) which provide a methodology powerful enough to model the
propagation of the COVID-19 virus in the globe. Based on MASS, the author
presents data-driven modelling and analytics for mitigating COVID-19. The
model is very versatile and captures many aspects: untested cases, age-
structure, decision-making, gender, pre-existing health conditions, location,
testing capacity, hospital capacity, and a mobility map of local areas,
including in-cities, inter-cities, and international aspects. The author shows
that this data-driven model can capture the trends of the reported data on
COVID-19.

The report by Bremaud [13] models the propagation of epidemics in which
individual nodes have control on some parameters such as vaccination rate or
social interactions. The report focuses on two models based on the standard
SIR model. The first model focuses on vaccination control. The existence of a
unique mean field equilibrium is shown and a numerical method is presented for
computing the equilibrium. The second model captures the effect of social
contacts. Here again, a mean field equilibrium is computed numerically.

Bremaud and Ullmo [14] consider a SIR compartmental model with social
structure, where individuals are grouped by age and interact in different
settings where individuals have contact with the other (schools, households,
community, etc.). The mean field Nash equilibrium is computed in this setting
and is compared with the social optimum. They also investigate how an
approximation of socially optimum policy can be obtained with social policies
like lockdown.

Aurell, Carmona, Dayanikli, and Lauriere [7] discuss the modeling of epidemics
using graphon games. A crucial assumption in mean field game theory is that
nodes are indistinguishable and interact identically regardless of with whom
they interact. In the context of modeling of epidemics, there is often a need
to model the diversity of individual nodes and the variation of their
interactions (for example, travel restrictions, multiple age groups with
distinct social behavior and risk profiles, and a wide range of co-morbid
conditions, etc.). Games with a large number of non-identical players can be
analyzed with so-called graphon games whenever the network specifying the
interactions is dense. A graphon can be viewed as the limit of a dense random
graph. The authors develop a framework for epidemic modeling using graphon
games and analyze their Nash equilibria. They provide a sufficient condition
for the existence of a Nash equilibrium and propose a numerical approach based
on machine learning tools to compute the equilibrium. The paper also presents
numerical results on several applications of compartmental models for
epidemics.

In the paper by Kordonis, Logos, and Papavassilopoulos [35], the authors study
a dynamic game model that captures social distancing behaviors during an
epidemic, assuming a continuum of players and infection dynamics caused by
individual choice. The authors assume a slight variant of the SIR model. The
players have incomplete information about their infection state, and their
choice of actions is determined by the individual beliefs on the probabilities
of being susceptible, infected, or recovered. The cost of each player is
determined by the infection and the contact factor. The authors show that a
Nash equilibrium is guaranteed to exist and develop an efficient computational
procedure for the same. Even when the players have the same parameters, they
could exhibit different behaviors. The work studies the effect of various
parameters like the vulnerability (co-morbidity for example) of players, the
time horizon, and the various interventions on the optimal policies and the
costs of the nodes.

The paper by Gao, Li, Pan, and Poor [22] is concerned with accurate modeling
of COVID-19 evolution with mean field evolutionary dynamics (MFEDs) by
invoking optimal transport theory and mean field games on graphs. The authors
compute the payoff functions for different individual states from the commonly
used replicator dynamics (RDs) and employ them to govern the evolution of
epidemics. The authors compare epidemic modeling based on MFEDs with that
based on RDs through numerical experiments. The efficacy of MFEDs is
demonstrated by fitting the model to the COVID-19 statistics of Wuhan, China.
The authors also analyze the effects of one-time social distancing as well as
the seasonality of COVID- 19 through the post-pandemic period.

###  6.2 Mean Field Game Modeling of the Effect of Vaccinations

Doncel, Gast, and Gaujal [20, 24] present a mean field game model under the
SIR model when the individual nodes in the population choose when to get
vaccinated. The authors prove the existence of a unique mean field equilibrium
which shows a bang-bang control behavior. Specifically, there is a threshold
time until which the individuals will get vaccinated at maximal rate and
beyond which they do not get vaccinated at all. Interestingly, the vaccination
strategy that minimizes the total cost has the same structure as the mean
field equilibrium. However, the vaccination period of the mean field
equilibrium is always smaller than the one that minimizes the total cost. This
essentially means that vaccination should be subsidized appropriately in order
to nudge the people to exhibit optimal vaccination behavior.

Hubert and Turinici [28] consider an SIR model with vaccination where the
vaccination is not mandatory. In particular, they study newborn vaccination.
The evolution of each individual node is modelled as a Markov chain and the
dynamics of the population is modelled using a mean field approach. The
vaccination decision optimizes a criterion depending on the time-dependent
societal vaccination rate and the future epidemic dynamics. It is shown that a
Nash mean field equilibrium exists in the proposed model. The paper presents a
method for numerical computation of the equilibrium.

Laguzet, Turinici, and Yahiaoui [38] analyzes individual vaccination
strategies for a Susceptible-Infected-Recovered (SIR) model. The decision on
whether or not to get vaccinated is taken by individual nodes and is based on
the risk of infection, the possible side effects of the vaccine and the
overall severity of the epidemic course. An important realistic condition is
captured, namely limited capacity of vaccination. It is shown that a mean
field equilibrium exists between the individual decisions and the epidemic
evolution. This enables one to compute an optimal vaccination policy.

Salvarani and Turinici [50] model the behavior of individual nodes in
protecting themselves against an epidemic when the vaccination is voluntary.
They capture two aspects of vaccines namely imperfect efficacy and limited
duration of protection. They show the existence of a Nash equilibrium,
assuming non-persistent immunity. They also provide a numerical method for
computing the equilibrium. Numerical experiments reveal interesting insights
which are useful in planning a vaccination campaign (timing, etc.) in the
realistic scenario of imperfect vaccines.

Lee, Liu, Li, and Osher [39] deals with the problem of transporting and
distributing the COVID-19 vaccine(s), to achieve optimal control of the
pandemic. The methodology is based on a mean-field variational problem in a
spatial domain, which controls the propagation of pandemic by the optimal
transportation strategy of vaccine distribution. The authors integrate the
vaccine distribution into the mean-field SIR model designed by the authors in
[40]. Experimental results show that the proposed model indeed provides
effective strategies in vaccine distribution on a spatial domain.

###  6.3 Mean Field Game Approach to Control and Policy Design for Epidemics

Aurell, Carmona, Dayanikli, and Lauriere [8] model the spread and regulation
of an epidemic as a Stackelberg game between the federal Government and the
population. The Government which formulates mitigation policies through
incentives is the leader in the game while the mean field of nodes (citizens)
representing the population is the follower. A standard SIR model is
considered to represent the spread of the epidemic. The paper considers for
the first time, a compartmental model of epidemics capturing the interplay
between independent autonomous nodes and a regulator. The individual nodes
interact via a noncooperative game where utility maximization is equivalent to
minimizing the individual cost by controlling the degree of interactions with
other nodes (rates of transitions between the states). The Nash equilibrium of
this non-cooperative game is determined. The leader, guided by a social
objective, applies incentive policies and non-pharmaceutical interventions
which determine the Nash equilibrium of the mean field game among the
individual nodes.

The mean field game proposed is an extended MFG in which dependency on the
joint distribution of action and state is explicitly taken into account (in
earlier models of extended MFG, this dependency involved only the marginal
distributions). In terms of the technical novelty of the model, this paper
formulates the leader’s problem under the constraint of the mean field Nash
equilibrium of the individual nodes as an optimal control problem with two
forward stochastic equations. The numerical solution of this problem uses an
approximation of the population by an interacting particle system and the
approximation of controls by neural networks, including the leader’s policy.
The optimization of the leader’s cost is carried out using a variant of
stochastic gradient descent to update the neural networks’ parameters (see [8]
for details).

The numerical experiments reveal two interesting insights: (a) In the first
case, when the leader applies a containment policy, it is found that the nodes
are more cautious about their interactions in the resulting Nash equilibrium
when compared to a free spread scenario without any containment policy in
place. An early lockdown has a higher impact than any action taken at a later
point in time. (b) When the leader optimizes over its policies to minimize its
own cost, the Nash equilibrium of the nodes shows interesting outcomes. For
example, when the SIR model is expanded to include two additional states
namely Exposed (E) and Deceased (D), the nodes are found to choose lower
contact levels in their Nash equilibrium than recommended by the regulator.

Hubert, Mastrolia, Possamaï, and Warin [27] investigate the optimal control of
an epidemic by offering incentives to lockdown and testing. The interaction
between the Government and the individual nodes in the population is modelled
as a principal–node problem with moral hazard. This results in a Stackelberg
game model. The spread of the epidemic is modelled by stochastic model with
SIS or SIR compartments. The transmission rate of the epidemic is proposed to
be controlled by decreasing the contact rate between individuals (through
various means such as reducing physical interactions). This causes a social
cost as well as monetary cost to the individual nodes. The paper proposes that
the government can aid this through a tax or subsidy as well as implement a
testing policy. The testing policy will enable to determine accurately the
spread of the epidemic, facilitating isolation of infected individuals. The
work derives an optimal form of the tax which is indexed on the (a) proportion
of infected individuals and (b) the optimal effort of the population, namely
the transmission rate chosen in response to this tax. This yields an
optimization problem to be solved by the government, namely solving an
appropriate Hamilton–Jacobi–Bellman equation. Experimental results show that
the imposition of a tax policy will induce the individual nodes to cut down on
their interactions.

The paper also studies the influence a testing policy can have on limiting the
spread of the epidemic. If the testing is done aggressively, then individuals
who have tested positive can be isolated. This will enable the individual
nodes to interact more freely than if there were no testing and no isolation
of individuals who tested positive.

Charpentier, Elie, Lauriere, and Tran [16] consider an extended SIR model with
several realistic features of COVID-19 pandemic incorporated. The authors
derive an optimal policy for controlling the spread of the epidemic using and
taking into account: (a) lockdown intervention as well as detection and
isolation intervention, (b) the trade-off between the sanitary and the socio-
economic cost of the pandemic, and (c) limited capacity of intensive care
units (ICUs). A detailed sensitivity analysis is carried out with parameters
chosen from the COVID-19 literature. It is shown that the optimal lockdown
policy is structured into 4 phases: (1) A quick and strong lockdown
intervention to stop the exponential growth of the contagion; (2) A short
transition to reduce the prevalence of the virus; (3) a long period with full
ICU capacity and stable virus prevalence; (4) a return to normal social
interactions with disappearance of the virus. This optimal scenario avoids the
second wave of infection, provided the lockdown is released sufficiently
slowly. It is also shown that with aggressive testing followed by isolation of
infected individuals, social distancing norms can be relaxed.

Lee, Liu, Tembine, and Osher [40] presents a mean field game model to control
the spread of epidemics in the spatial domain. A standard SIR model is
considered and the spatial velocities in the three states S,I, and R are
chosen as the control variables. The authors consider three crowds,
Susceptible, Infected, and Recovered, which evolve spatially. This is due to
mobility, interactions, etc. The central planner seeks to mitigate the risk of
infection by controlling the spatial velocity of the nodes in the three
states. The paper provides efficient algorithms based on proximal primal-dual
methods for obtaining the solutions. It is shown that the proposed model can
be effectively used for identifying the infected and susceptible populations
in a spatial domain.

Xu, Wu, and Topcu [58] discusses three specific COVID-19 epidemic control
models: (1) the susceptible, exposed, infectious, recovered (SEIR) model with
vaccination control; (2) the SEIR model with shield immunity control; and (3)
the susceptible, un-quarantined infected, quarantined infected, confirmed
infected (SUQC) model with quarantine control. The paper expresses control
outcomes using metric temporal logic (MTL) which is a formal specification
language. An example of a control outcome would be: the population immune from
the disease should exceed 200 thousand in the next 100 to 200 days [58]. The
paper presents methods to synthesize control strategies with MTL
specifications and presents simulation results for three scenarios: (a)
vaccination control for the COVID-19 epidemic; (b) shield immunity control for
the COVID-19; and (c) quarantine control for the COVID-19.

##  7 Future Directions

A prominent direction that has been pursued by many researchers is to
investigate how socially optimal control and mean field game based control
differ in an epidemic model. The control policies influence various costs
including infection cost, vaccination cost, quarantine cost, etc. Assuming
that these costs can be suitably chosen (with the support of a regulatory
authority), a few researchers have looked into the problem of nudging the
individuals of the population in a way that mean field game control approaches
the performance of socially optimal control. Typically, researchers conduct
mean field game analyses and subsequently suggest certain incentives or
deterrents based on the associated equilibria. However, the analyses do not
take into account how the individuals react to the policies, except for the
work by Aurell, Carmona, Dayanikli, and Lauriere [8] who arrive at an optimal
contract by explicitly taking into account the reaction of the individuals
through a Stackelberg game type of model. This is a promising research
direction. One specific problem here is to extend the work of [8] by taking
vaccinations into account.

In the existing literature, the population is treated as consisting of a
single group of individuals. In reality, however, there are multiple logical
groups such that the weights on the compartmental model would be different for
different groups: (a) healthy, robust individuals; (b) individuals with co-
morbidities; (c) elderly individuals (which are more vulnerable), etc. It
would be good to extend the analysis and control taking these groupings also
into account.

The ongoing pandemic is continuously evolving. Research and innovation have
led to multiple vaccines and vaccinated people are well protected. However,
compliance to vaccination and pandemic appropriate behaviour is not 100
percent. It would be valuable and interesting to capture some of these
phenomena in the epidemic models. Data about the ongoing epidemic will be very
useful for these studies. The models developed will be much more credible if
available data is incorporated appropriately.

Testing plays an important role in controlling the spread of the epidemic and
also in better management of the epidemic effects. For infected individuals,
early detection is important for medical intervention and prevention of
permanent harm to their bodies. It is important to study this problem as a
social problem and compute the optimal testing strategy for the community as a
whole through an appropriate model.

Analytical techniques are hard to develop for the problems mentioned in this
paper. This offers a formidable technical challenge and will be an interesting
research direction. Most existing solutions are numerical. Therefore, another
promising direction will be to develop improved numerical methods.

## References

  * [1] A. Abakuks, “An optimal isolation policy for an epidemic,” _Journal of Applied Probability_ , vol. 10, no. 2, pp. 247–262, 1973.
  * [2] A. Abakuks, “Optimal immunisation policies for epidemics,” _Advances in Applied Probability_ , vol. 6, no. 3, pp. 494–511, 1974.
  * [3] M. A. Acuña-Zegarra, S. Díaz-Infante, D. Baca-Carrasco, and D. Olmos-Liceaga, “Covid-19 optimal vaccination policies: A modeling study on efficacy, natural and vaccine-induced immunity responses,” _Mathematical biosciences_ , vol. 337, p. 108614, 2021.
  * [4] G. Albi, L. Pareschi, and M. Zanella, “On the optimal control of opinion dynamics on evolving networks,” in _IFIP Conference on System Modeling and Optimization_. Springer, 2015, pp. 58–67.
  * [5] M. R. Arefin, K. A. Kabir, and J. Tanimoto, “A mean-field vaccination game scheme to analyze the effect of a single vaccination strategy on a two-strain epidemic spreading,” _Journal of Statistical Mechanics: Theory and Experiment_ , vol. 2020, no. 3, p. 033501, 2020.
  * [6] E. F. Arruda, S. S. Das, C. M. Dias, and D. H. Pastore, “Modelling and optimal control of multi strain epidemics, with application to covid-19,” _PLOS ONE_ , vol. 16, no. 9, pp. 1–18, 09 2021. [Online]. Available: https://doi.org/10.1371/journal.pone.0257512
  * [7] A. Aurell, R. Carmona, G. Dayanikli, and M. Lauriere, “Finite state graphon games with applications to epidemics,” _arXiv preprint arXiv:2106.07859_ , 2021.
  * [8] A. Aurell, R. Carmona, G. Dayanikli, and M. Lauriere, “Optimal incentives to mitigate epidemics: a stackelberg mean field game approach,” _SIAM Journal on Control and Optimization_ , no. 0, pp. S294–S322, 2022.
  * [9] C. T. Bauch and D. J. Earn, “Vaccination and the theory of games,” _Proceedings of the National Academy of Sciences_ , vol. 101, no. 36, pp. 13 391–13 394, 2004.
  * [10] H. Behncke, “Optimal control of deterministic epidemics,” _Optimal control applications and methods_ , vol. 21, no. 6, pp. 269–285, 2000.
  * [11] D. Bertsekas, _Dynamic programming and optimal control: Volume I_. Athena scientific, 2012, vol. 1.
  * [12] P.-A. Bliman, M. Duprez, Y. Privat, and N. Vauchelet, “Optimal immunity control and final size minimization by social distancing for the sir epidemic model,” _Journal of Optimization Theory and Applications_ , vol. 189, no. 2, pp. 408–436, 2021.
  * [13] L. Brémaud and F. Restagno, “Mean field game modeling of epidemic propagation,” 2021. [Online]. Available: http://lptms.u-psud.fr/membres/ullmo/Articles/stageM2-Louis-Bremaud.pdf
  * [14] L. Bremaud and D. Ullmo, “Social structure description of epidemic propagation with a mean-field game paradigm,” _Physical Review E_ , vol. 106, no. 6, p. L062301, 2022.
  * [15] R. Carmona, F. Delarue, _et al._ , _Probabilistic theory of mean field games with applications I-II_. Springer, 2018.
  * [16] A. Charpentier, R. Elie, M. Laurière, and V. C. Tran, “Covid-19 pandemic control: balancing detection policy and lockdown intervention under icu sustainability,” _Mathematical Modelling of Natural Phenomena_ , vol. 15, p. 57, 2020.
  * [17] S. Cho, “Mean-field game analysis of sir model with social distancing,” _arXiv preprint arXiv:2005.06758_ , 2020.
  * [18] R. Darling, “Fluid limits of pure jump markov processes: a practical guide,” _arXiv preprint math/0210109_ , 2002.
  * [19] G. Dimarco, G. Toscani, and M. Zanella, “Optimal control of epidemic spreading in the presence of social heterogeneity,” _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , vol. 380, no. 2224, p. 20210160, 2022. [Online]. Available: https://royalsocietypublishing.org/doi/abs/10.1098/rsta.2021.0160
  * [20] J. Doncel, N. Gast, and B. Gaujal, “A mean field game analysis of sir dynamics with vaccination,” _Probability in the Engineering and Informational Sciences_ , pp. 1–18, 2020.
  * [21] R. Elie, E. Hubert, and G. Turinici, “Contact rate epidemic control of covid-19: an equilibrium view,” _Mathematical Modelling of Natural Phenomena_ , vol. 15, p. 35, 2020.
  * [22] H. Gao, W. Li, M. Pan, Z. Han, and H. V. Poor, “Modeling covid-19 with mean field evolutionary dynamics: Social distancing and seasonality,” _Journal of Communications and Networks_ , vol. 23, no. 5, pp. 314–325, 2021\.
  * [23] N. Gast, B. Gaujal, and J.-Y. Le Boudec, “Mean field for markov decision processes: From discrete to continuous optimization,” _IEEE Transactions on Automatic Control_ , vol. 57, no. 9, pp. 2266–2280, 2012.
  * [24] B. Gaujal, J. Doncel, and N. Gast, “Vaccination in a large population: mean field equilibrium versus social optimum,” in _NETGCOOP 2020-10th International Conference on NETwork Games, COntrol and OPtimization_ , 2021, pp. 1–9.
  * [25] J. Guan, Y. Wei, Y. Zhao, and F. Chen, “Modeling the transmission dynamics of covid-19 epidemic: a systematic review,” _Journal of Biomedical Research_ , vol. 34, no. 6, p. 422, 2020.
  * [26] Y. Huang and Q. Zhu, “Game-theoretic frameworks for epidemic spreading and human decision-making: A review,” _Dynamic Games and Applications_ , pp. 1–42, 2022.
  * [27] E. Hubert, T. Mastrolia, D. Possamaï, and X. Warin, “Incentives, lockdown, and testing: from thucydides’ analysis to the covid-19 pandemic,” _Journal of Mathematical Biology_ , vol. 84, no. 5, pp. 1–48, 2022.
  * [28] E. Hubert and G. Turinici, “Nash-mfg equilibrium in a sir model with time dependent newborn vaccination,” _Ricerche di matematica_ , vol. 67, no. 1, pp. 227–246, 2018.
  * [29] M. Kantner and T. Koprucki, “Beyond just “flattening the curve”: Optimal control of epidemics with purely non-pharmaceutical interventions,” _Journal of Mathematics in Industry_ , vol. 10, no. 1, pp. 1–23, 2020.
  * [30] T. Kar and A. Batabyal, “Stability analysis and optimal control of an sir epidemic model with vaccination,” _Biosystems_ , vol. 104, no. 2, pp. 127–135, 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0303264711000396
  * [31] M. R. Khouzani, S. Sarkar, and E. Altman, “Dispatch then stop: Optimal dissemination of security patches in mobile wireless networks,” in _49th IEEE Conference on Decision and Control (CDC)_. IEEE, 2010, pp. 2354–2359.
  * [32] M. Khouzani, E. Altman, and S. Sarkar, “Optimal quarantining of wireless malware through power control,” in _2009 Information Theory and Applications Workshop_. IEEE, 2009, pp. 301–310.
  * [33] M. Khouzani, S. Sarkar, and E. Altman, “Optimal control of epidemic evolution,” in _2011 Proceedings IEEE INFOCOM_. IEEE, 2011, pp. 1683–1691.
  * [34] J. Köhler, L. Schwenkel, A. Koch, J. Berberich, P. Pauli, and F. Allgöwer, “Robust and optimal predictive control of the covid-19 outbreak,” _Annual Reviews in Control_ , vol. 51, pp. 525–539, 2021.
  * [35] I. Kordonis, A.-R. Lagos, and G. P. Papavassilopoulos, “Dynamic games of social distancing during an epidemic: Analysis of asymmetric solutions,” _Dynamic Games and Applications_ , vol. 12, no. 1, pp. 214–236, 2022.
  * [36] T. Kruse and P. Strack, “Optimal control of an epidemic through social distancing,” Cowles Foundation for Research in Economics, Yale University, Cowles Foundation Discussion Papers 2229R, 2020. [Online]. Available: https://EconPapers.repec.org/RePEc:cwl:cwldpp:2229r
  * [37] K. Kuga, J. Tanimoto, and M. Jusup, “To vaccinate or not to vaccinate: A comprehensive study of vaccination-subsidizing policies with multi-agent simulations and mean-field modeling,” _Journal of theoretical biology_ , vol. 469, pp. 107–126, 2019.
  * [38] L. Laguzet, G. Turinici, and G. Yahiaoui, “Equilibrium in an individual-societal sir vaccination model in presence of discounting and finite vaccination capacity,” in _New Trends in Differential Equations, Control Theory and Optimization: Proceedings of the 8th Congress of Romanian Mathematicians_. World Scientific, 2016, pp. 201–214.
  * [39] W. Lee, S. Liu, W. Li, and S. Osher, “Mean field control problems for vaccine distribution,” _Research in the Mathematical Sciences_ , vol. 9, no. 3, p. 51, 2022.
  * [40] W. Lee, S. Liu, H. Tembine, W. Li, and S. Osher, “Controlling propagation of epidemics via mean-field control,” _SIAM Journal on Applied Mathematics_ , vol. 81, no. 1, pp. 190–207, 2021.
  * [41] S. Lenhart and J. T. Workman, _Optimal control applied to biological models_. Chapman and Hall/CRC, 2007.
  * [42] D. H. Morris, F. W. Rossine, J. B. Plotkin, and S. A. Levin, “Optimal, near-optimal, and robust epidemic control,” _Communications Physics_ , vol. 4, no. 1, pp. 1–8, 2021.
  * [43] R. Morton and K. H. Wickwire, “On the optimal control of a deterministic epidemic,” _Advances in Applied Probability_ , vol. 6, no. 4, pp. 622–635, 1974.
  * [44] C. Nowzari, V. M. Preciado, and G. J. Pappas, “Analysis and control of epidemics: A survey of spreading processes on complex networks,” _IEEE Control Systems Magazine_ , vol. 36, no. 1, pp. 26–46, 2016.
  * [45] S. Y. Olmez, S. Aggarwal, J. W. Kim, E. Miehling, T. Başar, M. West, and P. G. Mehta, “How does a rational agent act in an epidemic?” in _2022 IEEE 61st Conference on Decision and Control (CDC)_. IEEE, 2022, pp. 5536–5543.
  * [46] S. Y. Olmez, S. Aggarwal, J. W. Kim, E. Miehling, T. Başar, M. West, and P. G. Mehta, “Modeling presymptomatic spread in epidemics via mean-field games,” in _2022 American Control Conference (ACC)_. IEEE, 2022, pp. 3648–3655.
  * [47] T. A. Perkins and G. España, “Optimal control of the covid-19 pandemic with non-pharmaceutical interventions,” _Bulletin of Mathematical Biology_ , vol. 82, no. 9, pp. 1–24, 2020.
  * [48] V. Petrakova and O. Krivorotko, “Mean field game for modeling of covid-19 spread,” _Journal of Mathematical Analysis and Applications_ , p. 126271, 2022.
  * [49] Q. Richard, S. Alizon, M. Choisy, M. T. Sofonea, and R. Djidjou-Demasse, “Age-structured non-pharmaceutical interventions for optimal control of covid-19 epidemic,” _PLOS Computational Biology_ , vol. 17, no. 3, pp. 1–25, 03 2021. [Online]. Available: https://doi.org/10.1371/journal.pcbi.1008776
  * [50] F. Salvarani and G. Turinici, “Optimal individual strategies for influenza vaccines with imperfect efficacy and durability of protection,” _Mathematical Biosciences and Engineering_ , vol. 15, no. 3, pp. 629–652, 2018.
  * [51] C. J. Silva, C. Cruz, D. F. Torres, A. P. Muñuzuri, A. Carballosa, I. Area, J. J. Nieto, R. Fonseca-Pinto, R. Passadouro, E. S. Dos Santos, _et al._ , “Optimal control of the covid-19 pandemic: controlled sanitary deconfinement in portugal,” _Scientific reports_ , vol. 11, no. 1, pp. 1–15, 2021.
  * [52] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [53] J. Tchuenche, S. Khamis, F. Agusto, and S. Mpeshe, “Optimal control and sensitivity analysis of an influenza model with treatment and vaccination,” _Acta biotheoretica_ , vol. 59, no. 1, pp. 1–28, 2011.
  * [54] H. Tembine, “Covid-19: Data-driven mean-field-type game perspective,” _Games_ , vol. 11, no. 4, p. 51, 2020.
  * [55] C. Tsay, F. Lejarza, M. A. Stadtherr, and M. Baldea, “Modeling, state estimation, and optimal control for the us covid-19 outbreak,” _Scientific reports_ , vol. 10, no. 1, pp. 1–12, 2020.
  * [56] K. Wickwire, “Mathematical models for the control of pests and infectious diseases: a survey,” _Theoretical population biology_ , vol. 11, no. 2, pp. 182–238, 1977.
  * [57] K. H. Wickwire, “Optimal isolation policies for deterministic and stochastic epidemics,” _Mathematical biosciences_ , vol. 26, no. 3-4, pp. 325–346, 1975\.
  * [58] Z. Xu, B. Wu, and U. Topcu, “Control strategies for covid-19 epidemic with vaccination, shield immunity and quarantine: A metric temporal logic approach,” _PLOS ONE_ , vol. 16, no. 3, pp. 1–20, 03 2021. [Online]. Available: https://doi.org/10.1371/journal.pone.0247660
  * [59] G. Zaman, Y. Han Kang, and I. H. Jung, “Stability analysis and optimal vaccination of an sir epidemic model,” _Biosystems_ , vol. 93, no. 3, pp. 240–249, 2008. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0303264708001044