[1]Yi Ouyang

1]Preferred Networks America, Inc., Burlingame, CA, USA

2]Google, Mountain View, CA, USA

3]Department of Electrical Engineering and Computer Science, University of
Michigan, Ann Arbor, MI, USA

# An Approach to Stochastic Dynamic Games with Asymmetric Information and
Hidden Actions

<EMAIL_ADDRESS>Hamidreza Tavafoghi<EMAIL_ADDRESS>Demosthenis Teneketzis<EMAIL_ADDRESS>[  [  [

###### Abstract

We consider in discrete time, a general class of sequential stochastic dynamic
games with asymmetric information with the following features. The underlying
system has Markovian dynamics controlled by the agents’ joint actions. Each
agent’s instantaneous utility depends on the current system state and the
agents’ joint actions. At each time instant each agent makes a private noisy
observation of the current system state and the agents’ actions in the
previous time instant. In addition, at each time instant all agents have a
common noisy observation of the current system state and their actions in the
previous time instant. Each agent’s actions are part of his private
information. The objective is to determine Bayesian Nash Equilibrium (BNE)
strategy profiles that are based on a compressed version of the agents’
information and can be sequentially computed; such BNE strategy profiles may
not always exist. We present an approach/methodology that achieves the above-
stated objective, along with an instance of a game where BNE strategy profiles
with the above-mentioned characteristics exist. We show that the methodology
also works for the case where the agents have no common observations.

###### keywords:

Dynamic games, asymmetric information, hidden actions, common information,
information compression, sequential decomposition

##  1 Introduction

We study, in discrete time, a general class of sequential stochastic dynamic
games with asymmetric information. We consider a setting where the underlying
system has Markovian dynamics controlled by the agents’ joint actions. Each
agent’s instantaneous utility depends on the agents’ joint actions and the
system state. At each time instant each agent makes a private noisy
observation that depends on the current system state and the agents’ actions
in the previous time instant. In addition, at each time instant all agents may
have a common noisy observation of the system state and their actions in the
previous time instant. The agents’ actions are hidden, that is, each agent’s
actions are not directly observable by the other agents. Therefore, at every
time instant agents have asymmetric and imperfect information about the game’s
history. Dynamic games with the above features arise in engineering
(cybersecurity, transportation, energy markets), in economics (industrial
organization), and in socio-technological applications.

As pointed out in Tang et al (2022), the key challenges in the study of
dynamic games with asymmetric information are: (i) The domain of agents’
strategies increases with time, as the agents acquire information over time.
Thus, the computational complexity of the agents’ strategies increases with
time. (ii) Due to signaling111 Signaling in games is more complex than
signaling in teams because the agents have diverging incentives and their
strategies are their own private information.  (Ho, 1980), in many instances
an agent’s assessment of the game’s status at time $t$, therefore his strategy
at time $t$, depends on the strategies of agents who acted before him.
Consequently, we cannot obtain the standard sequential decomposition (that
sequentially determines the components of an equilibrium strategy profile) of
the kind provided by the standard dynamic programming algorithm (where the
agent’s optimal strategy at any time $t$ does not depend on past strategies
(Kumar and Varaiya, 1986, Chapter 6.5)).

To address these challenges, we can look for equilibrium strategy profiles
that are based on a compressed version of the agents’ information and can be
sequentially computed. However, such equilibrium strategy profiles may not
exist.

In this paper we propose an approach, described in detail in Section 3, that
addresses the above-stated challenges. According to this approach, we first
compress the agents’ private and common information at each time instant.
Then, we define strategies based on the compressed information and show that
Bayesian Nash Equilibria (BNE) based on these strategies can be determined
sequentially in time moving backwards, if each step of this backwards
procedure has a solution. Finally, we provide an example where a BNE strategy
profile based on compressed information exists.

We show that the proposed approach works for the case where the agents have no
common observations and their actions are hidden.

###  1.1 Related Literature

Dynamic games with asymmetric information have been extensively investigated
in the literature in the context of repeated discounted games; see Zamir
(1992); Forges (1992); Aumann et al (1995); Mailath and Samuelson (2006) and
the references therein. The key feature of these games is the absence of a
dynamic system. Moreover, the works on repeated games study primarily their
asymptotic properties when the horizon is infinite and agents are sufficiently
patient (i.e. the discount factor is close one). In repeated games, agents
play a stage (static) game repeatedly over time. The main objective of this
strand of literature is to explore situations where agents can form self-
enforcing punishment/reward mechanisms so as to create additional equilibria
that improve upon the payoffs they can get by simply playing an equilibrium of
the stage game over time. Recent works (see Hörner et al (2011); Escobar and
Toikka (2013); Sugaya (2012)) adopt approaches similar to those used in
repeated games to study infinite horizon dynamic games with asymmetric
information when there is an underlying dynamic Markovian system. Under
certain conditions on the system dynamics and information structure, the
authors of Hörner et al (2011); Escobar and Toikka (2013); Sugaya (2012)
characterize a set of asymptotic equilibria attained when the agents are
sufficiently patient.

The problem we study in this paper is different from the ones in Zamir (1992);
Forges (1992); Aumann et al (1995); Mailath and Samuelson (2006); Hörner et al
(2011); Escobar and Toikka (2013); Sugaya (2012) in two aspects. First, we
consider a class of dynamic games where the underlying system has general
Markovian dynamics and a general information structure, and we do not restrict
attention to asymptotic behaviors when the horizon is infinite and the agents
are sufficiently patient. Second, we study situations where the decision
problem that each agent faces, in the absence of strategic interactions with
other agents, is a Partially Observed Markov Decision Process (POMDP), which
is a complex problem to solve by itself. Therefore, reaching (and computing) a
set of equilibrium strategies, which take into account the strategic
interactions among the agents, is a very challenging task. As a result, it is
not very plausible for the agents to seek reaching equilibria that are
generated by the formation of self-enforcing punishment/reward mechanisms
similar to those used in infinitely repeated games. We believe that our
results provide new insight into the behavior of strategic agents in complex
and dynamic environments, and complement the existing results in the repeated
games literature.

Stochastic dynamic zero-sum games with asymmetric information have been
studied in Renault (2006); Cardaliaguet et al (2015); Gensbittel and Renault
(2015); Li et al (2017); Kartik and Nayyar (2021); Zheng and Castañón (2013);
Li and Shamma (2014). The authors of Renault (2006); Cardaliaguet et al
(2015); Zheng and Castañón (2013); Li and Shamma (2014) study zero-sum games
with Markovian dynamics and lack of information on one side (i.e. one informed
and one uninformed agent). The authors of Gensbittel and Renault (2015); Li et
al (2017); Kartik and Nayyar (2021) study zero-sum games with Markovian
dynamics and lack of information on both sides. The works of Renault (2006);
Cardaliaguet et al (2015); Gensbittel and Renault (2015); Li et al (2017);
Kartik and Nayyar (2021); Zheng and Castañón (2013); Li and Shamma (2014)
consider specific information structures. Specifically: the actions of both
agents are publicly observed; in Renault (2006); Cardaliaguet et al (2015);
Zheng and Castañón (2013); Li and Shamma (2014) the informed agent observes
perfectly the state of the dynamic system, the other agent has no direct
observation of the system’s state; in Gensbittel and Renault (2015); Li et al
(2017) each agent observes perfectly part of the system’s state and the states
observed by the two agents are either independent or conditionally independent
(given the observed actions). The authors of Kartik and Nayyar (2021) consider
a general information structure where each agent has some private information
and the agents share some information about the dynamic system’s state and
their actions. The authors of Renault (2006); Cardaliaguet et al (2015);
Gensbittel and Renault (2015); Li et al (2017); Kartik and Nayyar (2021);
Zheng and Castañón (2013); Li and Shamma (2014) derive their results by taking
advantage of properties of zero-sum games such as the interchangeability of
equilibrium strategies and the unique value of the game. These properties do
not extend to non-zero sum games. We study a general class of stochastic
dynamic games that include zero-sum stochastic dynamic games with asymmetric
information as a special case. We consider general Markovian dynamics for the
underlying system in contrast to Renault (2006); Cardaliaguet et al (2015);
Gensbittel and Renault (2015); Li et al (2017); Zheng and Castañón (2013); Li
and Shamma (2014), where the system has the special structure described above.
We consider a general information structure that allows us to capture
scenarios with unobservable actions and imperfect observations that are not
captured by Renault (2006); Cardaliaguet et al (2015); Gensbittel and Renault
(2015); Li et al (2017); Zheng and Castañón (2013); Li and Shamma (2014).

The problems investigated in Tang et al (2022); Nayyar et al (2014); Gupta et
al (2014); Ouyang et al (2015, 2017); Vasal and Anastasopoulos (2016); Sinha
and Anastasopoulos (2016); Gupta et al (2016); Nayyar et al (2013a) are the
most closely related to our problem. The authors of Nayyar et al (2014); Gupta
et al (2014, 2016); Nayyar et al (2013a) study a class of dynamic games where
the agents’ common information based belief (defined in Nayyar et al (2014))
is independent of their strategies, that is, there is no signaling among them.
This property allows them to apply ideas from the common information approach
developed in Nayyar et al (2011, 2013b), and define an equivalent dynamic game
with symmetric information among fictitious agents. Consequently, they
characterize a class of equilibria for dynamic games called Common Information
based Markov Perfect Equilibria.

Our results are different from those in Nayyar et al (2014); Gupta et al
(2014, 2016); Nayyar et al (2013a) in two aspects. First, we consider a
general class of dynamic games where the agents’ CIB beliefs are strategy-
dependent, thus, signaling is present. Second, the proposed approach in Nayyar
et al (2014); Gupta et al (2014, 2016); Nayyar et al (2013a) requires the
agents to keep track of all of their private information over time. We propose
an approach to effectively compress the agents’ private information, and
consequently, reduce the number of variables which the agents need to form CIB
beliefs.

The authors of Tang et al (2022); Ouyang et al (2015, 2017); Vasal and
Anastasopoulos (2016); Sinha and Anastasopoulos (2016) study a class of
dynamic games with asymmetric information where signaling occurs. When the
horizon in finite, the authors of Ouyang et al (2015, 2017) introduce the
notion of Common Information Based Perfect Bayesian Equilibrium, and provide a
sequential decomposition of the game over time. The authors of Vasal and
Anastasopoulos (2016); Sinha and Anastasopoulos (2016) extend the results of
Ouyang et al (2015, 2017) to finite horizon Linear-Quadratic-Gaussian (LQG)
dynamic games and infinite horizon dynamic games, respectively.

The work of Tang et al (2022) extends the model of Ouyang et al (2017) to
games among teams of agents. Each agent has his own private information which
he shares with the members of his own team with delay $d$; teams also have
common information. The authors of Tang et al (2022) consider two classes of
strategies: sufficient private information based (SPIB) strategies, which only
compress private information, and sufficient private and common information
based (SPCIB) strategies, which compress both common and private information.
They show that SPIB-strategy-based BNE exist and the set of payoff profiles of
such equilibria is the same as the set of all BNE. They develop a backward
inductive sequential procedure, whose solution, if it exists, provides a SPCIB
BNE, and identify instances which guarantee the existence of SPCIB BNE. The
class of dynamic games studied in Tang et al (2022); Ouyang et al (2015,
2017); Vasal and Anastasopoulos (2016); Sinha and Anastasopoulos (2016)
satisfy the following assumptions: (i) agents’ actions are observable (ii)
each agent has a perfect observation of his own local states/type (iii)
conditioned on the agents’ actions, the evolution of the local states are
independent. We relax assumptions (i)-(iii) of Tang et al (2022); Ouyang et al
(2015, 2017); Vasal and Anastasopoulos (2016); Sinha and Anastasopoulos
(2016), and study a general class of dynamic games with asymmetric
information, hidden actions, imperfect observations, and controlled and
coupled dynamics.

###  1.2 Contribution

We study/analyze, in discrete time, a general class of sequential stochastic
dynamic games with asymmetric information, where the underlying system is
dynamic, the information structure is non-classical, at each time instant the
agents have private and common information and their actions are hidden (each
agent’s actions are not directly observable by the other agents). Our key
contribution is a methodology for the discovery of Bayesian Nash Equilibrium
(BNE) strategy profiles that are based on the agents’ compressed private and
common information and can be determined sequentially in time moving
backwards, if each step of this backward procedure has a solution. We present
an example where such a BNE strategy profile exists. We show that our
methodology works also for the case where the agents have no common
observations and their actions are hidden.

###  1.3 Organization

The rest of the paper is organized as follows: We present the game’s model
along with the equilibrium concept in Section 2. We state our objective and
present the methodology that achieves it in Section 3. In Section 4 we first
introduce compressed versions of the agents’ private and common information
that are sufficient for decision making purposes; then we define Sufficient
Information Based (SIB) strategies that are based on the agents’ compressed
information. In Section 5 we first introduce Sufficient Information Based
Bayesian Nash Equilibrium (SIB-BNE); then we present a sequential
decomposition of the game, that is, a backward inductive procedure that
determines SIB-BNE if each step of this procedure has a solution. In Section 6
we present an example that highlights our solution methodology and where a
SIB-BNE exists. In Section 7 we show that our solution methodology works for
stochastic dynamic games where the agents have no common observations and each
agent’s actions are part of his private information. The comparison of the
definitions of compressed private information as it appears in this paper and
in Tavafoghi et al (2022), along with some of the technical details related to
the existence of SIB-BNE for the example of Section 6 are presented in the
Appendices.

##  2 Model

We present our model for dynamic decision problems with strategic agents
(dynamic games) below; this model is an analogue to the model of Tavafoghi et
al (2022) for dynamic decision problems with non-strategic agents.

###  2.1 System Dynamics

There are $N$ strategic agents who live in a dynamic Markovian world over
horizon $\mathcal{T}:=\\{1,2,...,T\\}$, $T<\infty$. Let
$X_{t}\in\mathcal{X}_{t}$ denote the state of the world at $t\in\mathcal{T}$.
At time $t$, each agent, indexed by $i\in\mathcal{N}:=\\{1,2,...,N\\}$,
chooses an action $a^{i}_{t}\in\mathcal{A}^{i}_{t}$, where
$\mathcal{A}^{i}_{t}$ denotes the set of available actions to him at $t$.
Given the collective action profile $A_{t}:=(A_{t}^{1},...,A_{t}^{N})$, the
state of the world evolves according to the following stochastic dynamic
equation,

$\displaystyle X_{t+1}=f_{t}(X_{t},A_{t},W_{t}^{x}),\vspace*{-2pt}$ (1)

where $W_{1:T-1}^{x}$ is a sequence of independent random variables. The
initial state $X_{1}$ is a random variable that has a probability distribution
$\mu_{0}\in\Delta(\mathcal{X}_{1})$.

At every time $t\in\mathcal{T}$, before taking an action, agent $i$ receives a
noisy private observation $Y_{t}^{i}\in\mathcal{Y}_{t}^{i}$ of the current
state of the world $X_{t}$ and the action profile $A_{t-1}$, given by

$\displaystyle Y_{t}^{i}=O_{t}^{i}(X_{t},A_{t-1},W_{t}^{i}),\vspace*{-2pt}$
(2)

where $W_{1:T}^{i}$, $i\in\mathcal{N}$, are sequences of independent random
variables. Moreover, at every $t\in\mathcal{T}$, all agents receive a common
observation $Z_{t}\in\mathcal{Z}_{t}$ of the current state of the world
$X_{t}$ and the action profile $A_{t-1}$, given by

$\displaystyle Z_{t}=O_{t}^{c}(X_{t},A_{t-1},W_{t}^{c}),\vspace*{-3pt}$ (3)

where $W_{1:T}^{c}$, is a sequence of independent random variables. We assume
that the random variables $X_{1}$, $W_{1:T-1}^{x}$, $W_{1:T}^{c}$, and
$W_{1:T}^{i}$, $i\in\mathcal{N}$ are mutually independent.

To avoid measure-theoretic technical difficulties and for clarity and
convenience of exposition, we assume that all the random variables take values
in finite sets.

######  Assumption 1.

(finite game) The sets $\mathcal{N}$, $\mathcal{X}_{t}$, $\mathcal{Z}_{t}$,
$\mathcal{Y}_{t}^{i}$, $\mathcal{A}_{t}^{i}$, $i\in\mathcal{N}$, are finite.

###  2.2  Information Structure

Let $H_{t}$ denote the aggregate information of all agents at time $t$.
Assuming that agents have perfect recall, we have
$H_{t}=\\{Z_{1:t},Y_{1:t}^{1:N},A_{1:t-1}^{1:N}\\}$, i.e. $H_{t}$ denotes the
set of all agents’ past and present observations and all agents’ past actions.
The set of all possible realizations of the agents’ aggregate information is
given by $\mathcal{H}_{t}:=\prod_{\tau\leq
t}\mathcal{Z}_{\tau}\times\prod_{i\in\mathcal{N}}\prod_{\tau\leq
t}\mathcal{Y}_{\tau}^{i}\times\prod_{i\in\mathcal{N}}\prod_{\tau<t}\mathcal{A}_{\tau}^{i}$.

At time $t\in\mathcal{T}$, the aggregate information $H_{t}$ is not fully
known to all agents. Let $C_{t}:=\\{Z_{1:t}\\}\in\mathcal{C}_{t}$ denote the
agents’ common information about $H_{t}$ and
$P_{t}^{i}:=\\{Y_{1:t}^{i},A_{1:t-1}^{i}\\}\backslash
C_{t}\in\mathcal{P}_{t}^{i}$ denote agent $i$’s private information about
$H_{t}$, where $\mathcal{P}_{t}^{i}$ and $\mathcal{C}_{t}$ denote the set of
all possible realizations of agent $i$’s private and common information at
time $t$, respectively. We assume that observations $Y_{\tau}^{i}$,
$\tau\in\\{1,2...,t\\}$, and actions $A_{\tau}^{i}$,
$\tau\in\\{1,2...,t-1\\}$, are known to agent $i$ but are not necessarily
fully known to all other agents, denoted by $-i$, at $t\in\mathcal{T}$.
Therefore, we have $P_{t}^{i}\subseteq\\{Y_{1:t}^{i},A_{1:t-1}^{i}\\}$ for all
$i\in\mathcal{N}$, and
$H_{t}=\left(\bigcup_{i\in\mathcal{N}}P_{t}^{i}\right)\cup C_{t}$ for all
$t\in\mathcal{T}$. As such, $\left\\{C_{t},P_{t}^{i},i\in\mathcal{N}\right\\}$
form a partition of $\mathcal{H}_{t}$ at every time $t\in\mathcal{T}$. In
Section 2.5, we discuss several instances of information structures that can
be captured as special cases of our model.

###  2.3 Strategies and Utilities:

Let $H_{t}^{i}:=\\{C_{t},P_{t}^{i}\\}\in\mathcal{H}_{t}^{i}$ denote the
information available to agent $i$ at $t$, where $\mathcal{H}_{t}^{i}$ denote
the set of all possible realizations of agent $i$’s information at $t$. Agent
$i$’s behavioral strategy at $t$, denoted by $g_{t}^{i}$, is defined by

$\displaystyle
g^{i}_{t}:\mathcal{H}_{t}^{i}\rightarrow\Delta(\mathcal{A}_{t}^{i})$ (4)

where $\Delta(\mathcal{A}_{t}^{i})$ is the set of Probability Mass Functions
(PMFs) on $\mathcal{A}_{t}^{i}$. We denote by

$\displaystyle g^{i}:=(g^{i}_{1},g^{i}_{2},\ldots,g^{i}_{T})$ (5)

a strategy of agent $i$; $g^{i}\in\mathcal{G}^{i}$, where $\mathcal{G}^{i}$ is
the set of admissible strategies described by (4)-(5). We denote a strategy
profile $g$ by

$\displaystyle g:=(g^{1},g^{2},\ldots,g^{N})$ (6)

$g\in\mathcal{G}$, where $\mathcal{G}$ is the set of admissible strategy
profiles described by (4)-(6). We denote by

$\displaystyle g^{-i}:=(g^{1},\ldots,g^{i-1},g^{i+1},\ldots,g^{N})$ (7)

Agent $i$’s instantaneous utility at $t$ depends on the system state $X_{t}$
and the collective action profile $A_{t}$, and is given by
$u_{t}^{i}(X_{t},A_{t})$. Agent $i$’s total utility over horizon
$\mathcal{T}$, is given by,

$\displaystyle
U^{i}(X_{1:T},A_{1:T})=\sum_{t\in\mathcal{T}}u_{t}^{i}(X_{t},A_{t}).\vspace*{-2pt}$
(8)

###  2.4 Equilibrium Concept:

We consider Bayesian Nash Equilibrium (BNE) as the solution concept (Fudenberg
and Tirole, 1991). A strategy profile $g^{*}=(g^{*1},g^{*2},\ldots,g^{*N})$ is
a BNE if for all $i\in\mathcal{N}$

$\displaystyle\mathbb{E}^{g^{*}}\\{U^{i}(X_{1:T},A_{1:T})\\}\geq\mathbb{E}^{g^{*-i},\hat{g}^{i}}\\{U^{i}(X_{1:T},A_{1:T})\\},\quad\forall\hat{g}^{i}\in\mathcal{G}^{i}.$
(9)

###  2.5 Special Cases

We discuss several instances of dynamic games with asymmetric information that
are special cases of the general model described above.

1) Nested information structure: Consider a two-player game with one informed
player and one uninformed player and general Markovian dynamics. At every time
$t\in\mathcal{T}$, the informed player makes a private perfect observation of
the state $X_{t}$, i.e. $Y_{t}^{1}=X_{t}$. The uninformed player does not have
any observation of the state $X_{t}$. Both the informed and uninformed players
observe each others’ actions, i.e. $Z_{t}=\\{A_{t-1}\\}$. Therefore, we have
$P_{t}^{1}=\\{X_{1:t}\\}$, $P_{t}^{2}=\emptyset$, and
$C_{t}=\\{A_{1:t-1}^{1},A_{1:t-1}^{2}\\}$ for all $t\in\mathcal{T}$. The above
nested information structure corresponds to dynamic games considered in
Renault (2006); Cardaliaguet et al (2015); Renault (2012); Li and Shamma
(2014, 2017); Zheng and Castañón (2013), where in Renault (2012); Li and
Shamma (2017) the state $X_{t}$ is static.

2) Delayed sharing information structure: Consider a $N$-player game with
observable actions where agents observe each others’ observations with
$d$-step delay. That is, $P_{t}^{i}=\\{Y_{t-d+1:t}^{i}\\}$ and
$C_{t}=\\{Y_{1:t-d},A_{1:t-1}\\}$. We note that in our model we assume that
the agents’ common observation $Z_{t}$ at $t$ is only a function of $X_{t}$
and and $A_{t-1}$. Therefore, to describe the game with delayed sharing
information structure within the context of our model we need to augment our
state space to include the agents’ last $d$ observations as part of the
augmented state. Define
$\tilde{X}_{t}:=\\{X_{t},M^{1}_{t},M^{2}_{t},...,M^{d}_{t}\\}$ as the
augmented system state where
$M_{t}^{i}:=\\{A_{t-i},Y_{t-i}\\}\in\mathcal{A}_{t-i}\times\mathcal{Y}_{t-i}$,
$i\in\mathcal{N}$; that is, $M_{t}^{i}$ serves as a temporal memory for the
agents’ observation $Y_{t-i}$ at $t-i$. Then, we have
$\tilde{X}_{t+1}=\\{X_{t+1},M_{t+1}^{1},M_{t+1}^{2},...,M_{t+1}^{d}\\}=\\{f_{t}(X_{t},A_{t},W_{t}^{x}),(Y_{t}),M_{t}^{1},...,M_{t}^{d-1}\\}$
and $Z_{t}=\\{M_{t}^{d},A_{t-1}\\}=\\{Y_{t-d},A_{t-1}\\}$.

The above environment captures a connection between the symmetric information
structure and asymmetric information structure. The information asymmetry
among the agents increases as $d$ increases. The above delayed sharing
information structure corresponds to the dynamic game considered in Tavafoghi
et al (2016).

3) Perfectly controlled dynamics with hidden actions: Consider a $N$-player
game where the state $X_{t}:=(X_{t}^{1},X_{t}^{2},...,X_{t}^{N})$ has $N$
components. Agent $i$, $i\in\mathcal{N}$, perfectly controls $X_{t}^{i}$, i.e.
$X_{t+1}^{i}=A_{t}^{i}$. Agent $i$’s actions $A_{t}^{i}$, $t\in\mathcal{T}$,
are not observable by all other agents $-i$. Every agent $i$,
$i\in\mathcal{N}$, makes a noisy private observation
$Y_{i}^{t}(X_{t},W_{t}^{i})$ of the system state at $t\in\mathcal{T}$.
Therefore, we have $P_{t}^{i}:=\\{A_{1:t},Y_{1:t}^{i}\\}$, $C_{t}=\emptyset$.

##  3 Objective and Methodology

###  3.1 Objective

Our objective is twofold: (i) To determine BNE strategy profiles that are
based on compressed versions of the agents’ private and common information.
(ii) To compute the above-mentioned strategy profiles by a sequential
decomposition of the game, that is, by a backward inductive sequential
procedure that identifies an equilibrium strategy profile when every step of
the procedure has a solution.

###  3.2 Methodology

We present a methodology that achieves the above-state objective and proceeds
as follows:

  * •

Step 1. We determine a mutually consistent compression of the agents’ private
information that is sufficient for decision-making purposes (such a mutually
consistent compression may not be unique). Based on this compression we
introduce the Sufficient Private Information Based (SPIB) belief system.

  * •

Step 2. Based on the result of Step 1, we determine a compression of the
agents’ common information that is sufficient for decision-making purposes by
defining the Common Information Based (CIB) belief system. The CIB belief
system ensures that at each time instant each agent’s CIB belief is consistent
with his SPIB belief even when the agent deviates from his equilibrium
strategy and plays an arbitrary strategy. Such a consistency implies that each
agent forms his own CIB belief system, and each agent’s CIB belief system is
common knowledge among all agents.

  * •

Step 3. Based on the compression of the agents’ private and common information
we introduce Sufficient Information Based (SIB) strategies for each agent
(i.e., strategies that depend at each time on the agent’s sufficient private
information and the CIB belief system) and SIB BNE. We show that SIB
strategies satisfy a key closedness of best response property. Based on this
property we provide a sequential decomposition of the game, that is, a
backward inductive sequential procedure that determines a SIB BNE if each step
of the procedure has a solution.

  * •

Step 4. We provide an example of a stochastic dynamic game with asymmetric
information and hidden/unobservable actions where a SIB BNE exists.

##  4 Compression of Private and Common Information

In Section 4.1 we characterize/determine mutually consistent compressions of
all agents’ private information that are sufficient for decision-making
purposes. In Section 4.2 we introduce the common information based belief, a
compressed version of the agents’ common information, that is sufficient for
decision making purposes.

###  4.1 Sufficient private information (Step 1)

We present/consider a compression of the agents’ private information that is
done in a mutually consistent manner so that the compressed information is
sufficient for decision making purposes.

######  Definition 1 (Sufficient private information).

We say that $S^{i}_{t},i=1,\ldots,N$, is sufficient private information for
the agents if

  1. (i)

$S^{i}_{t}$ is a function of $H^{i}_{t}$ such that
$S^{i}_{t}=\zeta^{i}_{t}(H^{i}_{t})$ for some commonly known functions
$\zeta^{i}_{t},i=1,2,\ldots,N$.

  2. (ii)

$S^{i}_{t}$ can be sequentially updated as
$S^{i}_{t}=\phi^{i}_{t}(S^{i}_{t-1},Y^{i}_{t},Z_{t},A^{i}_{t-1})$ using some
commonly known functions $\phi^{i}_{t},i=1,2,\ldots,N$.

  3. (iii)

For any realization $x_{t},p^{-i}_{t},p^{i}_{t},c_{t}$, and the corresponding
$s^{-i}_{t}=\zeta^{-i}_{t}(p^{-i}_{t},c_{t})$ and
$s^{i}_{t}=\zeta^{i}_{t}(p^{i}_{t},c_{t})$, and any strategy profile $g$,
where $g_{t}^{i}:\mathcal{S}_{t}^{i}\times
C_{t}\rightarrow\Delta(\mathcal{A}_{t}^{i}),\forall i,\forall t$, such that
$\operatorname{\mathbb{P}}^{g}(p^{i}_{t},c_{t})>0$,

$\displaystyle\operatorname{\mathbb{P}}^{g}(x_{t},s^{-i}_{t}\mid
s^{i}_{t},c_{t})=\operatorname{\mathbb{P}}^{g}(x_{t},s^{-i}_{t}\mid
p^{i}_{t},c_{t})$ (10)

######  Remark 1.

A similar definition of sufficient private information for dynamic teams
appears in (Tavafoghi et al, 2022, Definition 2). This definition is slightly
different from Definition 1 above because the objectives in Tavafoghi et al
(2022) and this paper are different. In Appendix .1 we show that sufficient
private information satisfying Definition 1 may violate condition (ii) of
Definition 2 in Tavafoghi et al (2022). In Tavafoghi et al (2022) the
compression of private (and common) information must entail no loss in
performance, that is, we must be able to determine globally optimal team
strategy profiles that are based on compressed private and common information.
In this paper the goal is to determine BNE strategy profiles that are based on
compressed information and be sequentially computed (if such BNE strategy
profiles exist). We are not concerned about the equilibria we may lose when we
compress information; therefore, we don’t need condition (ii) of Definition 2
in Tavafoghi et al (2022).

Definition 1 characterizes a set of compressions for agents’ private
information. In the following, we show the set of sufficient private
information $S_{t}^{i}$, $i\in\mathcal{N}$, $t\in\mathcal{N}$, is rich enough
to form belief systems on information sets of realizations with positive or
zero probability. Let $\tilde{g}^{i}$ denote the uniform strategy that assigns
equal probability to every action of agent $i\in\mathcal{N}$. Below we show
that the policy-independence property of belief (Tavafoghi et al, 2022,
Theorem 1) for agent $i$ is still true when the private information
$p_{t}^{i}$ is replaced with the sufficient private information $s_{t}^{i}$.
That is,
$\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(x_{t},x^{-i}_{t}\mid
s^{i}_{t},c_{t})$ constructed by $(\tilde{g}^{i},g^{-i})$ captures agent $i$’s
belief based on $h^{i}_{t}$ even when he plays an arbitrary strategy
$\hat{g}^{i}$, not necessarily the same as $g^{i}$ or $\tilde{g}^{i}$,
provided that agents $-i$ play $g^{-i}$.

######  Lemma 1.

For $h^{i}_{t}$ such that
$\operatorname{\mathbb{P}}^{\hat{g}^{i},g^{-i}}(h^{i}_{t})>0$, we have
$\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(h^{i}_{t})>0$ and

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},g^{-i}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t})=\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t})=\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(x_{t},s^{-i}_{t}\mid
s^{i}_{t},c_{t}).$ (11)

Note that
$\operatorname{\mathbb{P}}^{\tilde{g}^{i}}(a^{i}_{t})=1/|\mathcal{A}^{i}_{t}|$,
so $\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(h^{i}_{t})>0$ given that
$\operatorname{\mathbb{P}}^{g}(h^{i}_{t})>0$. Then from part (i) of the
definition of sufficient private information and part (i) of Theorem 1 in
Tavafoghi et al (2022) we have

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},g^{-i}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t})=$
$\displaystyle\sum_{h^{-i}_{t}:\zeta^{-i}_{t}(h^{-i}_{t})=s^{-i}_{t}}\operatorname{\mathbb{P}}^{\hat{g}^{i},g^{-i}}(x_{t},h^{-i}_{t}\mid
h^{i}_{t})$ $\displaystyle=$
$\displaystyle\sum_{h^{-i}_{t}:\zeta^{-i}_{t}(h^{-i}_{t})=s^{-i}_{t}}\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(x_{t},h_{t}^{-i}\mid
h_{t}^{i})$ $\displaystyle=$
$\displaystyle\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t}).$ (12)

Furthermore, from condition (iii) of the definition of sufficient private
information we have

$\displaystyle\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t})=\operatorname{\mathbb{P}}^{\tilde{g}^{i},g^{-i}}(x_{t},s^{-i}_{t}\mid
s^{i}_{t},c_{t}).$ (13)

###  4.2 CIB Belief System (Step 2)

Given the compressed private information, we next compress the agents’ common
information in the form of a belief system. We call such a compressed belief
system the Common Information Based (CIB) belief system. Similar to Tang et al
(2022); Ouyang et al (2017), the CIB belief system is sufficient for decision-
making if it is common knowledge among all agents, and every agent $i$ can
compute his belief about the system state and the other agents’ sufficient
private information using the CIB belief system and his compressed private
information. More specifically, agent $i$ should be able to compute
$\operatorname{\mathbb{P}}^{\hat{g}^{i},g^{-i}}(x_{t},s_{t}\mid h^{i}_{t})$
using the CIB belief system and his sufficient private information $s^{i}_{t}$
whenever other agents follow the strategy profile $g^{-i}$ and agent $i$ plays
an arbitrary strategy $\hat{g}^{i}$.

To determine a CIB belief system that satisfies the above sufficiency
requirement we proceed as follows. We first define $N$ CIB belief systems
$\Pi^{\psi}:=\\{\Pi^{\psi,1},\Pi^{\psi,2},\ldots,\Pi^{\psi,N}\\}$, one for
each agent (Definition 2 below). Each belief system $\Pi^{\psi,i}$ consists of
a sequence of PMFs on $\mathcal{X}_{t}\times\mathcal{S}_{t}$ that are
sequentially updated according to an update rule
$\psi=(\psi^{1},\psi^{2},\ldots,\psi^{N})$ that is common knowledge among the
agents; for each realization $c_{t}$ of the common information available at
$t$, $\pi^{\psi,i}_{t}$ describes the belief on
$\mathcal{X}_{t}\times\mathcal{S}_{t}$ based on $c_{t}$ from agent $i$’s point
of view. We want $\pi^{\psi,i}_{t}$, combined with $s^{i}_{t}$, to enable
agent $i$ to form his own sufficient information-based private belief (given
by $\operatorname{\mathbb{P}}^{\hat{g}^{i},g^{*-i}}(x_{t},s_{t}\mid
s^{i}_{t},c_{t})$) about the current status of the game. Furthermore, we want
the CIB belief system to capture the current status of the game when agents
utilize strategies based on $(S_{t},\Pi^{\psi}_{t})$. For that matter, we
define the notion/concept of Sufficient Information Based (SIB) strategy
profile $\sigma:=(\sigma^{i},i\in\mathcal{N})$,
$\sigma^{i}:=(\sigma^{i}_{t},t\in\mathcal{T}),i\in\mathcal{N}$. Each component
$\sigma^{i}_{t}$ of $\sigma$ is a function of $s^{i}_{t}$, agent $i$’s
sufficient private information at $t$, and
$\pi^{\psi}_{t}=(\pi^{\psi,i}_{t},i\in\mathcal{N})$ (see Definition 3 below).
Using the $N$ CIB belief systems and the SIB strategy profile $\sigma$ we
define update equations for each $\pi^{\psi,i}_{t}$ so that each
$\pi^{\psi,i}_{t}$ is consistent with $s^{i}_{t}$ and with agent $i$’s
sufficient private information-based belief
$\operatorname{\mathbb{P}}^{\hat{g}^{i},g^{*-i}}(x_{t},s_{t}\mid
s^{i}_{t},c_{t})$, defined in Section 4.1 (Definition 1), and each
$\pi^{\psi,i}_{t}$ is common knowledge among all agents (see Definition 4
below). We proceed with the (formal) definitions.

######  Definition 2 (Common information based (CIB) belief system).

Given a sequence of update functions
$\psi=\\{\psi^{i}_{t},i\in\mathcal{N},t\in\mathcal{T}\\}$ that are common
knowledge among the $N$ agents, sequentially define

$\displaystyle\Pi_{t}^{\psi,i}=\psi_{t}^{i}(\Pi^{\psi}_{t-1},Z_{t}),i\in\mathcal{N},t\in\mathcal{T}$
(14)

where

$\displaystyle\Pi^{\psi}_{t}:=\left[\begin{array}[]{c}\Pi^{\psi,1}_{t}\\\
\vdots\\\ \Pi^{\psi,N}_{t}\end{array}\right],t\in\mathcal{T}$ (18)
$\displaystyle\Pi^{\psi}_{0}:=\left[\begin{array}[]{c}\mu_{0}\\\ \vdots\\\
\mu_{0}\end{array}\right]$ (22)

The sequence
$\Pi^{\psi}_{1:T}=(\Pi^{\psi}_{1},\Pi^{\psi}_{2},\ldots,\Pi^{\psi}_{T})$
defines a CIB belief system; $\Pi^{\psi,i}_{t}$ denotes the CIB belief over
$\mathcal{X}_{t}\times\mathcal{S}_{t}$ based on $C_{t}$ from agent $i$’s point
of view.

######  Definition 3 (SIB strategy).

Given a CIB belief system $\Pi^{\psi}_{1:T}$, we define a Sufficient
Information Based (SIB) strategy profile
$\sigma:=(\sigma^{1},\sigma^{2},\ldots,\sigma^{N})$,
$\sigma^{i}:=(\sigma^{i}_{1},\sigma^{i}_{2},\ldots,\sigma^{i}_{T})$ by the
maps

$\displaystyle\sigma_{t}^{i}:\mathcal{S}^{i}_{t}\times[\Delta(\mathcal{X}_{t}\times\mathcal{S}_{t})]^{N}\rightarrow\Delta(\mathcal{A}_{t}),t=1,2,\ldots,i=1,2,\ldots,N.$
(23)

Based on Definitions 2 and 3 we present a set of conditions that an individual
CIB belief system $(\Pi^{\psi,i}_{t},t\in\mathcal{T})$ must satisfy so as to
ensure that each agent $i$ can form his own (private) belief about the current
status of the game, given by $(X_{t},S_{t})$, using $\Pi^{\psi}_{t}$ and
$S^{i}_{t}$ when all other agents $-i$ employ SIB strategies $\sigma^{-i}$.
This set of conditions describe a sequential update rule of
$\Pi^{\psi,i}_{t}$; the update rule depends on whether or not the (new) common
observation at $t$ is feasible under the agents’ strategies.

######  Definition 4 (Consistent CIB belief system).

Consider a SIB strategy profile $\sigma$. Let
$F_{t}^{i}(x_{t+1},s_{t+1},z_{t+1})(\pi^{\psi}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})$ denote the CIB belief about
$(x_{t+1},s_{t+1},z_{t+1})$ constructed recursively by assuming that (i)
$(x_{t},s_{t})$ is distributed according to $\pi_{t}^{\psi,i}$ (ii) agent $i$
employs the uniform strategy $\tilde{g}^{i}$ at $t$ (i.e., the strategy that
chooses every action $a^{i}_{t}\in\mathcal{A}^{i}_{t}$ with equal
probability), and (iii) agent $-i$ plays according $\sigma_{t}^{-i}$. That is,

$\displaystyle F_{0}^{i}(x_{1},s_{1},z_{1})=$
$\displaystyle\sum_{y_{1}}\Bigg{[}\mathbb{P}\\{z_{1},y_{1}\mid
x_{1}\\}\mu_{0}(x_{1})\left(\prod_{j}\mathbbm{1}\\{s_{1}^{j}=\phi_{1}^{j}(z_{1},y_{1}^{j})\\}\right)\Bigg{]}$
(24)

at $t=1$, and for $t\geq 1$.

$\displaystyle F_{t}^{i}(x_{t+1},s_{t+1},z_{t+1})(\pi^{\psi}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})$ $\displaystyle=$
$\displaystyle\sum_{y_{t+1},x_{t},s_{t},a_{t}}\Bigg{[}\mathbb{P}\\{z_{t+1},y_{t+1},x_{t+1}\mid
x_{t},a_{t}\\}\left(\prod_{j}\mathbbm{1}\\{s_{t+1}^{j}=\phi_{t+1}^{j}(s_{t}^{j},z_{t+1},y_{t+1}^{j},a_{t}^{j})\\}\right)$
$\displaystyle\hskip 50.0pt\left(\frac{1}{\mid A_{t}^{i}\mid}\prod_{j\neq
i}\sigma^{j}_{t}(a^{j}_{t})(\pi^{\psi}_{t},s_{t}^{j})\right)\pi_{t}^{\psi,i}(x_{t},s_{t})\Bigg{]}$
(25)

We define the update rule
$\psi^{\sigma}=(\psi^{\sigma,i}_{t},i\in\mathcal{N},t\in\mathcal{T})$ and the
corresponding CIB belief system $\Pi^{\psi^{\sigma}}_{1:T}$ as follows. At any
$t$

  1. (i)

If
$\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1},z_{t+1})(\pi^{\psi^{\sigma}}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})>0$ (i.e. the new common observation $z_{t+1}$ is
feasible from the agent $i$’s point of view), then
$\pi^{\psi^{\sigma},i}_{t+1}$ can be updated recursively as

$\displaystyle\pi_{t+1}^{\psi^{\sigma},i}(x_{t+1},s_{t+1})=\frac{F_{t}^{i}(x_{t+1},s_{t+1},z_{t+1})(\pi^{\psi^{\sigma}}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})}{\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1},z_{t+1})(\pi_{t}^{\psi^{\sigma}}\mathchar
24635\relax\;\sigma^{-i}_{t})},$ (26)

via Bayes rule.

  2. (ii)

If
$\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1},z_{t+1})(\pi^{\psi^{\sigma}}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})=0$ (i.e. the new common observation $z_{t+1}$ is
infeasible from the agent $i$’s point of view), then the update rule is

$\displaystyle\pi_{t+1}^{\psi^{\sigma},i}(x_{t+1},s_{t+1})=\frac{1}{\left|\mathcal{X}_{t+1}\times\mathcal{S}_{t+1}\right|}.$
(27)

Based on (26) and (27) we can write

$\displaystyle\Pi_{t+1}^{\psi^{\sigma},i}=\psi_{t+1}^{\sigma,i}(\Pi_{t}^{\psi^{\sigma}},Z_{t+1}).$
(28)
$\displaystyle\Pi_{t+1}^{\psi^{\sigma}}=\psi^{\sigma}_{t+1}(\Pi_{t}^{\psi^{\sigma}},Z_{t+1}).$
(29)

Furthermore, for all $i\in\mathcal{N}$, each agent can determine if
$\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1},z_{t+1})(\pi_{t}^{\sigma^{\psi}}\mathchar
24635\relax\;\sigma^{-i}_{t})$ is positive or zero; thus each agent knows how
agent $i$ computes $\pi^{\psi^{\sigma},i}_{t+1}$ from
$\sigma^{i}_{t},z_{t+1},\sigma^{-i}_{t}$ and $\psi^{\sigma}$. Therefore,
$\pi^{\psi^{\sigma},i}_{t}$ (hence $\pi^{\psi^{\sigma}}_{t}$) is common
knowledge among all agents. We call $\Pi^{\psi^{\sigma}}_{1:T}$ the CIB belief
system consistent with the SIB strategy profile $\sigma$.

######  Remark 2.

Since the sufficient private information is a function of the agent’s
available information, a SIB strategy $\sigma^{i}_{t}$ corresponds to a
strategy $g_{t}^{i,\sigma}$ given by
$g_{t}^{i,\sigma}(h^{i}_{t}):=\sigma^{i}_{t}(\zeta^{i}_{t}(h^{i}_{t}),\pi_{t}^{\psi^{\sigma}})$.
Therefore, in the rest of the paper we use the following convention:
$\operatorname{\mathbb{P}}^{\sigma}(\cdot)=\operatorname{\mathbb{P}}^{g^{\sigma}}(\cdot)$
and
$\operatorname{\mathbb{E}}^{\sigma}[\cdot]=\operatorname{\mathbb{E}}^{g^{\sigma}}[\cdot]$.

######  Remark 3.

There are many alternative specifications of the update rule
$\psi^{\sigma}_{t},t\in\mathcal{T}$ defined by (28)-(29), that result in
consistent CIB belief systems, that is, CIB belief systems which ensure that
(i) agent $i$ can form his private belief over $(X_{t},S^{-i}_{t})$ by
incorporating his private sufficient information $S^{i}_{t}$ into his CIB
belief $\Pi^{\psi^{\sigma},i}_{t}$ given that agents $-i$ play according to
$\sigma^{-i}$, (ii) agent $i$’s private belief formed according to $i$ is
identical to the probability distribution over $(X_{t},S^{-i}_{t})$
conditional on his complete history $H^{i}_{t}$ even when he plays an
arbitrary strategy $\hat{g}^{i}$ different from $\sigma^{i}$. An example of
such an alternative update rule is described by (26) (Bayes’ rule) when
$\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1},z_{t+1})(\pi_{t}^{\psi^{\sigma}}\mathchar
24635\relax\;\sigma^{-i}_{t})>0$ and a arbitrary PMF
$\pi^{\psi^{\sigma},i}_{t+1}(\cdot,\cdot)$ on $X_{t+1}\times S_{t+1}$ when
$\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1},z_{t+1})(\pi_{t}^{\psi^{\sigma}}\mathchar
24635\relax\;\sigma^{-i}_{t})=0$.

Definition 4 ensures that agent $i$ can form his beliefs over
$(X_{t},S_{t}^{-i})$ by incorporating his sufficient private information
$S_{t}^{i}$ into his CIB belief $\Pi_{t}^{\psi^{\sigma},i}$ given that agents
$-i$ play according to $\sigma^{-i}$. Moreover, this belief is sufficient to
compute the probability distribution over $(X_{t},S_{t}^{-i})$ conditional on
his complete history $H_{t}^{i}$ even when he plays an arbitrary strategy
$\hat{g}^{i}$ different from $\sigma^{i}$. We formalize the above discussion
in Lemma 2 below, by using the notation
$\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(\cdot)$ to
indicate the belief resulting when agent $i$ plays $\hat{g}^{i}$ and agents
$-i$ play
$g^{-i,\sigma}(h^{-i}_{t})=\sigma^{-i}_{t}(\zeta^{-i}_{t}(h^{-i}_{t}),\pi_{t}^{\psi^{\sigma}})$
using the update rule $\psi^{\sigma}$ .

######  Lemma 2.

Consider a SIB strategy profile $\sigma$, along with an associated consistent
CIB belief system $\Pi_{t}^{\psi^{\sigma}}$. Suppose
$(x_{t},h^{i}_{t},h^{-i}_{t}$ is a realization with positive probability under
$(\hat{g}^{i},\sigma^{-i})$, where $\hat{g}^{i}$ denotes an arbitrary strategy
for agent $i$. Let $s^{i}_{t}=\zeta^{i}_{t}(h^{i}_{t})$ and
$s^{-i}_{t}=\zeta^{-i}_{t}(h^{-i}_{t})$ be the associated sufficient private
information. Then agent $i$’s belief at time $t$ can be computed using
$\pi_{t}^{\psi^{\sigma}}$ as

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t})=\frac{\pi^{\psi^{\sigma},i}_{t}(x_{t},s_{t})}{\sum_{s^{-i}_{t},x_{t}}\pi^{\psi^{\sigma},i}_{t}(x_{t},s^{i}_{t},s^{-i}_{t})}$
(30)

From Lemma 1 we have

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t})=\operatorname{\mathbb{P}}^{\tilde{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t},s^{-i}_{t}\mid
h^{i}_{t}).=\operatorname{\mathbb{P}}^{\tilde{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t},s^{-i}_{t}\mid
c_{t},s^{i}_{t}).$ (31)

By Bayes’ rule we obtain

$\displaystyle\operatorname{\mathbb{P}}^{\tilde{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t},s^{-i}_{t}\mid
c_{t},s^{i}_{t})=\frac{\operatorname{\mathbb{P}}^{\tilde{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t},s_{t}\mid
c_{t})}{\operatorname{\mathbb{P}}^{\tilde{g}^{i},\sigma^{-i},\psi^{\sigma}}(s^{i}_{t}\mid
c_{t})}=\frac{\pi^{\psi^{\sigma},i}_{t}(x_{t},s_{t})}{\sum_{s^{-i}_{t},x_{t}}\pi^{\psi^{\sigma},i}_{t}(x_{t},s^{i}_{t},s^{-i}_{t})}.$
(32)

Combination of (31) and (32) establishes the assertion of Lemma 2.

######  Remark 4.

Suppose $X_{t}=(X^{1}_{t},X^{2}_{t},.\ldots,X^{N}_{t})$ and we have the
conditional independence property, namely, that for any strategy profile $g$
$\operatorname{\mathbb{P}}^{g}(x_{t},s_{t}\mid
c_{t})=\prod_{i}\operatorname{\mathbb{P}}^{g^{i}}(x^{i}_{t},s^{i}_{t}\mid
c_{t})$. Then one can show for any $i$ that

$\displaystyle\pi^{\psi^{\sigma},i}_{t}(x_{t},s_{t})=\prod_{j}\pi^{\psi^{\sigma},i}(x^{j}_{t},s^{j}_{t})=\operatorname{\mathbb{P}}^{\tilde{g}^{i}_{t}}(x^{i}_{t},s^{i}_{t}\mid
c_{t})\prod_{j\neq
i}\operatorname{\mathbb{P}}^{\sigma^{j}}(x^{j}_{t},s^{j}_{t}\mid c_{t})$

Therefore, for settings with the conditional independence property as in Tang
et al (2022); Ouyang et al (2017), one can use the simplified beliefs
$\operatorname{\mathbb{P}}^{\tilde{g}^{i}_{t}}(x^{i}_{t},s^{i}_{t}\mid c_{t})$
and $\operatorname{\mathbb{P}}^{\sigma^{j}}(x^{j}_{t},s^{j}_{t}\mid c_{t})$ as
the compressed common information to compute the CIB belief
$\pi^{\psi^{\sigma},i}_{t}(x_{t},s_{t})$. The conditional independence among
the system components in the models of Tang et al (2022); Ouyang et al (2017)
could be lost when the agents’ actions are not observable.

##  5  Sequential decomposition (Step 3)

In this section we present a sequential decomposition of the game, that is, a
backward inductive sequential procedure that determines a Sufficient
Information Based Bayesian Nash Equilibrium (SIB-BNE), defined below, if each
step of this procedure has a solution. We proceed as follows. We first
establish a key closedness of best response property (Section 5.1); we use
this property to provide a sequential decomposition of the game (Section 5.2)

######  Definition 5 (SIB-BNE).

Consider a SIB strategy profile
$\sigma^{*}=(\sigma^{*1},\sigma^{*2},\ldots,\sigma^{*n})$ and its
corresponding consistent update rule $\psi^{\sigma^{*}}$. The SIB strategy
profile $\sigma^{*}$ is a SIB-BNE if it is a BNE of the dynamic game. That is,
for all $i\in\mathcal{N}$,

$\displaystyle\mathbb{E}^{\hat{g}^{i},\sigma^{*-i},\psi^{\sigma^{*}}}\\{U^{i}(X_{1:T},A_{1:T})\\}\leq\mathbb{E}^{\sigma^{*},\psi^{\sigma^{*}}}\\{U^{i}(X_{1:T},A_{1:T})\\},$
$\displaystyle\text{ for all strategies (not necessarily SIB strategies)
}\hat{g}^{i}.$ (33)

###  5.1 Closedness of best response

The key result of this subsection is presented in the following theorem.

######  Theorem 1.

Consider a fixed and known SIB strategy profile $\sigma$ and the corresponding
update rule $\psi^{\sigma}$. Suppose agents $-i$ use $\sigma^{-i}$ with
$\psi^{\sigma}$. Then, there exists a SIB strategy $\hat{\sigma}^{i}$ that
uses $\psi^{\sigma}$ and is a best response to $\sigma^{-i}$ with
$\psi^{\sigma}$.

The proof is based on Lemmas 3, 4, and 5 that we state and prove below.

######  Lemma 3.

Consider a SIB strategy profile $\sigma$ and the corresponding update rule
$\psi^{\sigma}$ along with the consistent CIB belief system
$\Pi_{1:T}^{\psi^{\sigma}}$.

If agents $-i$ play according to the SIB strategies $\sigma^{-i}$ and use the
update rule $\psi^{\sigma}$, the best response problem for agent $i$ is a
POMDP with state and observation processes

$\displaystyle\tilde{X}_{t}=(S_{t},\Pi_{t}^{\psi^{\sigma}},X_{t}),t\in\mathcal{T}$
(34) $\displaystyle\tilde{Y}_{t}=(Y^{i}_{t},Z_{t}),t\in\mathcal{T}$ (35)

respectively, and instantaneous utility

$\displaystyle\tilde{u}^{i}_{t}(\tilde{X}_{t},A^{i}_{t})=\sum_{a^{-i}_{t}}\big{(}\prod_{j\neq
i}\sigma^{j}_{t}(a^{j}_{t}\mid
S^{j}_{t},\Pi_{t}^{\psi^{\sigma}})\big{)}u^{i}_{t}(X_{t},a^{-i}_{t},A^{i}_{t}),t\in\mathcal{T}$
(36)

The assertion of Lemma 3 is a direct consequence of Lemmas 4 and 5.

######  Lemma 4.

Consider a SIB strategy profile $\sigma$ and the corresponding update rule
$\psi^{\sigma}$. Suppose agents $-i$ play according to the SIB strategies
$\sigma^{-i}$ using $\psi^{\sigma}$ and agent $i$ follows an arbitrary
strategy $\hat{g}^{i}$ (not necessarily a SIB strategy). Then

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(\tilde{x}_{t+1},\tilde{y}_{t+1}\mid\tilde{x}_{1:t},\tilde{y}_{1:t},a^{i}_{1:t})=\operatorname{\mathbb{P}}^{\hat{g}^{i}\sigma^{-i},\psi^{\sigma}}(\tilde{x}_{t+1},\tilde{y}_{t+1}\mid\tilde{x}_{t},a^{i}_{t})$
(37)

The probability for the next state and observation
$\tilde{x}_{t+1},\tilde{y}_{t+1}$ can be computed by

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(\tilde{x}_{t+1},\tilde{y}_{t+1}\mid\tilde{x}_{1:t},\tilde{y}_{1:t},a^{i}_{1:t})$
$\displaystyle=$
$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t+1},\pi^{\psi^{\sigma}}_{t+1},s_{t+1},y^{i}_{t+1},z_{t+1}\mid
x_{1:t},\pi^{\psi^{\sigma}}_{1:t},s_{1:t},y^{i}_{1:t},z_{1:t},a^{i}_{1:t})$
$\displaystyle=$
$\displaystyle\sum_{y^{-i}_{t+1},a^{-i}_{t}}\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(x_{t+1},\pi^{\psi^{\sigma}}_{t+1},s_{t+1},y_{t+1},z_{t+1},a^{-i}_{t}\mid
x_{1:t},\pi^{\psi^{\sigma}}_{1:t},s_{1:t},y^{i}_{1:t},z_{1:t},a^{i}_{1:t})$
$\displaystyle=$
$\displaystyle\sum_{y^{-i}_{t+1},a^{-i}_{t}}\big{(}\prod_{j}\mathds{1}(s^{j}_{t+1}=\phi^{j}_{t+1}(s^{j}_{t},y^{j}_{t+1},z_{t+1},a^{j}_{t}))\big{)}\mathbb{P}\\{z_{t+1},y_{t+1},x_{t+1}\mid
x_{t},a_{t}\\}$ $\displaystyle\hskip
56.9055pt\mathds{1}(\pi^{\psi^{\sigma}}_{t+1}=\psi^{\sigma}_{t+1}(\pi^{\psi^{\sigma}}_{t},z_{t+1}))\big{(}\prod_{j\neq
i}\sigma^{j}_{t}(a^{j}_{t}\mid s^{j}_{t},\pi^{\psi^{\sigma}}_{t})\big{)}$ (38)

where the last equality follows from the system dynamics, part (ii) of
Definition 1, Definition 4, and the form of SIB strategies of agents ${-i}$.
Since the right hand side of (38) depends only on $(\tilde{x}_{t},a^{i}_{t})$
we conclude that

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(\tilde{x}_{t+1},\tilde{y}_{t+1}\mid\tilde{x}_{1:t},\tilde{y}_{1:t},a^{i}_{1:t})=\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(\tilde{x}_{t+1},\tilde{y}_{t+1}\mid\tilde{x}_{t},a^{i}_{t})$
(39)

Lemma 4 shows that $\\{\tilde{X}_{t},\tilde{Y}_{t},t\in\mathcal{T}\\}$ is a
Markov process conditional on $\\{A^{i}_{t},t\in\mathcal{T}\\}$

######  Lemma 5.

Consider a SIB strategy profile $\sigma$ and the corresponding update rule
$\psi^{\sigma}$. Suppose agents $-i$ follow the SIB strategies $\sigma^{-i}$
using $\psi^{\sigma}$ and agent $i$ follows an arbitrary strategy
$\hat{g}^{i}$ (not necessarily a SIB strategy). Then there are utility
functions $\tilde{u}^{i}_{t}$ such that
$\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}[\tilde{u}^{i}_{t}(\tilde{X}_{t},A^{i}_{t})]=\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}[u^{i}_{t}(X_{t},A_{t})]$
for all $t\in\mathcal{T}$.

Recall that $\tilde{X}_{t}=(S_{t},\Pi^{\psi^{\sigma}}_{t},X_{t})$. Then

$\displaystyle\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}[u^{i}_{t}(X_{t},A_{t})]$
$\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}[u^{i}_{t}(X_{t},A^{-i}_{t},A^{i}_{t})]$
$\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}\big{[}\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}[u^{i}_{t}(X_{t},A^{-i}_{t},A^{i}_{t})\mid\tilde{X}_{t},A^{i}_{t}]\big{]}$
$\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}\big{[}\sum_{a^{-i}_{t}}\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(a^{-i}_{t}\mid
S_{t},\Pi^{\psi^{\sigma}}_{t},X_{t},A^{i}_{t})u^{i}_{t}(X_{t},a^{-i}_{t},A^{i}_{t})]\big{]}$
$\displaystyle=$
$\displaystyle\operatorname{\mathbb{E}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}\big{[}\sum_{a^{-i}_{t}}\big{(}\prod_{j\neq
i}\sigma^{j}_{t}(a^{j}_{t}\mid
S^{j}_{t},\Pi^{\psi^{\sigma}}_{t})\big{)}u^{i}_{t}(X_{t},a^{-i}_{t},A^{i}_{t})]\big{]}$
(40)

Therefore, we establish the claim of the lemma by defining

$\displaystyle\tilde{u}^{i}_{t}(\tilde{X}_{t},A^{i}_{t})=\sum_{a^{-i}_{t}}\big{(}\prod_{j\neq
i}\sigma^{j}_{t}(a^{j}_{t}\mid
S^{j}_{t},\Pi^{\psi^{\sigma}}_{t})\big{)}u^{i}_{t}(X_{t},a^{-i}_{t},A^{i}_{t})]$
(41)

[Proof of Theorem 1] From Lemma 3 we conclude that the best response of agent
$i$ to $\sigma^{-i}$ is a POMDP with state $\tilde{X}_{t}$. From the theory of
POMDP (Kumar and Varaiya, 1986, Chapter 6) we know that: (i) the belief on the
state $\tilde{X}_{t}=(S_{t},\Pi^{\psi^{\sigma}}_{t},X_{t})$ conditioned on
available information $h^{i}_{t}$ is an information state for the agent; (ii)
for each $t\in\mathcal{T}$ there exists an optimal strategy for agent $i$ that
is a function of the information state at $t$. We now prove that
$(S^{i}_{t},\Pi^{\psi^{\sigma}}_{t})$ is an information state for agent $i$ at
$t,t\in\mathcal{T}$.
We note that
$S^{i}_{t+1}=\phi^{i}_{t}(S^{i}_{t},Y^{i}_{t+1},Z_{t+1},A^{i}_{t})$ from part
(ii) of Definition 1, and
$\Pi^{\psi^{\sigma}}_{t+1}=\psi^{\sigma}_{t+1}(\Pi^{\psi^{\sigma}}_{t},Z_{t+1})$
from (29).

Thus, we only need to show that for any strategy $\hat{g}^{i}$ and any
realization $h^{i}_{t}$ such that
$\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(h^{i}_{t})>0$
the following equality is true:

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(s_{t},\pi^{\psi^{\sigma}}_{t},x_{t}\mid
h^{i}_{t})=\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(s_{t},\pi^{\psi^{\sigma}}_{t},x_{t}\mid
s^{i}_{t},\pi^{\psi^{\sigma}}_{t})$ (42)

For that matter, we note that $s^{i}_{t},\pi^{\psi^{\sigma}}_{t}$ are
perfectly known to agent $i$. Furthermore, from the definition of sufficient
private information and Lemma 2 we have

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(s^{-i}_{t},x_{t}\mid
h^{i}_{t})=\frac{\pi^{{\psi^{\sigma}},i}_{t}(s_{t},x_{t})}{\sum_{s^{-i}_{t},x_{t}}\pi^{{\psi^{\sigma}},i}_{t}(s^{i}_{t},s^{-i}_{t},x_{t})},$
(43)

which is a function of $(s^{i}_{t},\pi^{\psi^{\sigma}}_{t})$. Therefore,

$\displaystyle\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(s_{t},\pi^{\psi^{\sigma}}_{t},x_{t}\mid
h^{i}_{t})=\mathds{1}(s^{i}_{t}=\zeta^{i}_{t}(h^{i}_{t}))\mathds{1}(\pi^{\psi^{\sigma}}_{t}=\gamma^{\psi^{\sigma}}(h^{i}_{t}))\operatorname{\mathbb{P}}^{\hat{g}^{i},\sigma^{-i},\psi^{\sigma}}(s^{-i}_{t},x_{t}\mid
p^{i}_{t},c_{t})$ (44)

where
$\gamma^{\psi^{\sigma}}(h^{i}_{t})=\psi^{\sigma}_{t}(\psi^{\sigma}_{t-1},\cdots)$
is the composition of $\psi^{\sigma}$ from $1$ to $t$. Then, equation (42) is
true because of (43) and (44). Consequently,
$(S^{i}_{t},\Pi^{\psi^{\sigma}}_{t}),t\in\mathcal{T}$ is an information state
for the best response problem for agent $i$ and the assertion of Theorem 1 is
true.

As a result of Theorem 1, a definition of SIB BNE equivalent to Definition 5
is the following

######  Definition 6 (Equivalent definition of SIB BNE).

Consider a SIB strategy profile
$\sigma^{*}=(\sigma^{*1},\sigma^{*2},\ldots,\sigma^{*n})$ and its
corresponding consistent update rule $\psi^{\sigma^{*}}$. The SIB strategy
profile $\sigma^{*}$ is a SIB BNE if for all $i\in\mathcal{N}$,

$\displaystyle\mathbb{E}^{\sigma^{i},\sigma^{*-i},\psi^{\sigma^{*}}}\\{U^{i}(X_{1:T},A_{1:T})\\}\leq\mathbb{E}^{\sigma^{*},\psi^{\sigma^{*}}}\\{U^{i}(X_{1:T},A_{1:T})\\}$
(45)

for all $\sigma^{i}\in\Lambda^{i}$ where $\Lambda^{i}$ is the set of SIB
strategy profiles of agent $i$.

A consequence of Lemmas 3-5 and Theorem 1 is the following. Consider a SIB
strategy profile $\sigma$, the corresponding update rule $\psi^{\sigma}$ along
with the consistent CIB belief system $\Pi^{\psi^{\sigma}}_{1:T}$; if agents
$-i$ play according to $\sigma^{-i}$, then the best response of agent $i$
could be determined by the dynamic program

$\displaystyle\breve{V}_{T+1}^{i}(\cdot,\cdot)=0\text{ for all }i$ (46)
$\displaystyle\breve{V}^{i}_{t}(\pi^{\psi^{\sigma}}_{t},s^{i}_{t})=$
$\displaystyle\max_{\tilde{\sigma}^{i}_{t}\in\Lambda^{i}_{t}}\operatorname{\mathbb{E}}^{\tilde{\sigma}^{i}_{t},\sigma^{-i}_{t},\psi^{\sigma}}\\{u^{i}_{t}(X_{t},A_{t})+\breve{V}^{i}_{t+1}(\psi^{\sigma}_{t+1}(\pi^{\psi^{\sigma}}_{t},Z_{t+1}),S^{i}_{t+1})\mid
s^{i}_{t}\\},$
$\displaystyle\,\forall\pi^{\psi^{\sigma}}_{t}\in\Delta(\mathcal{X}_{t}\times\mathcal{S}_{t})^{N},\forall
s^{i}_{t}\in\mathcal{S}^{i}_{s},t\in\mathcal{T}$ (47)

where $\Lambda^{i}_{t}$ is the set of SIB strategies of agent $i$ at time $t$.

###  5.2 Sequential decomposition

Given a set of value functions
$V_{t+1}=\\{V^{i}_{t+1}:\mathbf{\Pi}_{t+1}\times\mathcal{S}^{i}_{t+1}\rightarrow\mathbb{R},i\in\mathcal{N}\\}$,
a SIB strategy profile $\sigma$, the corresponding update rule
$\psi^{\sigma}_{t+1}$ defined by $\eqref{eq:psi_t}$, and the consistent CIB
belief $\pi^{\psi^{\sigma}}_{t}$, define the stage-game
$G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})$ as follows.

(i) There are $N$ agents. (ii) The system state is $X_{t}$. (iii) Each agent
$i$ observes private information $S^{i}_{t}$ and common information
$\pi^{\psi^{\sigma}}_{t}$. (iv) Agent $i$’s belief about the state $X_{t}$ and
other agents’ private information $S^{-i}_{t}$ is given by
$\pi^{\psi^{\sigma},i}_{t}(x_{t},s^{-i}_{t})$, that is,

$\displaystyle\pi^{\psi^{\sigma},i}_{t}(x_{t},s^{-i}_{t})\in\Delta(\mathcal{X}_{t}\times\mathcal{S}^{-i}_{t}).$
(48)

(v) Each agent $i$ selects action $A^{i}_{t}$ based on his available
information; let $\hat{\sigma}^{i}_{t}$ denote agent $i$’s strategy for this
stage-game; then,

$\displaystyle\operatorname{\mathbb{P}}^{\hat{\sigma}_{t},\psi^{\sigma}}(A^{i}_{t}=a^{i}_{t}\mid
s^{i}_{t},\pi^{\psi^{\sigma}}_{t})=\hat{\sigma}^{i}_{t}(a^{i}_{t}\mid
s^{i}_{t},\pi^{\psi^{\sigma}}_{t}).$ (49)

(vi) Each agent $i$ has utility

$\displaystyle
U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}=u^{i}_{t}(X_{t},A_{t})+V^{i}_{t+1}(\psi^{\sigma}_{t+1}(\pi^{\psi^{\sigma}}_{t},Z_{t+1}),S^{i}_{t+1})$
(50)

where $(Z_{t+1},S^{i}_{t+1})$ conditioned on $(X_{t},S_{t},A_{t})$ follows the
conditional probability
$\sum_{x_{t+1},s^{-i}_{t+1}}\operatorname{\mathbb{P}}(z_{t+1},x_{t+1},s_{t+1}\mid
x_{t},s_{t},a_{t})$ and the conditional probability
$\operatorname{\mathbb{P}}(z_{t+1},x_{t+1},s_{t+1}\mid x_{t},s_{t},a_{t})$ is
given by

$\displaystyle\operatorname{\mathbb{P}}(z_{t+1},x_{t+1},s_{t+1}\mid
x_{t},s_{t},a_{t})$ $\displaystyle=$
$\displaystyle\sum_{y_{t+1}}\mathbb{P}\\{x_{t+1}\mid
x_{t},a_{t}\\}\mathbb{P}\\{z_{t+1},y_{t+1}\mid x_{t+1},a_{t}\\}$
$\displaystyle\hskip
28.45274pt\left(\prod_{j}\mathbbm{1}\\{s_{t+1}^{j}=\phi_{t+1}^{j}(s_{t}^{j},z_{t+1},y_{t+1}^{j},a_{t}^{j})\\}\right)$
(51)

(vii) Given a strategy profile $\hat{\sigma}_{t}$ for the stage-game, the
expected utility of each player $i$ is given by

$\displaystyle\operatorname{\mathbb{E}}^{\hat{\sigma}_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]$ $\displaystyle=$
$\displaystyle\sum_{x_{t},s_{t}^{-i},a_{t},z_{t+1},x_{t+1},s_{t+1}}\pi^{\psi^{\sigma},i}_{t}(x_{t},s_{t}^{-i})\prod_{j}\hat{\sigma}^{j}_{t}(a^{i}_{t}\mid
s^{i}_{t},\pi^{\psi^{\sigma}}_{t})\operatorname{\mathbb{P}}(z_{t+1},x_{t+1},s_{t+1}\mid
x_{t},s_{t},a_{t})$ $\displaystyle\hskip
56.9055pt(u^{i}_{t}(x_{t},a_{t})+V^{i}_{t+1}(\psi^{\sigma}_{t+1}(\pi_{t}^{\psi^{\sigma}},z_{t+1}),s^{i}_{t+1}))$
(52)

Note that all the random variables of the stage-game
$G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})$ may not necessarily be the same as
their counterparts in the original dynamic game since each agent $i$ is
allowed to choose an arbitrary SIB strategy $\hat{\sigma}^{i}_{t}$ which may
be different from $\sigma^{i}_{t}$ specified by the SIB strategy profile
$\sigma$. The stage-game random variables will coincide with their
counterparts in the original game if all agents follow $\sigma$.

######  Theorem 2 (Sequential decomposition).

Consider a SIB strategy profile $\sigma=\\{\sigma_{t},t\in\mathcal{T}\\}$ and
the corresponding update rule
$\psi^{\sigma}=\\{\psi^{\sigma}_{t},t\in\mathcal{T}\\}$ defined by (28)-(29).
Define

$\displaystyle V_{T+1}^{i}(\cdot,\cdot)=0\text{ for all }i$ (53)
$\displaystyle
V^{i}_{t}(\pi^{\psi^{\sigma}_{t}},s^{i}_{t})=\operatorname{\mathbb{E}}^{\sigma_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]$ (54)

where the right hand side of (54) is given by (52). If for all
$t\in\mathcal{T}$, there is a SIB strategy profile $\hat{\sigma}_{t}$ such
that $\hat{\sigma}_{t}$ is a BNE of the stage-game
$G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})$, that is,

$\displaystyle\operatorname{\mathbb{E}}^{\hat{\sigma}^{i}_{t},\hat{\sigma}^{-i}_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]=\max_{\tilde{\sigma}^{i}_{t}\in\Lambda^{i}_{t}}\operatorname{\mathbb{E}}^{\tilde{\sigma}^{i}_{t},\hat{\sigma}^{-i}_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]$ (55)

for all $i\in\mathcal{N}$ where $\Lambda^{i}_{t}$ is the set of SIB strategies
of agent $i$ at time $t$, and

$\displaystyle\hat{\sigma}_{t}=\sigma_{t},$ (56)

then the SIB strategy profile $\sigma$ is a SIB-BNE of the original dynamic
game.

Suppose that for all $t\in\mathcal{T}$ there is a SIB strategy profile
$\hat{\sigma}_{t}=(\hat{\sigma}^{1}_{t},\hat{\sigma}^{2}_{t},\ldots,\hat{\sigma}^{N}_{t})$
that is a BNE of the stage game $G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})$. Then
for all
$\pi^{\psi^{\sigma}}_{t}\in\Delta(\mathcal{X}_{t}\times\mathcal{S}_{t})^{N},s^{i}_{t}\in\mathcal{S}^{i}_{s}$

$\displaystyle\operatorname{\mathbb{E}}^{\hat{\sigma}^{i}_{t},\hat{\sigma}^{-i}_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]$ $\displaystyle=$
$\displaystyle\max_{\tilde{\sigma}^{i}_{t}\in\Lambda^{i}_{t}}\operatorname{\mathbb{E}}^{\tilde{\sigma}^{i}_{t},\hat{\sigma}^{-i}_{t},\psi^{\sigma}}[u^{i}_{t}(X_{t},A_{t})+V^{i}_{t+1}(\psi^{\sigma}_{t+1}(\pi^{\psi^{\sigma}}_{t},Z_{t+1}),S^{i}_{t+1})\mid
s^{i}_{t}].$ (57)

Equation (57) holds for all $t\in\mathcal{T}$ with
$V^{i}_{T+1}(\cdot,\cdot)=0$ and for all $i\in\mathcal{N}$. When
$\hat{\sigma}_{t}=\sigma_{t}$ for all $t\in\mathcal{T}$, Equation (57) gives,
for all
$\pi^{\psi^{\sigma}}_{t}\in\Delta(\mathcal{X}_{t}\times\mathcal{S}_{t})^{N},s^{i}_{t}\in\mathcal{S}^{i}_{s}$,

$\displaystyle V^{i}_{t}(\pi^{\psi^{\sigma}}_{t},s^{i}_{t})=$
$\displaystyle\operatorname{\mathbb{E}}^{\sigma^{i}_{t},\sigma^{-i}_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]$ $\displaystyle=$
$\displaystyle\max_{\tilde{\sigma}^{i}_{t}\in\Lambda^{i}_{t}}\operatorname{\mathbb{E}}^{\tilde{\sigma}^{i}_{t},\sigma^{-i}_{t},\psi^{\sigma}}[u^{i}_{t}(X_{t},A_{t})+V^{i}_{t+1}(\psi^{\sigma}_{t+1}(\pi^{\psi^{\sigma}}_{t},Z_{t+1}),S^{i}_{t+1})\mid
s^{i}_{t}]$ (58)

for all $i\in\mathcal{N}$.

By induction, (58), and the fact that the update rule $\psi^{\sigma}$ is
consistent with $\sigma$ we have, for all $i\in\mathcal{N}$ and
$t\in\mathcal{T}$,

$\displaystyle\operatorname{\mathbb{E}}^{\tilde{\sigma}^{i}_{t:T},\sigma^{-i}_{t:T},\psi^{\sigma}}[\sum_{\tau=t}^{T}u^{i}_{\tau}(X_{\tau},A_{\tau})\mid
s^{i}_{\tau}]\leq\operatorname{\mathbb{E}}^{\sigma^{i}_{t:T},\sigma^{-i}_{t:T},\psi^{\sigma}}[\sum_{\tau=t}^{T}u^{i}_{\tau}(X_{\tau},A_{\tau})\mid
s^{i}_{\tau}]$ (59)

Then (59) at time $t=1$ gives

$\displaystyle\mathbb{E}^{\tilde{\sigma}^{i},\sigma^{-i},\psi^{\sigma}}\\{U^{i}(X_{1:T},A_{1:T})\\}\leq\mathbb{E}^{\sigma,\psi^{\sigma}}\\{U^{i}(X_{1:T},A_{1:T})\\}$
(60)

for all $\tilde{\sigma}^{i}\in\Lambda^{i}$ for all $i\in\mathcal{N}$.
Therefore, the strategy profile $\sigma$ is a SIB-BNE of the original dynamic
game (sf. Definition 6).

######  Remark 5.

Note that even when the stage-game $G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})$
has a BNE $\hat{\sigma}_{t}$, it is possible that
$\hat{\sigma}_{t}\neq\sigma_{t}$. Thus, the existence of BNE for every stage-
game $G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})$ is not sufficient to establish
the existence of BNE for the original dynamic game.

######  Remark 6.

In the model of Tang et al (2022) when each team consists of one agent, a SIB
BNE coincides with a SPCIB BNE introduced in Tang et al (2022) with an
appropriate mapping of the information state as discussed in Remark 4.

######  Remark 7.

There may not be a solution for the set of value functions in the sequential
decomposition equations described by (53)-(56) for all $i\in\mathcal{N}$ and
for all $t\in\mathcal{T}$.

######  Remark 8.

In Definition 4, (27) could be defined differently, and different (27) would
lead to different choices of $\psi$. And for any choice of (27), the claim of
Theorem 2 will still hold.

######  Remark 9.

The value functions of the sequential decomposition equations defined by
Theorem 2 (Eqs. (53)-(56) for all $i\in\mathcal{N},t\in\mathcal{T}$) may not
be continuous in the CIB belief $\Pi^{\psi^{\sigma}}_{t}$.

##  6 An illustrative example (Step 4)

In Section 5 we argued (cf. Remark 7) that the sequential decomposition
equations defined by (53)-(56) for all $i\in\mathcal{N},t\in\mathcal{T}$ may
not have a solution, and that the value functions defined by (53)-(56) may not
be continuous in the CIB belief $\Pi^{\psi^{\sigma}}_{t}$ (cf. Remark 9). In
this section we present an example that illustrates/highlights the above
remarks. In the example, a two-stage stochastic dynamic game, the agents’
utilities depend on a parameter $c$. We show that: (i) the value functions of
the corresponding sequential decomposition equations are not continuous in the
CIB belief $\Pi^{\psi^{\sigma}}_{t}$; (ii) for certain values of $c$ a SIB-BNE
exists.

###  6.1 Model

We consider the following two-stage stochastic dynamic game. There are two
players/agents, Alice and Bob. At stage one, $t=1$, the system’s state $X_{1}$
is distributed on $\\{-1,1\\}$ with
$\mu_{0}(-1)=\operatorname{\mathbb{P}}(X_{1}=-1)=0.5$ and
$\mu_{1}(1)=\operatorname{\mathbb{P}}(X_{1}=1)=0.5$. Alice observes perfectly
$X_{1}$, i.e., $Y^{Alice}_{1}=X_{1}$, and takes action
$A^{Alice}_{1}\in\\{-1,1\\}$; $A^{Alice}_{1}$ is not observable by Bob and
$Y^{Bob}_{1}=\emptyset$. Bob does not act at $t=1$. At stage $2$, $t=2$, the
system state is $X_{2}=X_{1}A^{Alice}_{1}$. Alice and Bob have a common
observation $Z_{2}=X_{2}A^{Alice}_{1}W_{1}=X_{1}W_{1}$, where
$W_{1}\in\\{-1,1\\}$ and $\operatorname{\mathbb{P}}(Z=i\mid
X_{1}=i)=1-p=0.8,i\in\\{-1,1\\}$, and there are no private observations, i.e.,
$Y^{Alice}_{2}=Y^{Bob}_{2}=\emptyset$. Here
$p=0.2=\operatorname{\mathbb{P}}(W_{1}=-1)$. Bob acts at $t=2$. Alice does not
act at $t=2$. Bob’s action $A^{Bob}_{2}\in\\{-1,1\\}$. Alice’s payoffs at
$t=1$ and $t=2$ are

$\displaystyle u^{Alice}_{1}(X_{1},A_{1})=$
$\displaystyle\left\\{\begin{array}[]{ll}c&\text{ if }A^{Alice}_{1}=1\\\
0&\text{ if }A^{Alice}_{1}=-1\end{array}\right.$ (63)

and

$\displaystyle u^{Alice}_{2}(X_{2},A_{2})=$
$\displaystyle\left\\{\begin{array}[]{ll}2&\text{ if }X_{2}=1,A^{Bob}_{2}=1\\\
1&\text{ if }X_{2}=-1,A^{Bob}_{2}=-1\\\ 0&\text{ otherwise
}\end{array}\right.$ (67)

respectively. Bob’s payoffs are
$u^{Bob}_{t}(X_{t},A_{t})=-u^{Alice}_{t}(X_{t},A_{t}),t=1,2$.

The game’s information structure is

$\displaystyle H^{Alice}_{1}=$ $\displaystyle\\{X_{1}\\}$ (68) $\displaystyle
H^{Alice}_{2}=$ $\displaystyle\\{X_{1},A^{Alice}_{1},X_{2},Z_{2}\\}$ (69)
$\displaystyle H^{Bob}_{1}=$ $\displaystyle\emptyset$ (70) $\displaystyle
H^{Bob}_{2}=$ $\displaystyle\\{Z_{2}\\}$ (71)

where $H^{Alice}_{t},H^{Bob}_{t},t=1,2$, describe the information available to
Alice and Bob, respectively, at stages $1$ and $2$.

This example has the same dynamics and utility functions as Example 3 in Tang
et al (2022), but Bob doesn’t observe Alice’s action as in (Tang et al, 2022,
Example 3).

###  6.2 Sequential decomposition

Since Alice perfectly observes the state at both times, i.e.,
$Y^{Alice}_{1}=X_{1}$ and $Y^{Alice}_{2}=X_{2}$, and Bob doesn’t have private
information, $S^{Alice}_{1}=X_{1},S^{Bob}_{1}=\emptyset$ are sufficient
private information for Alice and Bob at stage $t=1$, respectively, and
$S^{Alice}_{2}=X_{2},S^{Bob}_{2}=\emptyset$ are sufficient private information
for Alice and Bob, respectively, at stage $t=2$ according to Definition 1.

Suppose $\sigma=(\sigma_{1},\sigma_{2})=(\sigma^{Alice}_{1},\sigma^{Bob}_{2})$
is a SIB strategy and $\psi^{\sigma}$ is the corresponding update rule. Here
$\sigma$ is an equilibrium strategy candidate which serves as the strategy
prediction for Alice and Bob. Note that
$\Pi_{1}^{\psi^{\sigma},Alice}(x_{1})=\mu_{0}(x_{1})$ and
$\Pi_{1}^{\psi^{\sigma},Bob}(x_{1})=\mu_{0}(x_{1})$ for all
$x_{1}\in\mathcal{X}_{1}$.

To get a BNE using the sequential decomposition of Theorem 2, we first
consider the stage-game $G_{2}(0,\pi_{2}^{\psi^{\sigma}})$ at time $2$. Since
Bob is the only agent who acts at time $2$ and $S^{Bob}_{2}=\emptyset$, any
BNE $\sigma_{2}$ of $G_{2}(0,\pi_{2}^{\psi^{\sigma}})$ must satisfy

$\displaystyle\hat{\sigma}_{2}^{Bob}=$
$\displaystyle\operatorname*{arg\,max\,}_{\tilde{\sigma}^{Bob}_{2}}\operatorname{\mathbb{E}}^{\tilde{\sigma}^{Bob}_{2},\psi^{\sigma}}[u^{Bob}_{2}(X_{2},A_{2})]$
$\displaystyle=$
$\displaystyle\operatorname*{arg\,max\,}_{\tilde{\sigma}^{Bob}_{2}}\Big{(}-2\operatorname{\mathbb{P}}^{\tilde{\sigma}^{Bob}_{2},^{\psi^{\sigma}}}(X_{2}=A^{Bob}_{2}=1)-\operatorname{\mathbb{P}}^{\tilde{\sigma}^{Bob}_{2},\psi^{\sigma}}(X_{2}=A^{Bob}_{2}=-1)\Big{)}$
$\displaystyle=$
$\displaystyle\operatorname*{arg\,max\,}_{\tilde{\sigma}^{Bob}_{2}}\Big{(}-2\pi^{\psi^{\sigma},Bob}_{2}(1)\tilde{\sigma}^{\psi^{\sigma},Bob}_{2}(1\mid\pi_{2}^{\psi^{\sigma}})$
$\displaystyle\hskip
56.9055pt-(1-\pi^{\psi^{\sigma},Bob}_{2}(1))(1-\tilde{\sigma}^{\psi^{\sigma},Bob}_{2}(1\mid\pi_{2}^{\psi^{\sigma}}))\Big{)}$
(72)

From (72) we conclude that one of the equilibrium SIB strategies is given by

$\displaystyle\sigma^{Bob}_{2}(\pi_{2}^{\psi^{\sigma}})=1\text{, if
}\pi^{\psi^{\sigma},Bob}_{2}(1)\leq 1/3,$
$\displaystyle\sigma^{Bob}_{2}(\pi_{2}^{\psi^{\sigma}})=0\text{, if
}\pi^{\psi^{\sigma},Bob}_{2}(1)>1/3,$

or equivalently

$\displaystyle\sigma^{Bob}_{2}(\pi_{2}^{\psi^{\sigma}})=\mathds{1}(\pi^{\psi^{\sigma},Bob}_{2}(1)\leq
1/3)$ (73)

Note that $\sigma^{Bob}_{2}(\pi_{2}^{\psi^{\sigma}})$ can take any value in
$[0,1]$ if $\pi^{\psi^{\sigma},Bob}_{2}(1)=1/3$ and $\sigma_{2}$ is still a
BNE of the stage-game.

Alice’s sufficient private information at time $2$ is $S^{Alice}_{2}=X_{2}$.
With the stage-game equilibrium SIB strategy $\sigma^{Bob}_{2}(\pi_{2})$ given
by (73), the value function for Alice at $t=2$ is then given, according to
(54), by

$\displaystyle V^{Alice}_{2}(\pi_{2}^{\psi^{\sigma}},x_{2})=$
$\displaystyle\operatorname{\mathbb{E}}^{\sigma_{2},\psi^{\sigma}}[u^{Alice}_{2}(X_{2},A_{2})\mid
x_{2}]$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{ll}2\mathds{1}(\pi^{\psi^{\sigma},Bob}_{2}(1)\leq
1/3)&\text{if }x_{2}=1\\\ 1-\mathds{1}(\pi^{\psi^{\sigma},Bob}_{2}(1)\leq
1/3)&\text{if }x_{2}=-1\end{array}\right.$ (76)

Given the above value functions at time $t=2$, we now consider the stage-game
$G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})$ at time $t=1$. The utility for the
stage-game for Alice is given as follows.

$\displaystyle
U^{Alice}_{G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})}=u^{Alice}_{1}(X_{1},A_{1})+V^{Alice}_{2}(\psi^{\sigma}_{2}(\pi_{1},Z),X_{2})$
(77)

If Alice uses the SIB strategy $\tilde{\sigma}^{Alice}_{1}$, the expected
utility of the stage-game can be calculated for $X_{1}=-1$ and $X_{1}=1$,
according to (52), by

$\displaystyle\operatorname{\mathbb{E}}^{\tilde{\sigma}^{Alice}_{1},\psi^{\sigma}}[U^{Alice}_{G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})}\mid
X_{1}=-1]$ $\displaystyle=$ $\displaystyle
c\tilde{\sigma}^{Alice}_{1}(1\mid-1)+\operatorname{\mathbb{E}}^{\tilde{\sigma}^{Alice}_{1},\psi^{\sigma}}[V^{A}_{2}(\psi_{2}^{\sigma}(\pi_{1}^{\psi^{\sigma}},X_{1}W_{1}),X_{1}A^{Alice}_{1})\mid
X_{1}=-1]$ $\displaystyle=$
$\displaystyle(1+c)(1-\tilde{\alpha}_{1})+(3\tilde{\alpha}_{1}-1)((1-p)\mathds{1}(q_{-1}\leq
1/3)+p\mathds{1}(q_{1}\leq 1/3))$ $\displaystyle=:$ $\displaystyle
r^{A}_{-1}(\tilde{\alpha}_{1},q)$ (78)
$\displaystyle\operatorname{\mathbb{E}}^{\tilde{\sigma}^{Alice}_{1},\psi^{\sigma}}[U^{Alice}_{G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})}\mid
X_{1}=1]$ $\displaystyle=$ $\displaystyle c\tilde{\sigma}^{Alice}_{1}(1\mid
1)+\operatorname{\mathbb{E}}^{\tilde{\sigma}^{Alice}_{1},\psi^{\sigma}}[V^{A}_{2}(\psi^{\sigma}_{2}(\pi_{1}^{\psi^{\sigma}},X_{1}W_{1}),X_{1}A^{Alice}_{1})\mid
X_{1}=1]$ $\displaystyle=$ $\displaystyle
1+(c-1)\tilde{\alpha}_{2}+(3\tilde{\alpha}_{2}-1)((1-p)\mathds{1}(q_{1}\leq
1/3)+p\mathds{1}(q_{-1}\leq 1/3))$ $\displaystyle=:$ $\displaystyle
r^{A}_{1}(\tilde{\alpha}_{2},q)$ (79)

where $q=(q_{-1},q_{1})$,
$q_{-1}=\psi_{2}^{\sigma,Bob}(\pi_{1}^{\psi^{\sigma}},-1)(1)$ and
$q_{1}=\psi_{2}^{\sigma,Bob}(\pi_{1}^{\psi^{\sigma}},1)(1)$ are the CIB
beliefs $\pi^{\psi^{\sigma},Bob}_{2}(1)$ of $\\{X_{2}=1\\}$ when $Z=-1$ and
$Z=1$, respectively, and
$\tilde{\alpha}=(\tilde{\alpha}_{1},\tilde{\alpha}_{2})$,
$\tilde{\alpha}_{1}=\tilde{\sigma}^{Alice}_{1}(-1\mid-1),\tilde{\alpha}_{2}=\tilde{\sigma}^{Alice}_{1}(1\mid
1)$ represents Alice’s SIB strategy $\tilde{\sigma}^{Alice}_{1}$.

Note that from Bayes’ rule in Definition 4, under the SIB strategy
$\sigma^{Alice}_{1}$, represented by $\alpha_{1}=\sigma^{Alice}_{1}(-1\mid-1)$
and $\alpha_{2}=\sigma^{Alice}_{1}(1\mid 1)$, we have

$\displaystyle
q_{-1}=\psi_{2}^{\psi^{\sigma},Bob}(\pi_{1}^{\psi^{\sigma}},-1)(1)=\frac{\operatorname{\mathbb{P}}^{\alpha}(X_{2}=1,Z=-1)}{\operatorname{\mathbb{P}}^{\alpha}(Z=-1)}=\alpha_{2}p+\alpha_{1}(1-p)$
(80) $\displaystyle
q_{1}=\psi_{2}^{\psi^{\sigma},Bob}(\pi_{1}^{\psi^{\sigma}},1)(1)=\frac{\operatorname{\mathbb{P}}^{\alpha}(X_{2}=1,Z=1)}{\operatorname{\mathbb{P}}^{\alpha}(Z=1)}=\alpha_{2}(1-p)+\alpha_{1}p$
(81)

Therefore, a SIB strategy $\hat{\sigma}^{Alice}_{1}$, represented by
$\hat{\alpha}_{1}=\hat{\sigma}^{Alice}_{1}(-1\mid-1)$ and
$\hat{\alpha}_{2}=\hat{\sigma}^{Alice}_{1}(1\mid 1)$, is a BNE of the stage-
game $G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})$ at time $t=1$ if

$\displaystyle\hat{\alpha}_{1}\in\operatorname*{arg\,max\,}_{\tilde{\alpha}_{1}}r^{A}_{-1}(\tilde{\alpha}_{1},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p))$
(82)
$\displaystyle\hat{\alpha}_{2}\in\operatorname*{arg\,max\,}_{\tilde{\alpha}_{2}}r^{A}_{1}(\tilde{\alpha}_{2},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p))$
(83)

Consequently, the SIB strategy $\sigma^{Alice}_{1}$, represented by
$\alpha_{1}=\sigma^{Alice}_{1}(-1\mid-1)$ and
$\alpha_{2}=\sigma^{Alice}_{1}(1\mid 1)$ will satisfy the sequential
decomposition equations (55)-(56) if

$\displaystyle\alpha_{1}\in\operatorname*{arg\,max\,}_{\tilde{\alpha}_{1}}r^{A}_{-1}(\tilde{\alpha}_{1},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p))$
(84)
$\displaystyle\alpha_{2}\in\operatorname*{arg\,max\,}_{\tilde{\alpha}_{2}}r^{A}_{1}(\tilde{\alpha}_{2},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p))$
(85)

######  Remark 10.

Note that the functions $r^{A}_{-1}(\tilde{\alpha}_{1},q)$ and
$r^{A}_{1}(\tilde{\alpha}_{2},q)$ are not continuous in $q$. Thus existence of
equilibria cannot be established by the standard method relying on the
continuity of the utility functions, and there may not no equilibria in the
general case.

###  6.3 Existence of SIB-BNE under conditions on the instantaneous utility.

The stage-game $G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})$ is a normal-form game
with a fixed $\sigma_{1}$. According to Remark 5, a BNE $\hat{\sigma}$ of
$G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})$ could be different from $\sigma_{1}$
and the existence of a regular BNE of $G_{1}(V_{2},\pi^{\psi^{\sigma}}_{1})$
is not sufficient to satisfy (56) at time $t=1$. In order to apply equilibrium
existence results for normal-form games to the sequential decomposition at
time $t=1$, we introduce an agent $0$ who picks the $q$-belief
$q=(q_{-1},q_{1})$ so that (56) is satisfied.

Formally, we construct an augmented stage-game $\hat{G}_{1}$ between Alice and
agent $0$. Alice chooses
$\tilde{\alpha}=(\tilde{\alpha}_{1},\tilde{\alpha}_{2})$ and agent $0$ chooses
$\tilde{q}=(\tilde{q}_{-1},\tilde{q}_{1})$. Alice’s utility is

$\displaystyle r^{A}_{1}(\tilde{\alpha},\tilde{q})=$ $\displaystyle
0.5r^{A}_{-1}(\tilde{\alpha}_{1},\tilde{q})+0.5r^{A}_{1}(\tilde{\alpha}_{2},\tilde{q})$
$\displaystyle=$ $\displaystyle
0.5c(1-\tilde{\alpha}_{1}+\tilde{\alpha}_{2})+0.5(2-\tilde{\alpha}_{1}-\tilde{\alpha}_{2})$
$\displaystyle+0.5(3(\tilde{\alpha}_{2}p+\tilde{\alpha}_{1}(1-p))-1)\mathds{1}(\tilde{q}_{-1}\leq
1/3)$
$\displaystyle+0.5(3(\tilde{\alpha}_{2}(1-p)+\tilde{\alpha}_{1}p)-1)\mathds{1}(\tilde{q}_{1}\leq
1/3).$ (86)

Agent $0$’s utility is

$\displaystyle
r^{0}_{1}(\tilde{\alpha},\tilde{q})=-(\tilde{q}_{-1}-\tilde{\alpha}_{2}p-\tilde{\alpha}_{1}(1-p))^{2}-(\tilde{q}_{1}-\tilde{\alpha}_{2}(1-p)-\tilde{\alpha}_{1}p)^{2}.$
(87)

Both Alice and agent $0$ are utility maximizers. The game $\hat{G}_{1}$ with
utilities (87)-(86) is a normal-form game with strategies
$\tilde{\alpha}=(\tilde{\alpha}_{1},\tilde{\alpha}_{2})$
$\tilde{q}=(\tilde{q}_{-1},\tilde{q}_{1})$. Since the utility (87) of agent
$0$ is a quadratic function, any best response by agent $0$ must satisfy
$\tilde{q}_{-1}=\tilde{\alpha}_{2}p+\tilde{\alpha}_{1}(1-p),\tilde{q}_{1}=\tilde{\alpha}_{2}(1-p)+\tilde{\alpha}_{1}p$.

Note that in the augmented stage-game $\hat{G}_{1}$, the utility function
$r^{A}_{1}(\tilde{\alpha},\tilde{q})$ is not continuous in $\tilde{q}$. To
show the existence of a Nash equilibrium for $\hat{G}_{1}$, we proceed to
apply existence results for games with discontinuous utilities in Barelli and
Meneghel (2013).

Specifically, Proposition 2.4 of Barelli and Meneghel (2013) guarantees the
existence of a Nash equilibrium for games satisfying the generalized better
reply secure property. From Definition 2.3 in Barelli and Meneghel (2013), the
stage game is generalized better reply secure if for any
$(\bar{\alpha},\bar{q})$ not an equilibrium, at least one of the followings is
true

  * •

We can find an $\epsilon>0$ and a closed correspondence
$\phi^{0}(\tilde{\alpha},\tilde{q})$ such that

$\displaystyle
r^{0}_{1}(\tilde{\alpha},\phi^{0}(\tilde{\alpha},\tilde{q}))\geq
r^{0}_{1}(\bar{\alpha},\bar{q})+\epsilon$ (88)

for all
$\tilde{\alpha}_{1}\in(\bar{\alpha}_{1}-\epsilon,\bar{\alpha}_{1}+\epsilon)$,
$\tilde{\alpha}_{2}\in(\bar{\alpha}_{2}-\epsilon,\bar{\alpha}_{2}+\epsilon)$,
$\tilde{q}_{-1}\in(\bar{q}_{-1}-\epsilon,\bar{q}_{-1}+\epsilon)$,
$\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$

  * •

We can find an $\epsilon>0$ and a closed correspondence
$\phi^{A}(\tilde{\alpha},\tilde{q})$ such that

$\displaystyle r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})\geq
r^{A}_{1}(\bar{\alpha},\bar{q})+\epsilon$ (89)

for all
$\tilde{\alpha}_{1}\in(\bar{\alpha}_{1}-\epsilon,\bar{\alpha}_{2}+\epsilon)$,
$\tilde{\alpha}_{2}\in(\bar{\alpha}_{2}-\epsilon,\bar{\alpha}_{2}+\epsilon)$,
$\tilde{q}_{-1}\in(\bar{q}_{-1}-\epsilon,\bar{q}_{-1}+\epsilon)$,
$\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$

In Appendix .2, we show that when $c>24$ the augmented stage-game
$\hat{G}_{1}$ is generalized better reply secure. Thus, there exists a Nash
equilibrium of the augmented state-game $\hat{G}_{1}$ according to (Barelli
and Meneghel, 2013, Proposition 2.4).

Consider any Nash equilibrium $(\alpha,q)$ of $\hat{G}_{1}$. Since $q$ is a
best response to $\alpha$ for agent $0$, from agent $0$’s utility (87) we have

$\displaystyle q_{-1}=\alpha_{2}p+\alpha_{1}(1-p)$ (90) $\displaystyle
q_{1}=\alpha_{2}(1-p)+\alpha_{1}p$ (91)

Furthermore, since $\alpha$ is a best response to $q$ for Alice in
$\hat{G}_{1}$,

$\displaystyle\alpha\in$
$\displaystyle\operatorname*{arg\,max\,}_{\tilde{\alpha}}\Big{(}0.5r^{A}_{-1}(\tilde{\alpha}_{1},q)+0.5r^{A}_{1}(\tilde{\alpha}_{2},q)\Big{)}$
$\displaystyle=$
$\displaystyle\operatorname*{arg\,max\,}_{\tilde{\alpha}}\Big{(}0.5r^{A}_{-1}(\tilde{\alpha}_{1},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p))$
$\displaystyle\hskip
56.9055pt+0.5r^{A}_{1}(\tilde{\alpha}_{2},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p))\Big{)}$
$\displaystyle=$
$\displaystyle\Big{(}\operatorname*{arg\,max\,}_{\tilde{\alpha}_{1}}r^{A}_{-1}(\tilde{\alpha}_{1},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p)),$
$\displaystyle\hskip
28.45274pt\operatorname*{arg\,max\,}_{\tilde{\alpha}_{2}}r^{A}_{1}(\tilde{\alpha}_{2},(\alpha_{2}p+\alpha_{1}(1-p),\alpha_{2}(1-p)+\alpha_{1}p))\Big{)}$
(92)

Therefore, (84)-(85) hold for $\alpha$, and consequently the sequential
decomposition requirement (55)-(56) is satisfied at $t=1$ by the SIB strategy
$\sigma^{Alice}_{1}$ represented by $\alpha$, and we establish the existence
of a SIB equilibrium based on Theorem 2.

##  7 The case with no common observations

We consider the model of Section 2 but we assume that the agents have no
common observations, that is,

$\displaystyle Z_{t}=\emptyset\quad\forall t\in\mathcal{T}.$ (93)

The system’s dynamics, the agents’ private observations, the functional form
of the agents’ strategies, their utilities, and the equilibrium concept (BNE)
remain the same as in Section 2.

Even though the agents have no common observations in this special case, we
can still define SIB strategies by Definition 3, and construct the consistent
CIB belief system according to Definition 4 with $Z_{t}=\emptyset\,\forall
t\in\mathcal{T}$.

Since there is no common observations, for any realization we always have

$\displaystyle\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1},z_{t+1})(\pi^{\psi^{\sigma}}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})$ $\displaystyle=$
$\displaystyle\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1})(\pi^{\psi^{\sigma}}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})=1>0$ (94)

Therefore, case (ii) in Definition 4 would never happen, and (26) can be
simplified to

$\displaystyle\pi_{t+1}^{\psi^{\sigma},i}(x_{t+1},s_{t+1})$ $\displaystyle=$
$\displaystyle\frac{F_{t}^{i}(x_{t+1},s_{t+1})(\pi^{\psi^{\sigma}}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})}{\sum_{\hat{x}_{t+1},\hat{s}_{t+1}}F_{t}^{i}(\hat{x}_{t+1},\hat{s}_{t+1})(\pi_{t}^{\psi^{\sigma}}\mathchar
24635\relax\;\sigma^{-i}_{t})}$ $\displaystyle=$ $\displaystyle
F_{t}^{i}(x_{t+1},s_{t+1})(\pi^{\psi^{\sigma}}_{t}\mathchar
24635\relax\;\sigma^{-i}_{t})$ $\displaystyle=$
$\displaystyle\sum_{y_{t+1},x_{t},s_{t},a_{t}}\Bigg{[}\mathbb{P}\\{y_{t+1},x_{t+1}\mid
x_{t},a_{t}\\}\left(\prod_{j}\mathbbm{1}\\{s_{t+1}^{j}=\phi_{t+1}^{j}(s_{t}^{j},y_{t+1}^{j},a_{t}^{j})\\}\right)$
$\displaystyle\hskip 50.0pt\left(\frac{1}{|A_{t}^{i}|}\prod_{j\neq
i}\sigma^{j}_{t}(a^{j}_{t})(\pi^{\psi}_{t},s_{t}^{j})\right)\pi_{t}^{\psi,i}(x_{t},s_{t})\Bigg{]}.$
(95)

Based on (95) we can write

$\displaystyle\Pi_{t+1}^{\psi^{\sigma},i}=\psi_{t+1}^{\sigma,i}(\Pi_{t}^{\psi^{\sigma}})\quad\forall
i\in\mathcal{N},$ (96)
$\displaystyle\Pi_{t+1}^{\psi^{\sigma}}=\psi^{\sigma}_{t+1}(\Pi_{t}^{\psi^{\sigma}}).$
(97)

In other words, given a SIB strategy $\sigma$, the update rule $\psi^{\sigma}$
are deterministic functions given by (97), and the corresponding consistent
CIB belief system $\Pi_{t}^{\psi^{\sigma}},t\in\mathcal{T}$, evolves in a
deterministic manner. Furthermore, since case (ii) in Definition 4 never
happens without common observations, the update rule $\psi_{t+1}^{\sigma,i}$
given by (95) becomes exactly the Bayes rule. As a result, the CIB belief
$\Pi_{t}^{\psi^{\sigma},i}$ becomes a regular PMF given by

$\displaystyle\Pi_{t}^{\psi^{\sigma},i}(x_{t},s_{t})=\operatorname{\mathbb{P}}^{\tilde{g}^{i},\sigma^{-i}}(x_{t},s_{t})\quad\forall
i\in\mathcal{N}$ (98)

where $\tilde{g}^{i}$ denotes the uniform strategy (i.e., the strategy that
chooses every action $a^{i}_{t}\in\mathcal{A}^{i}_{t}$ with equal probability
for all $t\in\mathcal{T}$).

######  Remark 11.

If the $N$ agents have identical utilities, i.e. we have a dynamic team
problem, then $\Pi_{t}^{\psi^{\sigma}},t\in\mathcal{T}$ is similar to the
common knowledge that appears in Witsenhausen (1973) where a dynamic team is
analyzed. The common knowledge in Witsenhausen (1973) is a sequence (over
time) of PMFs on the system’s history $H_{t},t\in\mathcal{T}$. These PMFs
evolve in a deterministic manner, similar to (95) for
$\Pi_{t}^{\psi^{\sigma}},t\in\mathcal{T}$, in the model of this section.

For this special case with no common observations, Theorem 2 becomes

######  Corollary 1.

Consider a SIB strategy profile $\sigma=\\{\sigma_{t},t\in\mathcal{T}\\}$ and
the corresponding update rule
$\psi^{\sigma}=\\{\psi^{\sigma}_{t},t\in\mathcal{T}\\}$ defined by (96)-(97)
for the model of this section. Define

$\displaystyle V_{T+1}^{i}(\cdot,\cdot)=0\text{ for all }i$ (99)
$\displaystyle
V^{i}_{t}(\pi^{\psi^{\sigma}_{t}},s^{i}_{t})=\operatorname{\mathbb{E}}^{\sigma_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}_{t}})}\mid
s^{i}_{t}]$ (100)

where
$U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}=u^{i}_{t}(X_{t},A_{t})+V^{i}_{t+1}(\psi^{\sigma}_{t+1}(\pi^{\psi^{\sigma}}_{t}),S^{i}_{t+1})$,
and in the conditional expectation
$\operatorname{\mathbb{E}}^{\sigma_{t},\psi^{\sigma}}[\cdot]$, the
distribution of $(X_{t},S_{t})$ conditioned on $S^{i}_{t}$ is given by
$\pi^{\psi^{\sigma},i}_{t}(x_{t},s^{-i}_{t})$, $A^{i}_{t},i\in\mathcal{N}$,
are generated by $\sigma^{i}_{t}(a^{i}_{t}\mid
s^{i}_{t},\pi^{\psi^{\sigma}}_{t})$, $S^{i}_{t+1}$ conditioned on
$(X_{t},S_{t},A_{t})$ follows the conditional probability
$\sum_{x_{t+1},s^{-i}_{t+1}}\operatorname{\mathbb{P}}(x_{t+1},s_{t+1}\mid
x_{t},s_{t},a_{t})$ given by

$\displaystyle\operatorname{\mathbb{P}}(x_{t+1},s_{t+1}\mid
x_{t},s_{t},a_{t})$ $\displaystyle=$
$\displaystyle\sum_{y_{t+1}}\mathbb{P}\\{x_{t+1}\mid
x_{t},a_{t}\\}\mathbb{P}\\{y_{t+1}\mid
x_{t+1},a_{t}\\}\left(\prod_{j}\mathbbm{1}\\{s_{t+1}^{j}=\phi_{t+1}^{j}(s_{t}^{j},y_{t+1}^{j},a_{t}^{j})\\}\right).$
(101)

If for all $t\in\mathcal{T}$, there is a SIB strategy profile
$\hat{\sigma}_{t}$ such that $\hat{\sigma}_{t}$ is a BNE of the stage-game
$G_{t}(V_{t+1},\pi^{\psi^{\sigma}_{t}})$, that is,

$\displaystyle\operatorname{\mathbb{E}}^{\hat{\sigma}^{i}_{t},\hat{\sigma}^{-i}_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]=\max_{\tilde{\sigma}^{i}_{t}\in\Lambda^{i}_{t}}\operatorname{\mathbb{E}}^{\tilde{\sigma}^{i}_{t},\hat{\sigma}^{-i}_{t},\psi^{\sigma}}[U^{i}_{G_{t}(V_{t+1},\pi^{\psi^{\sigma}}_{t})}\mid
s^{i}_{t}]$ (102)

for all $i\in\mathcal{N}$, and

$\displaystyle\hat{\sigma}_{t}=\sigma_{t},$ (103)

then the SIB strategy profile $\sigma$ is a SIB-BNE of the dynamic game
without common observations defined in this section.

######  Remark 12.

The SIB-BNE strategy profiles $\\{\sigma_{t},t\in\mathcal{T}\\}$ determined by
sequential decomposition in Corollary 1, along with the beliefs
$\\{\Pi^{\psi^{\sigma}}_{t},t\in\mathcal{T}\\}$ are also Perfect Bayesian
Equilibria (PBE) Fudenberg and Tirole (1991). This is true because
$\\{\sigma_{t},t\in\mathcal{T}\\}$ satisfy sequential rationality (Eq. (102))
and consistency holds because the beliefs
$\\{\Pi^{\psi^{\sigma}}_{t},t\in\mathcal{T}\\}$ are always updated by Bayes
rule.

##  8 Conclusion

We considered stochastic dynamic games where the underlying system is dynamic,
the strategic agents’ actions are hidden (not observable) and their
information is asymmetric. We presented an approach for the computation of BNE
strategy profiles that are based on a compressed version of the agents’
information and can be determined sequentially in time moving backwards, if
each step of this backward procedure has a solution. The approach highlights:
(i) the importance of common information/common knowledge in identifying BNE
strategy profiles that can be sequentially computed; (ii) the difference
between common information that is sufficient for decision-making purposes in
games and common information that is sufficient for decision-making purposes
in teams. The difference is due to the fact that agents have an incentive to
deviate from their predicted strategies in games whereas they don’t have such
an incentive in teams. As a consqence of this incentive, at each time instant
each agent has his own view/belief of the game’s status based on the common
information, but all these different views/beliefs are common knowledge among
all agents. As a result the CIB belief system is described by the sequence
$\Pi^{\psi}_{1:T}$ specified by Definition 2.

Our investigation focused on determining SIB-BNE strategy profiles for the
games under consideration. We note that the SIB-BNE strategy profiles
determined by our methodology are also Perfect Bayesian Equilibrium (PBE)
strategy profiles when the agents have no common observations (i.e., for the
model of Section 7), but this is not true when the agents have common
observations (the general model of Section 2). Determining PBE strategy
profiles for the general model of Section 2 is an interesting problem worthy
of investigation.

###  .1 Sufficient Information

We compare conditions (i)-(iii) of Definition 1 to the conditions of
Definition 2 in Tavafoghi et al (2022); for ease of readability, we include
the definition from Tavafoghi et al (2022) below.

######  Definition 7 (Sufficient private information Tavafoghi et al (2022)).

We say $S_{t}^{i}=\zeta_{t}^{i}(P_{t}^{i},C_{t}\mathchar
24635\relax\;g_{1:t-1})$, $i\in\mathcal{N}$, $t\in\mathcal{T}$, is sufficient
private information for the agents if,

  1. (i)

it can be updated recursively as

$\displaystyle S_{t}^{i}=\phi_{t}^{i}(S_{t-1}^{i},H_{t}^{i}\backslash
H_{t-1}^{i}\mathchar 24635\relax\;g_{1:t-1})\text{ for
}t\in\mathcal{T}\backslash\\{1\\},$ (104)

  2. (ii)

for any strategy profile $g$ and for all realizations
$\\{c_{t},p_{t},p_{t+1},z_{t+1},a_{t}\\}\in\mathcal{C}_{t}\times\mathcal{P}_{t}\times\mathcal{P}_{t+1}\times\mathcal{Z}_{t+1}$
of positive probability,

$\displaystyle\mathbb{P}^{g_{1:t}}\left\\{s_{t+1},z_{t+1}\mid
p_{t},c_{t},a_{t}\right\\}=\mathbb{P}^{g_{1:t}}\left\\{s_{t+1},z_{t+1}\mid
s_{t},c_{t},a_{t}\right\\},$ (105)

where $s_{\tau}^{1:N}=\zeta_{\tau}^{1:N}(p_{\tau}^{1:N},c_{\tau}\mathchar
24635\relax\;g_{1:\tau-1})$ for $\tau\in\mathcal{T}$;

  3. (iii)

for every strategy profile $\tilde{g}$ of the form
$\tilde{g}:=\\{\tilde{g}^{i}_{t}:\mathcal{S}_{t}^{i}\times\mathcal{C}_{t}\rightarrow\Delta(\mathcal{A}_{t}^{i}),i\in\mathcal{N},t\in\mathcal{T}\\}$
and $a_{t}\in\mathcal{A}_{t}$, $t\in\mathcal{T}$;

$\displaystyle\mathbb{E}^{\tilde{g}_{1:t-1}}\left\\{u_{t}^{i}(X_{t},A_{t})\mid
c_{t},p_{t}^{i},a_{t}\right\\}=\mathbb{E}^{\tilde{g}_{1:t-1}}\left\\{u_{t}^{i}(X_{t},A_{t})\mid
c_{t},s_{t}^{i},a_{t}\right\\},$ (106)

for all realizations
$\\{c_{t},p_{t}^{i}\\}\in\mathcal{C}_{t}\times\mathcal{P}_{t}^{i}$ of positive
probability where
$s_{\tau}^{1:N}=\zeta_{\tau}^{1:N}(p_{\tau}^{1:N},c_{\tau}\mathchar
24635\relax\;\tilde{g}_{1:\tau-1})$ for $\tau\in\mathcal{T}$;

  4. (iv)

given an arbitrary strategy profile $\tilde{g}$ of the form
$\tilde{g}:=\\{\tilde{g}^{i}_{t}:\mathcal{S}_{t}^{i}\times\mathcal{C}_{t}\rightarrow\Delta(\mathcal{A}_{t}^{i}),i\in\mathcal{N},t\in\mathcal{T}\\}$,
$i\in\mathcal{N}$, and $t\in\mathcal{T}$,

$\displaystyle\mathbb{P}^{\tilde{g}_{1:t-1}}\left\\{s_{t}^{-i}\mid
p_{t}^{i},c_{t}\right\\}=\mathbb{P}^{\tilde{g}_{1:t-1}}\left\\{s_{t}^{-i}\mid
s_{t}^{i},c_{t}\right\\},$ (107)

for all realizations
$\\{c_{t},p_{t}^{i}\\}\in\mathcal{C}_{t}\times\mathcal{P}_{t}^{i}$ of positive
probability where
$s_{\tau}^{1:N}=\zeta_{\tau}^{1:N}(p_{\tau}^{1:N},c_{\tau}\mathchar
24635\relax\;\tilde{g}_{1:\tau-1})$ for $\tau\in\mathcal{T}$.

Condition (i) of Definition 1 appears in the definition of $S^{i}_{t}$ in
Definition 7, and condition (ii) of Definition 1 on recursive update is the
same as condition (i) in Definition 7. Condition (iii) of Definition 1
directly leads to (iii) and (iv) of Definition 7; the utility
$u_{t}^{i}(X_{t},A_{t})$ in condition (iii) and the random variable
$s^{-i}_{t}$ in condition $(iv)$ of Definition 7 are functions of
$(x_{t},s_{t})$ whose distribution conditioned on $(p^{i}_{t},c_{t})$ is the
same as conditioned on $(s^{i}_{t},c_{t})$ under condition (iii) of Definition
1.

However, condition (ii) of Definition 7 may not hold for sufficient private
information satisfying Definition 1. Consider the following example. Suppose
$X_{1}=Y^{1}_{1}\text{ XOR }Y^{2}_{1}$, and $Y^{1}_{1},Y^{2}_{1}$ takes values
in $\\{0,1\\}$ with equal probability. $Z_{1}=\emptyset$ and $Z_{2}=X_{1}$.
Then $S^{1}_{1}=S^{2}_{1}=\emptyset$ satisfies Definition 1 because
$\operatorname{\mathbb{P}}(x_{1},s^{-i}_{1}\mid
p^{i}_{1},c_{1})=\operatorname{\mathbb{P}}(x_{1}\mid
y^{i}_{1})=0.5=\operatorname{\mathbb{P}}(x_{1},s^{-i}_{1}\mid
s^{i}_{1},c_{1})$. However, they don’t satisfy condition (ii) of Definition 7
because $\operatorname{\mathbb{P}}(z_{2}\mid
p_{1},c_{1},a_{1})=\operatorname{\mathbb{P}}(x_{1}\mid
y^{1}_{1},y^{2}_{1})=\mathds{1}(x_{1}=y^{1}_{1}\text{ XOR
}y^{2}_{1})\neq\operatorname{\mathbb{P}}(z_{2}\mid
s_{1},c_{1},a_{1})=\operatorname{\mathbb{P}}(x_{1})=0.5$.

###  .2 Proof of the generalized better reply secure property for the
augmented stage-game

We show that when $c>24$ the augmented stage-game $\hat{G}_{1}$ in Section 6
is generalized better reply secure. For that matter, we set
$\beta^{*}(q)=\mathds{1}(q\leq 1/3)$ and consider the following five cases.

  1. Cise (i)

$r^{0}_{1}(\bar{\alpha},\bar{q})\neq 0$. In this case Bayes’ rule doesn’t hold
at $(\bar{\alpha},\bar{q})$. We focus on agent $0$ and select the belief to
satisfy Bayes’ rule as follows:

$\displaystyle\phi^{0}(\tilde{\alpha},\tilde{q})=(\tilde{\alpha}_{2}p+\tilde{\alpha}_{1}(1-p),\tilde{\alpha}_{2}(1-p)+\tilde{\alpha}_{1}p)$
(108)

Then this $\phi^{0}$ is a closed correspondence. From this construction of
$\phi^{0}$, we can pick $\epsilon>0$ such that

$\displaystyle
r^{0}_{1}(\tilde{\alpha},\phi^{0}(\tilde{\alpha},\tilde{q}))=0>r^{0}_{1}(\bar{\alpha},\bar{q})+\epsilon$

  2. Ciise (ii)

$r^{0}_{1}(\bar{\alpha},\bar{q})=0$, and $\bar{\pi}_{-1}\neq 1/3$ and
$\bar{\pi}_{1}\neq 1/3$.

Since $\beta^{*}(q)=1$ if $q<1/3$, $\beta^{*}(q)=0$ if $q>1/3$,
$\beta^{*}(\cdot)$ is continuous at points where $q\neq 1/3$. Hence, we can
find $\epsilon>0$ s.t. $\beta^{*}(\tilde{q}_{-1})=\beta^{*}(\bar{q}_{-1})$ for
all $\tilde{q}_{-1}\in(\bar{q}_{-1}-\epsilon,\bar{q}_{-1}+\epsilon)$, and
$\beta^{*}(\tilde{q}_{1})=\beta^{*}(\bar{q}_{1})$ for all
$\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$. In this region
we have

$\displaystyle r^{A}_{1}(\alpha,\tilde{q})=r^{A}_{1}(\alpha,\bar{q})$ (109)

for all $\alpha$. Let

$\displaystyle\phi^{A}(\tilde{\alpha},\tilde{q})=\operatorname*{arg\,max\,}_{\alpha}r^{A}_{1}(\alpha,\tilde{q})$
(110)

Because $r^{A}_{1}(\cdot)$ is continuous in the region under consideration,
$\phi^{A}(\cdot)$ has a closed graph from Berge’s maximum theorem. Note that
for all $\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$,
$\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$

$\displaystyle
r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})=\max_{\alpha}r^{A}_{1}(\alpha,\tilde{q})=\max_{\alpha}r^{A}_{1}(\alpha,\bar{q})$
(111)

If $\max_{\alpha}r^{A}_{1}(\alpha,\bar{q})>r^{A}_{1}(\bar{\alpha},\bar{q})$ we
can find $\epsilon>0$ such that for
$\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$,
$\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$,
$r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})=\max_{\alpha}r^{A}_{1}(\alpha,\tilde{q})\geq
r^{A}_{1}(\bar{\alpha},\bar{q})+\epsilon$.

If $\max_{\alpha}r^{A}_{1}(\alpha,\bar{q})=r^{A}_{1}(\bar{\alpha},\bar{q})$,
then Alice has no profitable deviation. Furthermore, since
$r^{0}_{1}(\bar{\alpha},\bar{q})=0$, agent $0$ has no profitable deviation.
Consequently, $(\bar{\alpha},\bar{q})$ is an equilibrium if
$max_{\alpha}r^{A}_{1}(\alpha,\bar{q})=r^{A}_{1}(\bar{\alpha},\bar{q})$.

  3. Ciiise (iii)

$r^{0}_{1}(\bar{\alpha},\bar{q})=0$, $\bar{\pi}_{-1}=1/3$ and
$\bar{\pi}_{1}\neq 1/3$.

Note that $\bar{q}_{-1}=0.8\bar{\alpha}_{1}+0.2\bar{\alpha}_{2}=1/3$ and
$\beta^{*}(\bar{q}_{-1})=1/3$. Since $\bar{\pi}_{1}\neq 1/3$, we can find
$\epsilon>0$ s.t. $\beta^{*}(\tilde{q}_{1})=\beta^{*}(\bar{q}_{1})$ for all
$\tilde{q}_{1}\in(\bar{q}_{1}-\epsilon,\bar{q}_{1}+\epsilon)$. Therefore,

$\displaystyle
r^{A}_{1}(\bar{\alpha},\bar{q})=0.5c(1-\bar{\alpha}_{1}+\bar{\alpha}_{2})+0.5(2-\bar{\alpha}_{1}-\bar{\alpha}_{2})+0.5(3\bar{q}_{1}-1)\beta^{*}(\bar{q}_{1})$
(112)

Pick for Alice

$\displaystyle\phi^{A}(\tilde{\alpha},\tilde{q})=(0,1)$ (113)

for all
$\tilde{\alpha}_{i}\in(\bar{\alpha}_{i}-\epsilon,\bar{\alpha}_{i}+\epsilon),i=1,2$,
$\tilde{q}_{i}\in(\bar{q}_{i}-\epsilon,\bar{q}_{i}+\epsilon),i=-1,1$. We get

$\displaystyle r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})=$
$\displaystyle
c+0.5+0.5(0.6-1)\beta^{*}(\tilde{q}_{-1})+0.5(2.4-1)\beta^{*}(\tilde{q}_{-1})$
$\displaystyle=$ $\displaystyle
c+0.5-0.2\beta^{*}(\tilde{q}_{-1})+0.7\beta^{*}(\bar{q}_{-1})$ (114)

and

$\displaystyle
r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})-r^{A}_{1}(\bar{\alpha},\bar{q})-\epsilon$
$\displaystyle=$ $\displaystyle
0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})-0.5(1+\bar{\alpha}_{1}+\bar{\alpha}_{2})$
$\displaystyle-0.2\beta^{*}(\tilde{q}_{-1})+0.5(2.4-3\bar{q}_{1})\beta^{*}(\bar{q}_{-1})-\epsilon$
$\displaystyle\geq$ $\displaystyle
0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})-0.5*3-0.2-0.5*0.6-\epsilon$ (115)

When $\bar{q}_{-1}=1/3$, then
$0.8\bar{\alpha}_{1}+0.2\bar{\alpha}_{2}=1/3\Rightarrow\bar{\alpha}_{1}=5/12-3/12\bar{\alpha}_{2}$.
Therefore,

$\displaystyle
1+\bar{\alpha}_{1}-\bar{\alpha}_{2}=17/12-15/12\bar{\alpha}_{2}\geq 1/6$ (116)

where the minimum is at $\bar{\alpha}_{1}=1/6$ and $\bar{\alpha}_{2}=1$.

When $c>24$, then

$\displaystyle 0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})\geq c/12>2$ (117)

and
$r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})-r^{A}_{1}(\bar{\alpha},\bar{q})-\epsilon>0$.

  4. Civse (iv)

$r^{0}_{1}(\bar{\alpha},\bar{q})=0$, and $\bar{\pi}_{1}=1/3$ and
$\bar{\pi}_{-1}\neq 1/3$.

This case is similar to case (iii). Since $\bar{\pi}_{-1}\neq 1/3$, we can
find $\epsilon>0$ s.t. $\beta^{*}(\tilde{q}_{-1})=\beta^{*}(\bar{q}_{-1})$ for
all $\tilde{q}_{-1}\in(\bar{q}_{-1}-\epsilon,\bar{q}_{-1}+\epsilon)$.
Furthermore,

$\displaystyle r^{A}_{1}(\bar{\alpha},\bar{q})$ $\displaystyle=$
$\displaystyle
0.5c(1-\bar{\alpha}_{1}+\bar{\alpha}_{2})+0.5(2-\bar{\alpha}_{1}-\bar{\alpha}_{2})+0.5(3\bar{q}_{-1}-1)\beta^{*}(\bar{q}_{-1})$
(118)

Pick for Alice the closed correspondence (as in case (iii))

$\displaystyle\phi^{A}(\tilde{\alpha},\tilde{q})=(0,1)$ (119)

for all
$\tilde{\alpha}_{i}\in(\bar{\alpha}_{i}-\epsilon,\bar{\alpha}_{i}+\epsilon),i=1,2$,
$\tilde{q}_{i}\in(\bar{q}_{i}-\epsilon,\bar{q}_{i}+\epsilon),i=-1,1$. Then

$\displaystyle r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})$
$\displaystyle=$ $\displaystyle
c+0.5-0.2\beta^{*}(\bar{q}_{-1})+0.7\beta^{*}(\tilde{q}_{-1})$ (120)

and

$\displaystyle
r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})-r^{A}_{1}(\bar{\alpha},\bar{q})-\epsilon$
$\displaystyle=$ $\displaystyle
0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})-0.5(1+\bar{\alpha}_{1}+\bar{\alpha}_{2})$
$\displaystyle+0.5(0.6-3\bar{q}_{-1})\beta^{*}(\bar{q}_{-1})+0.7\beta^{*}(\tilde{q}_{-1})-\epsilon$
$\displaystyle\geq$ $\displaystyle
0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})-0.5*3-0.5*2.4-\epsilon$ (121)

When $\bar{q}_{1}=1/3$,
$0.2\bar{\alpha}_{1}+0.8\bar{\alpha}_{2}=1/3\Rightarrow\bar{\alpha}_{2}=5/12-3/12\bar{\alpha}_{1}$.
Therefore,

$\displaystyle
1+\bar{\alpha}_{1}-\bar{\alpha}_{2}=7/12+15/12\bar{\alpha}_{1}\geq 7/12.$
(122)

When $c>24$, then

$\displaystyle 0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})\geq 7/24c>2.7$ (123)

and
$r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})-r^{A}_{1}(\bar{\alpha},\bar{q})-\epsilon>0$.

  5. Cvse (v)

$r^{0}_{1}(\bar{\alpha},\bar{q})=0$, and $\bar{\pi}_{1}=1/3$ and
$\bar{\pi}_{-1}=1/3$.

We have

$\displaystyle
r^{A}_{1}(\bar{\alpha},\bar{q})=0.5c(1-\bar{\alpha}_{1}+\bar{\alpha}_{2})+0.5(2-\bar{\alpha}_{1}-\bar{\alpha}_{2})$
(124)

Pick for Alice the closed correspondence (as in cases (iii) and (iv))

$\displaystyle\phi^{A}(\tilde{\alpha},\tilde{q})=(0,1)$ (125)

for all
$\tilde{\alpha}_{i}\in(\bar{\alpha}_{i}-\epsilon,\bar{\alpha}_{i}+\epsilon),i=1,2$,
$\tilde{q}_{i}\in(\bar{q}_{i}-\epsilon,\bar{q}_{i}+\epsilon),i=-1,1$. Then

$\displaystyle
r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})-r^{A}_{1}(\bar{\alpha},\bar{q})-\epsilon$
$\displaystyle=$ $\displaystyle
0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})-0.5(1+\bar{\alpha}_{1}+\bar{\alpha}_{2})-0.2\beta^{*}(\tilde{q}_{-1})+0.7\beta^{*}(\tilde{q}_{-1})-\epsilon$
$\displaystyle\geq$ $\displaystyle
0.5c(1+\bar{\alpha}_{1}-\bar{\alpha}_{2})-0.5*3-0.2-\epsilon$ (126)

Then we have
$r^{A}_{1}(\phi^{A}(\tilde{\alpha},\tilde{q}),\tilde{q})-r^{A}_{1}(\bar{\alpha},\bar{q})-\epsilon>0$
following the steps in (iv).

## References

  *   * Aumann et al (1995) Aumann R, Maschler M, Stearns R (1995) Repeated games with incomplete information. MIT press
  * Barelli and Meneghel (2013) Barelli P, Meneghel I (2013) A note on the equilibrium existence problem in discontinuous games. Econometrica 81(2):813–824
  * Cardaliaguet et al (2015) Cardaliaguet P, Rainer C, Rosenberg D, et al (2015) Markov games with frequent actions and incomplete information—the limit case. Mathematics of Operations Research 41(1):49–71
  * Escobar and Toikka (2013) Escobar J, Toikka J (2013) Efficiency in games with Markovian private information. Econometrica 81(5):1887–1934
  * Forges (1992) Forges F (1992) Repeated games of incomplete information: non-zero-sum. Handbook of Game Theory 1:109–154
  * Fudenberg and Tirole (1991) Fudenberg D, Tirole J (1991) Game theory. 1991. Cambridge, Massachusetts
  * Gensbittel and Renault (2015) Gensbittel F, Renault J (2015) The value of Markov chain games with incomplete information on both sides. Mathematics of Operations Research 40(4):820–841
  * Gupta et al (2014) Gupta A, Nayyar A, Langbort C, et al (2014) Common information based Markov perfect equilibria for linear-Gaussian games with asymmetric information. SIAM J Control Optim 52(5):3228–3260
  * Gupta et al (2016) Gupta A, Langbort C, Başar T (2016) Dynamic games with asymmetric information and resource constrained players with applications to security of cyberphysical systems. IEEE Transactions on Control of Network Systems 4(1):71–81
  * Ho (1980) Ho Y (1980) Team decision theory and information structures. Proceedings of the IEEE 68(6):644–654
  * Hörner et al (2011) Hörner J, Sugaya T, Takahashi S, et al (2011) Recursive methods in discounted stochastic games: An algorithm for $\delta\rightarrow 1$ and a folk theorem. Econometrica 79(4):1277–1318
  * Kartik and Nayyar (2021) Kartik D, Nayyar A (2021) Upper and lower values in zero-sum stochastic games with asymmetric information. Dynamic Games and Applications 11(2):363–388
  * Kumar and Varaiya (1986) Kumar P, Varaiya P (1986) Stochastic Systems: Estimation Identification and Adaptive Control. Prentice-Hall, Inc.
  * Li and Shamma (2014) Li L, Shamma J (2014) Lp formulation of asymmetric zero-sum stochastic games. In: 53rd IEEE conference on decision and control, IEEE, pp 1930–1935
  * Li and Shamma (2017) Li L, Shamma J (2017) Efficient strategy computation in zero-sum asymmetric repeated games. arXiv preprint arXiv:170301952
  * Li et al (2017) Li L, Langbort C, Shamma J (2017) Solving two-player zero-sum repeated Bayesian games. arXiv preprint arXiv:170301957
  * Mailath and Samuelson (2006) Mailath G, Samuelson L (2006) Repeated Games and Reputations. Oxford university press Oxford
  * Nayyar et al (2011) Nayyar A, Mahajan A, Teneketzis D (2011) Optimal control strategies in delayed sharing information structures. IEEE Transactions on Automatic Control 56(7):1606–1620
  * Nayyar et al (2013a) Nayyar A, Gupta A, Langbort C, et al (2013a) Common information based markov perfect equilibria for stochastic games with asymmetric information: Finite games. IEEE Transactions on Automatic Control 59(3):555–570
  * Nayyar et al (2013b) Nayyar A, Mahajan A, Teneketzis D (2013b) Decentralized stochastic control with partial history sharing: A common information approach. IEEE Transactions on Automatic Control 58(7):1644–1658
  * Nayyar et al (2014) Nayyar A, Gupta A, Langbort C, et al (2014) Common information based Markov perfect equilibria for stochastic games with asymmetric information: Finite games. IEEE Transactions on Automatic Control 59(3):555–570. 10.1109/TAC.2013.2283743
  * Ouyang et al (2015) Ouyang Y, Tavafoghi H, Teneketzis D (2015) Dynamic oligopoly games with private Markovian dynamics. In: 54th IEEE Conference on Decision and Control (CDC)
  * Ouyang et al (2017) Ouyang Y, Tavafoghi H, Teneketzis D (2017) Dynamic games with asymmetric information: Common information based perfect Bayesian equilibria and sequential decomposition. IEEE Transactions on Automatic Control
  * Renault (2006) Renault J (2006) The value of Markov chain games with lack of information on one side. Math Oper Res 31(3):490–512
  * Renault (2012) Renault J (2012) The value of repeated games with an informed controller. Mathematics of Operations Research 37(1):154–179
  * Sinha and Anastasopoulos (2016) Sinha A, Anastasopoulos A (2016) Structured perfect Bayesian equilibrium in infinite horizon dynamic games with asymmetric information. American Control Conference
  * Sugaya (2012) Sugaya T (2012) Efficiency in Markov games with incomplete and private information. working paper
  * Tang et al (2022) Tang D, Tavafoghi H, Subramanian V, et al (2022) Dynamic games among teams with delayed intra-team information sharing. Dynamic Games and Applications pp 1–59
  * Tavafoghi et al (2016) Tavafoghi H, Ouyang Y, Teneketzis D (2016) On stochastic dynamic games with delayed sharing information structure. In: 55th IEEE Conference on Decision and Control (CDC), pp 7002–7009
  * Tavafoghi et al (2022) Tavafoghi H, Ouyang Y, Teneketzis D (2022) A unified approach to dynamic decision problems with asymmetric information: Nonstrategic agents. IEEE Transactions on Automatic Control 67(3):1105–1119. 10.1109/TAC.2021.3060835
  * Vasal and Anastasopoulos (2016) Vasal D, Anastasopoulos A (2016) Signaling equilibria for dynamic LQG games with asymmetric information. In: 55th IEEE Conference on Decision and Control (CDC), pp 6901–6908
  * Witsenhausen (1973) Witsenhausen HS (1973) A standard form for sequential stochastic control. Mathematical Systems Theory 7(1):5–11
  * Zamir (1992) Zamir S (1992) Repeated games of incomplete information: Zero-sum. Handbook of Game Theory 1:109–154
  * Zheng and Castañón (2013) Zheng J, Castañón DA (2013) Decomposition techniques for markov zero-sum games with nested information. In: 52nd IEEE conference on decision and control, IEEE, pp 574–581