# Multi-Agent Actor-Critic
with Generative Cooperative Policy Network

Heechang Ryu
KAIST Hayong Shin
KAIST Jinkyoo Park
KAIST All authors are with the Department of Industrial and Systems
Engineering, Korea Advanced Institute of Science and Technology (KAIST),
Daejeon, Republic of Korea (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>jinkyoo.park@kaist.ac.kr).

###### Abstract

We propose an efficient multi-agent reinforcement learning approach to derive
equilibrium strategies for multi-agents who are participating in a Markov
game. Mainly, we are focused on obtaining decentralized policies for agents to
maximize the performance of a collaborative task by all the agents, which is
similar to solving a decentralized Markov decision process. We propose to use
two different policy networks: (1) decentralized greedy policy network used to
generate greedy action during training and execution period and (2) generative
cooperative policy network (GCPN) used to generate action samples to make
other agents improve their objectives during training period. We show that the
samples generated by GCPN enable other agents to explore the policy space more
effectively and favorably to reach a better policy in terms of achieving the
collaborative tasks.

##  1 Introduction

As modern engineering systems are composed of multiple sub-components
interacting with each other in a complex way, it becomes challenging to
understand the behavior of the system and operate it efficiently. Multi-agent
system modeling is a useful tool for modeling the complex and collective
behavior of the target system by modeling subsystems interacting with each
other. The control problem of a multi-agent system has drawn much attention in
the past years, due to the practical and potential applications in autonomous
vehicles [1, 2], robotics [3, 4], logistics [5], power grid [6], and resource
management [7]. Since the agents in multi-agent system lack full knowledge on
dynamic environment and other agents’ strategies, learning decision-making
strategy, policy, in a multi-agent system is much more challenging than in a
single agent system.

Deriving control policies for multi-agent system can be generally described as
Markov Game (MG), an extension of Markov Decision Process (MDP) to a multi-
agent system. In MG, given the global state, each agent takes an independent
action to maximize its own payoff (i.e., accumulated reward), while
interacting with other agents. The interactions imply that the payoff of an
individual agent and the evolution of global state are determined by the joint
action of all the agents given a specific joint state. Depending on the
relationships among the agents’ payoff, different equilibrium concepts are
employed to agents’ equilibrium policy. However, it is generally difficult to
derive analytically equilibrium strategies for a general MG because the
optimality concept does not apply anymore in MG.

Many researchers have employed Reinforcement learning (RL) [8] approaches to
derive policies for agents in various MG with different equilibrium concepts.
This field of study is referred to as multi-agent reinforcement learning
(MARL). For example, [9] proposed minimax Q-learning to solve the two-player
zero-sum MG. [10] proposed Nash Q-learning to derive the Nash equilibrium
strategies for two-player MG. [11] proposed friend-or-foe Q-learning to
resolve convergence issues of Nash Q-learning. In addition, [12] employed
correlated equilibrium concept to derive the equilibrium strategies. These
algorithms require each agent to compute equilibrium values of a stage-game
defined at each state and use the computed equilibrium values to update the Q
values of each agent. Solving the equilibrium value defined in each state
means that this algorithm requires each agent to make decision while
maintaining certain assumptions on other agents. This approach works only for
very simple systems with discrete and low dimensional state and action space
and usually does not guarantee the convergence since all the agents are
learning concurrently using the data observed during game playing.

On the other hand, in a single agent system, deep reinforcement learning (DRL)
algorithm has shown great successes in many complex tasks and even superior
performances to human in video games [13], robotics [14], and Go game [15,
16]. Motivated by such big successes in DRL, many researchers start to employ
DRL approaches to controlling multi-agent systems, which is referred to as
deep MARL. A deep neural network function approximator [17] has shown good
potential in effectively modeling complex interactions among multiple agents
while incorporating dynamics and stochastic environment. However, due to the
difficulty of imposing equilibrium concept among other agents, the majority of
Deep MARL algorithms are designed to solve cooperative or team game, where the
joint policy can be derived based on optimality principle, not equilibrium
concept. This approach seeks a decentralized policy of each agent to achieve
global objective when deriving a centralized policy is prohibited due to
problem complexity, and this problem is equivalent to solving decentralized
MDP (Dec-MDP) [18, 19, 20, 21]. The core part of this problem is to design a
consensus mechanism that allows each agent to understand the interactions
among other agents by looking even partial state.

To make all the agents coordinate effectively in Dec-MDP with Deep MARL,
different consensus mechanisms have been employed. The first approach is to
train centralized Q-network capturing the relationships among agents’ actions
and the global return. This approach uses the global Q-network to derive
decentralized policies for agents. For example, [21] proposed a counterfactual
actor-critic algorithm to derive the decentralized policy when the globally
shared return is used. [22] also proposed similar approach but they considered
more complex situation where each agent need to infer the global return (i.e.,
the sum of return from all agents) from individually received return. To
inference the global return based on local and individual return, [22]
employed parameter sharing among neighboring agents in the networked agents.

To model the interactions among agents, an approach of learning other agents’
policy has also been proposed. For example, [23] proposed multi-agent version
of deep deterministic policy gradient algorithm to derive agents’ policies for
both cooperative and competitive MG. This approach allows each agent to train
individual Q-network with its own individual reward while using other agents’
actions estimated from the policy networks for other agents. To infer other
agents’ actions, each agent learns other agent’s policy during the training
period. Although the authors insist that the algorithm can derive
decentralized strategies for non-cooperative MG, this approach has limited
performance because this algorithm does not guarantee to reach equilibrium
strategy when solving non-cooperative game. That is, this algorithm just tries
to find best response strategy given the other agents’ strategies trained from
the data, without making sure that the trained policy networks are consistent
among agents, the necessary condition for Nash equilibrium in non-cooperative
game.

In this paper, we propose a multi-agent actor-critic algorithm with generative
cooperative policy network (GCPN). This algorithm employing individual actor
and critic networks can utilize individual reward for deriving a decentralized
policy for each agent. This algorithm is mainly focused on improving the
cooperation among agents in achieving a collaborative task. The cooperative
policy can also be derived in the previously proposed algorithm [23] by
setting the identical (shared) reward for all the agents, but its performance
is limited due to the failure of modeling contribution by each agent [24]. To
resolve this issue, we introduce GCPN that is trained to increase not its own
return but other agent’s return. The generated samples from GCPN are then used
to train individual actor and critic networks. We experimentally show that the
samples generated by GCPN induce more consistent and efficient collaborative
behaviors among agents.

##  2 Background

Generally, partially-observable Markov decision process (POMDP) have been
widely used to model a single agent’s dynamic decision making under stochastic
environment. In this paper, we consider partially-observable Markov game
(POMG), an extension of POMDP to multi-agent system, to model the interaction
of multiple agents with the environment. $\mathnormal{N}$-agents POMG is
defined as follows; $\mathcal{S}$ describes a set of state for entire system.
$\mathcal{O}_{i}$ is a set of observation for agent ${i}$, and each agent
acquires a private and limited observation from the state through its own
observation function ${o_{i}}:\mathcal{S}\mapsto\mathcal{O}_{i}$.
$\mathcal{A}_{i}$ is a set of action for agent ${i}$. Each agent has a
deterministic policy $\mu_{\theta_{i}}:\mathcal{O}_{i}\mapsto\mathcal{A}_{i}$
that maps the local observation to the action to take. We consider a
deterministic policy that can be learned in a continuous and high-dimensional
action space (Silver et al. 2014) for each agent, the policy that is suitable
for controlling various physical systems. When all the agents take actions,
each agent gets its own reward
${r}_{i}:\mathcal{S}\times\mathcal{A}_{1}\times\dots\times\mathcal{A}_{N}\mapsto\mathbb{R}$
from the environment and the global state $\mathcal{S}$ evolves to the next
state according to the state transition model
$\mathcal{T}:\mathcal{S}\times\mathcal{A}_{1}\times\dots\times\mathcal{A}_{N}\mapsto\mathcal{S}$.
We assume that the initial state is determined by the initial state
distribution $\rho:\mathcal{S}\mapsto[0,1]$. The agent $i$ aims to maximize
its discounted return $R_{i}=\sum_{t=0}^{\infty}\gamma^{t}r_{i}^{t}$, where
$\gamma$ is a discount factor.

###  2.1 Deep Q-Networks (DQN)

Deep Q-Network (DQN), an RL approach to solve MDP [13], aims to find an
optimal policy $\pi$ that maximizes the expected return. DQN approximates the
expected return using a deep neural network as
$Q(s,a;\phi)\approx\mathbb{E}[R\arrowvert s^{t}=s,a^{t}=a]$. The parameter
$\phi$ of $Q(s,a;\phi)$ is then optimized as one that minimizes the loss
$\mathcal{L}$ defined as:

$\mathcal{L(\phi)}={\mathbb{E}_{s,a,r,{s^{\prime}}\sim
D}}[Q(s,a;\phi)-y]^{2},$ (1)

where
$y=r+\gamma{\max_{a^{\prime}}}{Q^{\prime}}({s^{\prime}},{a^{\prime}};\bar{\phi})$
is target computed by target network. In addition, $D$ is experience replay
buffer, where $(s,a,r,{s^{\prime}})$ is stored from the steps of episodes.
Both experience replay buffer and target network are intended to stabilize
learning. Once the optimum network parameter ${\phi^{*}}$ is computed, the
optimal policy can be expressed as
${\pi^{*}}(s)={\arg\\!\max_{a}}Q(s,a;\phi^{*})$.

###  2.2 Deep Deterministic Policy Gradient (DDPG)

A deep deterministic policy gradient (DDPG) [25], as an actor-critic
algorithm, aims to derive directly the deterministic policy network, referred
to as an actor, $a_{t}=\mu_{\theta}(s_{t})$ that maximize the expected return
defined as:

$\mathcal{J(\theta)}=\mathbb{E}_{s\sim\rho,a\sim\mu_{\theta}}[R]\approx\mathbb{E}_{s\sim\rho,a\sim\mu_{\theta}}[Q(s,a;\phi)]$
(2)

The Q-network in Equation 2, referred to as a critic, is optimized like
Equation 1 in DQN. The parameters $\theta$ of the policy network
$\mu_{\theta}(s_{t})$ are optimized using a gradient ascent algorithm with the
computed gradient as follows:

$\nabla_{\theta}\mathcal{J}(\theta)\approx\mathbb{E}_{s\sim
D}[\nabla_{\theta}Q(s,a;\phi){\arrowvert_{a=\mu_{\theta}(s)}}]=\mathbb{E}_{s\sim
D}[\nabla_{\theta}\mu_{\theta}(s){\nabla_{a}}Q(s,a;\phi)\arrowvert_{a=\mu_{\theta}(s)}]$
(3)

The gradient is computed using chain rule as the expected value of the product
between the gradient of Q-network with respect to actions and the gradient of
the policy network with respect to its parameters. It is called a
deterministic policy gradient (DPG) [26]. By iteratively updating the
parameters for the actor and critic networks, the algorithm induces the
optimal policy network $a_{t}^{*}=\mu_{\theta^{*}}(s_{t})$.

###  2.3 Multi-Agent Deep Deterministic Policy Gradient (MADDPG)

Recently, MADDPG [23], an extension of DDPG to multi-agent system, has been
proposed to solve POMG. Instead of optimizing a single actor network as in
DDPG, MADDPG provides a framework that each agent derives an individual policy
network, actor, $a_{i}=\mu_{i}(o_{i};\theta_{i})$ mapping the local
observation $o_{i}$ to the action $a_{i}$ in order to maximize the expected
return that is approximated as an individual Q-network $Q_{i}$, critic. MADDPG
allows each agent to receive its own individual reward signal
$r_{i}(s,a_{1}\dots,a_{N})$. In this setting, the core part of MADDPG are (1)
to learn each player’s critic network with local reward $r_{i}$ and (2) to
derive the decentralized actor network using the individual critic network.

The parameter $\phi_{i}$ of Q-network $Q_{i}$ for each agent $i$ is optimized
as minimizing the loss $\mathcal{L}_{i}$:

$\mathcal{L}_{i}(\phi_{i})={\mathbb{E}_{\mathbf{o},a,r,{\mathbf{o}^{\prime}}\sim
D}}[(Q_{i}(\mathbf{o},a_{1}\dots,a_{N};\phi_{i})-y_{i})^{2}],$ (4)

where $\mathbf{o}=(o_{1},\dots,o_{N})$ is the observations of all agents and
$y_{i}=r_{i}(\mathbf{o},a_{1},\dots,a_{N})+\gamma{Q^{\prime}_{i}}({\mathbf{o}^{\prime}},{a_{1}^{\prime}},\dots,{a_{N}^{\prime}};{\bar{\phi_{i}}})\arrowvert_{{a^{\prime}_{j}}={\mu_{i}^{j}}({o^{\prime}_{j}})}$
is target computed by critic target network with slowly updating parameter
$\bar{\phi_{i}}$. In addition, the experience replay buffer $D$ contains
$(\mathbf{o},a_{1},\dots,a_{N},r_{1},\dots,r_{N},{\mathbf{o}^{\prime}})$. Note
that, in the next observation ${\mathbf{o}^{\prime}}$, agent $i$ estimates
which action will be taken by other agents by using inferring policy networks
for other agents
${a^{\prime}_{j}}={\mu_{i}^{j}}({o^{\prime}_{j}};{\theta_{i}^{j}})$ that is
being trained by agent $i$ for $j=1,\dots,N$. In short, the individual critic
network captures the influences of other agents’ joint action, affecting the
future accumulated reward of agent $i$.

The parameter $\theta_{i}$ of policy network for agent $i$ is optimized using
a gradient ascent algorithm with the computed gradient as follows:

$\nabla_{\theta_{i}}\mathcal{J}(\theta_{i})=\mathbb{E}_{\mathbf{o},a\sim
D}[\nabla_{\theta_{i}}\mu_{i}(o_{i})\nabla_{a_{i}}Q_{i}(\mathbf{o},a_{1},\dots,a_{N};\phi_{i})\arrowvert_{a_{i}=\mu_{i}(o_{i};\theta_{i})}]$
(5)

Furthermore, for stabilizing learning, each agent has multiple sub-policy
networks for policy ensembles. By iteratively updating the parameters for the
actor and critic networks, the algorithm induces the optimal policy network
$a_{i}^{*}=\mu_{i}^{*}(o_{i})$.

###  2.4 Consensus Mechanism

Decentralized control aims to express a joint policy for all agents as a
product of individual policies of the agents. When decentralized policies are
properly derived, each agent’s local action determined solely based on the
local observation can lead the global equilibrium among agents, i.e.,
coordination for the cooperation game. Consensus mechanism refers to the
procedure of deriving the equilibrium among all the agents’ policies through
their individual goal-seeking behaviors. It means that each agent’s greedy
behavior induces the global cooperation among all the agents for a cooperative
game.

There are several ways of embedding consensus among agents into local policies
of agents during training. Such procedures are referred to as “ centralized
training and decentralized execution”. One of widely used approaches is to
learn other players’ policies and use these policies to estimate the other
players’ actions, the case of MADDPG. As a result, the individual Q-network
and the local policy derived from the Q-network implicitly embed other
players’ behaviors. When such individual policies are used, then the
decentralized agents are subject to coordinate with each other to achieve the
global joint objective.

Another approach is to train a global Q-network to capture influences of the
agents’ joint action on the global return [21, 22] and uses this Q-network to
update the local policy. The counterfactual actor-critic algorithm [21] trains
a single global Q-network, while FDMARL [22] trains the individual Q-network
with individual reward, and makes the Q-networks be equal by forcing them to
share the parameters of the Q-networks among neighboring agents as:

$\phi^{i}\leftarrow\sum_{j\in\mathcal{N}}c(i,j)\cdot\phi^{i}$ (6)

where $\mathcal{N}$ is a set of all agents, and $c(i,j)$ is an element of
doubly stochastic weight matrix for transmitting the information from $i$ to
$j$ in a communication network.

##  3 Methods

We propose a multi-agent actor-critic algorithm with GCPN to derive
collaborative strategies for POMG participants. We are specifically focused on
deriving decentralized policies for agents to maximize the performance of the
collaborative task conducted by all the agents.

Figure 1: MADDPG (left) and MADDPG-GCPN (right).

The proposed algorithm is the variant of MADDPG in that it maintains the
decentralized policy network and the individual Q-network. However, it is
different from MADDPG in that it uses an additional policy network called
GCPN. We refer to this algorithm as MADDPG-GCPN. In MADDPG-GCPN, each agent
$i$ has the GCPN $\mu_{i}^{c}$ to maximize the return of the other agent apart
from the original decentralized policy $\mu_{i}$ that is designed to maximize
its own return. Thus, each agent has two different actor networks: (1) the
behavioral actor network, GCPN, for generating samples and (2) greedy actor
network for choosing actions for maximizing its own return. The difference
between MADDPG and MADDPG-GCPN is shown in Figure 1. To embed the cooperative
behavior among agents into decentralized policy for each agent, we employ a
procedure of centralized training and decentralized execution. The training of
MADDPG-GCPN consists of training the followings:

  * •

Individual Q-network (critic): each agent trains individual Q-network, which
is used to update the individual greedy policy for choosing optimal action
during execution period, and the individual behavioral policy, GCPN, for
generating action samples for exploration during training period.

  * •

Individual greedy policy network: similar to MADDPG, individual policy is
trained to maximize the expected return represented as a individual critic
network.

  * •

Generative cooperative policy network: the GCPN $\mu_{i}^{c}$ generates action
samples to interact the environment during training period for learning the
actor and critic networks..

In execution period after training, the Q-networks and GCPNs of all agents are
removed. Only the individual greedy policy of each agent controls the action
based on local observation.

In MADDPG-GCPN, the parameter $\phi_{i}$ of the individual Q-network of each
agent $i$ is optimized as minimizing the loss $\mathcal{L}_{i}$:

$\mathcal{L}_{i}(\phi_{i})={\mathbb{E}_{\mathbf{o},{a^{c}},r,{\mathbf{o}^{\prime}}\sim{D^{c}}}}[(Q_{i}(\mathbf{o},a_{1}^{c},\dots,a_{N}^{c};\phi_{i})-y_{i})^{2},$
(7)

where
$y_{i}=r_{i}(\mathbf{o},a_{1}^{c},\dots,a_{N}^{c})+\gamma{Q^{\prime}_{i}}({\mathbf{o}^{\prime}},{a_{1}^{c^{\prime}}},\dots,{a_{N}^{c^{\prime}}};{\bar{\phi_{i}}})\arrowvert_{{a_{j}^{c^{\prime}}}={\mu_{j}^{c^{\prime}}({o^{\prime}_{j}})}}$
is the target computed by target network. The experience replay buffer $D^{c}$
contains
$(\mathbf{o},{a_{1}^{c}},\dots,{a_{N}^{c}},r_{1},\dots,r_{N},\mathbf{o^{\prime}})$
where the $a_{j}^{c}$ is generated not from greedy actor but from the GCPN. In
the target network in Equation 7, the actions of other agents, except agent
$i$, are estimated by using additional inferring policy networks
${a_{j}^{c^{\prime}}}={\mu_{j}^{c^{\prime}}}({{o^{\prime}}_{j}})$.

Then, the parameter $\theta_{i}$ of individual greedy policy $\mu_{i}$ of each
agent $i$ is optimized using a gradient ascent algorithm with the computed
gradient as follows:

$\nabla_{\theta_{i}}\mathcal{J}(\theta_{i})=\mathbb{E}_{\mathbf{o},{a^{c}}\sim{D^{c}}}[\nabla_{\theta_{i}}\mu_{i}(o_{i})\nabla_{a_{i}}Q_{i}(\mathbf{o},{a_{1}^{c}},...,{a_{i}},...,{a_{N}^{c}};\phi_{i})\arrowvert_{a_{i}=\mu_{i}(o_{i};\theta_{i})}]$
(8)

The individual greedy policy is trained in a similar way to MADDPG.

The major contributions of this study are introducing additional policy
network for generating samples and suggesting the training method for that
network. The parameter $\theta_{i}^{c}$ of the GCPN $\mu_{i}^{c}$ of each
agent $i$ is optimized using a gradient ascent algorithm with the specially
computed gradient as follows:

$\nabla_{\theta_{i}^{c}}\mathcal{J}(\theta_{i}^{c})=\mathbb{E}_{\mathbf{o},{a^{c}}\sim{D^{c}}}[\nabla_{\theta_{i}^{c}}\mu_{i}^{c}(o_{i})\nabla_{a_{i}^{c}}Q_{-i}(\mathbf{o},{a_{1}^{c}},...,{a_{N}^{c}};\phi_{i})\arrowvert_{a_{i}^{c}=\mu_{i}^{c}(o_{i};\theta_{i}^{c})}]$
(9)

where ${-i}$ is the agents in $N$-agents excluding agent $i$. The policy
gradient for the GCPN $\mu_{i}^{c}$ is computed using (1) the gradient of
Q-network of other agents with respect to actions and (2) the gradient of GCPN
with respect to its parameters. The trained GCPN then tends to generate action
samples that can induce a higher returns of other agents, which is the reason
why we call this as cooperative policy network.

For stabilizing training, the agent $i$ has $\arrowvert-i\arrowvert$-GCPNs,
where $\arrowvert-i\arrowvert$ is the cardinality of ${-i}$, and each GCPN is
trained from one agent of ${-i}$. Since each agent has several GCPNs, it
generates a sample by randomly selecting one of the GCPNs at every step or
considering the GCPN ensembles. In the step of updating the parameters, both
the GCPN and the individual greedy policy are learned while only the GCPNs are
considered in generating samples. Furthermore, there are $K$ different sub-
policies for the GCPN and individual greedy policy. There are experience
replay buffers $D_{i}^{c,k}$ for each sub policy of the GCPNs.

In addition, the consensus mechanism through GCPNs preserves the privacy among
the agents because it uses own local reward and the gradient from the critics
of the other agents during learning and uses only own decentralized policy and
local observations when executing.

##  4 Results

To evaluate the performance of the proposed algorithm, the MARL task in OpenAI
Gym [27] is used as a target POMG. Predator-prey is a task where several
predators can gain higher reward, score, when they cooperate with each other
to catch a prey. Figure 2 illustrates the game environment with three
predators and one prey with two landmark obstacles.

Figure 2: Predator-prey in OpenAI Gym.

In the game, the predators should cooperate with each other to catch the prey
because the prey is faster than predators. The predator-prey game is defined
as the following POMG:

  * •

The partially-observable state containing the relative positions, velocities
of the agents, and the positions of the landmarks.

  * •

The action of each agent is a movement toward all directions.

We consider the two reward function structures:

  * •

Shared reward function ${r_{i}=10}$ if any agent catch the prey, otherwise $0$
for $i=1,2,3$.

  * •

Individual reward function ${r_{i}=10}$ if agent i catch the prey, otherwise
$0$ for $i=1,2,3$.

That is, we consider two POMGs, one with shared reward structure and the other
with individual reward structure. For these two different POMGs, we will
compare the performances of different MARL approaches in terms of maximizing
the winning rate of the predators.

We compare the performances of the following four learning algorithms:

  * •

CF [21]: the MARL actor-critic algorithm designed for a team game with a
shared return for all agents. Each agent who is jointly maximizing its shared
return through single global Q-network directly induces the collaborative
behavior among the agents. Because of this model assumption, CF applies to
only the shared reward case. In this experiment, CF indicates that only single
global Q-network with MADDPG is used.

  * •

FDMARL [22]: the MARL actor-critic algorithm designed for a cooperative game
with individual return for each agent. The algorithm requires each agent to
infer the global return through parameter sharing and uses the inferred global
return to derive the decentralized policy. In this experiment, FDMARL
indicates that only parameter sharing with MADDPG is used.

  * •

MADDPG [23]: the MARL actor-critic algorithm designed for general non-
cooperative game. This algorithm requires each agent to learn other players’
policies and use the trained policies to approximate its own Q-network. In
addition, the individual policy is also trained to interact with the
individually updated Q-network.

  * •

MADDPG-GCPN1: the proposed algorithm with the randomly selected GCPNs in
sample-generating.

  * •

MADDPG-GCPN2: the proposed algorithm with GCPN ensembles in sample-generating.

In performing tasks, the prey is learned by DDPG because it is a single agent.

In Figure 3(a) for the shared reward case and Figure 3(b) for the individual
reward case, they show the normalized score and catching rate of the predators
with each algorithm. The scores of predators are normalized based on the
MADDPG score. The catching rate is the rate at which the prey is caught by the
predators. In the shared reward case in Figure 3(a), each algorithm has
similar scores although the score of CF is slightly lower. Although the
algorithms have the consensus mechanism for cooperation, the shared reward
leads to artificial and weak cooperation in training period, preventing the
agents from reaching equilibrium.

(a)

(b)

Figure 3: Normalized score (dark) and catching rate (light) in the shared (a)
(left) and individual (b) (right) reward cases with each algorithm.

(a)

(b)

Figure 4: Normalized average standard deviation of trained critics in the
shared (a) (left) and individual (b) (right) reward cases with each algorithm.

In the individual reward case in Figure 3(b), the normalized scores tend to be
very different. In particular, MADDPG-GCPNs have higher scores than other
algorithms. This is because when the agents receive individual rewards, they
try to perform various actions for each return of others through GCPN, leading
to more optimal training and equilibrium. On the other hand, FDMARL and MADDPG
also lead cooperation in the case of individual return because of the
existence of the consensus mechanism for cooperation. However, due to the
anthropogenic consensus mechanism, the agents in the algorithms are not
learned to achieve equilibrium for cooperation.

Being trained optimally for cooperation is seen in the individual Q-network
values of agents. In most cases, homogeneous agents exist in tasks requiring
the cooperation of agents. The homogeneous agents are considered to have
homogeneous actors and critics if they are trained optimally. In this
consideration, for the algorithms evaluated in Figure 3(a) and 3(b), the
critic values of the cooperating agents are calculated, and their standard
deviation is shown in Figure 4. In the figure, the CF is excluded because it
has global Q-network, not individual. The values of Figure 4 are normalized
based on the MADDPG. As Figure 4(a) shows, the average standard deviation of
the algorithms is similar in the shared reward case. However, in Figure 4(b),
the MADDPG-GCPNs show low standard deviations in the individual reward case
while the other algorithms show high standard deviations. This may suggest
that the agents working in MADDPG-GCPNs are trained to behave more
homogeneously.

##  5 ESS Control Application

The MADDPG-GCPN applies to the real-world problem of controlling energy
storage systems (ESSs) in several microgrids. The microgrids are decentralized
and localized energy distribution systems consisting of energy generators,
communities of households, and ESSs. Although renewable energy generators,
such as wind turbines, which can function as independent energy sources,
reduce the dependence on the central grid, such as energy company, it is
difficult to operate reliably due to its intermittent energy generation of
renewable energy subject to stochastic variations of environmental conditions.

Figure 5: Energy company and three microgrids.

An ESS can charge energy when renewable energy is over-produced or when the
energy price of the energy company is low, and it can discharge the energy
when the demand for energy from the community is high or when the energy price
of the energy company is high. When optimally planned, an ESS can relieve the
imbalance between energy production and consumption on the microgrid, leading
to reliable operation of the microgrid. Generally, the energy price depends on
the energy load on the energy company, and the higher the load, the higher the
price. The load on the energy company arises from the energy transactions of
microgrids and the energy company. In Figure 5, the nodes show energy company
$G$ and microgrids with wind turbines $W^{i}$, communities $L^{i}$, and ESSs
$B^{i}$, and there are three microgrids.

The problem of controlling multiple ESSs is the extension of single ESS
control problem [28]. Because the energy price is influenced not only by its
own microgrid but also by other microgrid, each microgrid must control the ESS
by considering other microgrids. Therefore, this problem can be modeled as a
multi-agent system and solved by our algorithm. We consider the ESS of each
microgrid as an agent and aim to control the energy charge and discharge
schedule between the ESS and each node with the optimal policy. The energy
demands from the communities and wind energy generated from the wind turbines
is provided by actual data from several European countries. Controlling
distributed ESSs in a decentralized manner can be formulated as the following
MG:

  * •

State contains the past trajectory of energy demand, generated wind energy,
energy price, and residual energy in ESS.

  * •

Action is energy flows $\mathcal{X}_{t}$ between the ESSs and each node in
Figure 5.

  * •

Energy cost is defined as $C_{t}C_{t}^{i}$ if the total load of all microgrids
is $C_{t}$ and the load of each microgrid $i$ is $C_{t}^{i}$ at time t. If the
sum of energy consumptions of all agents in a specific time is large, the
energy price increases, and thus all the players should pay more energy cost.

The goal of each agent is to minimize the accumulated energy cost over an
infinite horizon.

We employ MADDPG-GCPN to solve the MG. The individual Q-network approximates
the negative energy cost of each agent and, as suggested, two policy networks
are trained for exploration and exploitation. After training, each ESS agent
observes the past historical state and choose the ESS control actions by using
individual decentralized policy network.

(a) Normalized cost (dark) and PAR (light) with each algorithm.

(b) Total energy consumption of all microgrids.

Figure 6:

Figure 6(a) compares the results when three methods are employed:

  * •

Quadratic programming (QP) approach: the joint actions for the three ESSs are
optimized assuming there is a centralized controller that knows full knowledge
about the future demand and energy generation and the system’s dynamics.

  * •

DDPG approach: Ignoring the interactions among ESSs, each DDPG algorithm is
focused on training individual actor and critic networks.

  * •

MADDPG-GCPN: the actor networks trained by the proposed method are employed.

The cost in Figure 6(a) is the energy cost of all microgrid accumulated for
365 days of test data. The total energy costs are then normalized based on the
cost computed by QP for performance comparison. The PAR is a peak-to-average
ratio, which is a relative peak to average load.

DDPG results in high cost because each agent controls its ESS considering only
its own reward without considering the interactions among ESSs’ operation. On
the other hand, MADDPG-GCPN results in a cost closer to the cost of QP. This
is because each agent tries to reduce the hourly total energy consumption by
all agents through coordination, as shown in Figure 6(b), showing how the
total energy consumption per hour varies over time. The Figure 6(a) and 6(b)
also show that PAR of MADDPG-GCPN is slightly lower than that of DDPG, which
is hugely beneficial to operating microgrids reliably. Finally, note that the
cost-difference between QP and MADDPG-GCPN is caused by not only the
algorithm-difference but also mainly the knowledge-difference. QP is
implemented assuming that it knows exactly the future data and system’s
dynamics, which is not the case for MADDPG-GCPN.

##  6 Conclusion

In this paper, we proposed a multi-agent actor-critic algorithm with a
generative cooperative policy network. The samples generated by the GCPNs of
each agent lead other agents to explore over action space and train
collaborative policies effectively. We showed the proposed algorithm learned
more cooperative policy than existing algorithms. In addition, we implemented
the proposed algorithm to the real-world problem of controlling distributed
ESSs in microgrids.

Our algorithm has drawbacks in that it requires training additional policies
for generating samples. This requires more computation and memory than
existing algorithms. In future work, effective approximation methods for
computing gradients and effective sampling methods for drawing actions from
policies will be studied to address this issue.

## References

  * [1] J Alexander Fax and Richard M Murray.  Information flow and cooperative control of vehicle formations.  IEEE transactions on automatic control, 49(9):1465–1476, 2004.
  * [2] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen.  An overview of recent progress in the study of distributed multi-agent coordination.  IEEE Transactions on Industrial informatics, 9(1):427–438, 2013\.
  * [3] Peter Corke, Ron Peterson, and Daniela Rus.  Networked robots: Flying robot navigation using a sensor net.  In Robotics research. The eleventh international symposium, pages 234–243. Springer, 2005.
  * [4] Laëtitia Matignon, Laurent Jeanpierre, Abdel-Illah Mouaddib, et al.  Coordinated multi-robot exploration under communication constraints using decentralized markov decision processes.  In AAAI, 2012.
  * [5] Wang Ying and Sang Dayong.  Multi-agent framework for third party logistics in e-commerce.  Expert Systems with Applications, 29(2):431–436, 2005.
  * [6] Emiliano Dall’Anese, Hao Zhu, and Georgios B Giannakis.  Distributed optimal power flow for smart microgrids.  IEEE Trans. Smart Grid, 4(3):1464–1475, 2013.
  * [7] Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel.  A multi-agent reinforcement learning model of common-pool resource appropriation.  In Advances in Neural Information Processing Systems, pages 3643–3652, 2017.
  * [8] Richard S Sutton, Andrew G Barto, et al.  Reinforcement learning: An introduction.  MIT press, 1998.
  * [9] Michael L Littman.  Markov games as a framework for multi-agent reinforcement learning.  In Machine Learning Proceedings 1994, pages 157–163. Elsevier, 1994\.
  * [10] Junling Hu, Michael P Wellman, et al.  Multiagent reinforcement learning: theoretical framework and an algorithm.  In ICML, volume 98, pages 242–250. Citeseer, 1998.
  * [11] Michael L Littman.  Friend-or-foe q-learning in general-sum games.  In ICML, volume 1, pages 322–328, 2001.
  * [12] Amy Greenwald, Keith Hall, and Roberto Serrano.  Correlated q-learning.  In ICML, volume 3, pages 242–249, 2003.
  * [13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  Nature, 518(7540):529, 2015.
  * [14] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  The Journal of Machine Learning Research, 17(1):1334–1373, 2016\.
  * [15] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484, 2016.
  * [16] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  Nature, 550(7676):354, 2017.
  * [17] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio.  Deep learning, volume 1.  MIT press Cambridge, 2016.
  * [18] Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis.  Optimal and approximate q-value functions for decentralized pomdps.  Journal of Artificial Intelligence Research, 32:289–353, 2008.
  * [19] Landon Kraemer and Bikramjit Banerjee.  Multi-agent reinforcement learning as a rehearsal for decentralized planning.  Neurocomputing, 190:82–94, 2016.
  * [20] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson.  Learning to communicate with deep multi-agent reinforcement learning.  In Advances in Neural Information Processing Systems, pages 2137–2145, 2016.
  * [21] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson.  Counterfactual multi-agent policy gradients.  arXiv preprint arXiv:1705.08926, 2017.
  * [22] Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Başar.  Fully decentralized multi-agent reinforcement learning with networked agents.  arXiv preprint arXiv:1802.08757, 2018.
  * [23] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In Advances in Neural Information Processing Systems, pages 6379–6390, 2017.
  * [24] Yu-Han Chang, Tracey Ho, and Leslie P Kaelbling.  All learning is local: Multi-agent learning in global reward games.  In Advances in neural information processing systems, pages 807–814, 2004.
  * [25] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [26] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In ICML, 2014.
  * [27] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  arXiv preprint arXiv:1606.01540, 2016.
  * [28] Italo Atzeni, Luis G Ordóñez, Gesualdo Scutari, Daniel P Palomar, and Javier Rodríguez Fonollosa.  Demand-side management via distributed energy generation and storage optimization.  IEEE Transactions on Smart Grid, 4(2):866–876, 2013.


# Phase diagram of Holstein-Kondo lattice model at half-filling

Reza Nourafkan  Department of Physics, Sharif University of Technology,
P.O.Box: 11155-9161, Tehran, Iran  Nasser Nafari  Institute for Studies in
Theoretical Physics and Mathematics, P.O.Box: 19395-5531, Tehran, Iran

###### Abstract

We study the Kondo lattice model which is modified by the Holstein term,
involving both the Kondo exchange coupling and the electron-phonon coupling
constants, characterized by $J$ and $g$, respectively. The model is solved by
employing the dynamical mean-field theory in conjunction with exact
diagonalization technique. A zero temperature phase diagram of symmetry
unbroken states at half filling is mapped out which exhibits an interplay
between the two interactions and accounts for both spin and charge
fluctuations. When the Kondo exchange coupling is dominant the system is in
Kondo insulator state. Increasing $g$ for small values of $J$ leads to a Kondo
insulator-metal transition. Upon further enhancement of $g$ a transition to
the bipolaronic insulating phase takes place. Also a small region with non-
Fermi liquid behavior is found near the Kondo insulator-metal transition.

There has been a continued interest in a class of compounds called heavy
fermion semiconductors, which exhibit a spin and a charge gap at low
temperatures typically ranging between $1$ and $100$ meV Riseborough ; Misra .
In contrast to the ordinary band insulators, these two gaps are different,
indicating a separation of the spin and charge degrees of freedom brought
about by correlation effects. The gap formation in heavy fermion
semiconductors is attributed to the renormalized hybridization between a broad
band of conduction electrons and a nearly flat band of strongly correlated
$f$-electrons.

The Kondo lattice model (KLM) at half-filing is considered to be a good
starting point for investigating the properties of the heavy fermion
semiconductors. In this model, at each lattice site a local moment interacts
with the spin of a conduction electron, and thus, results in complex
correlation effects between them. In fact, a conduction and a localized
electron with antiparallel spins undergo a spin-flip process, causing
itinerant electrons to leave a trace of their spin exchange at each localized
spin site. As a result, the direction of a localized spin is affected by the
history of the electrons passing through it. There are similar correlation
effects in the periodic Anderson model due to the dynamical aspects of the
localized electrons.

Experiments involving the Kondo insulators at high magnetic fields indicate
the closure of the Kondo insulating gap, exemplifying a transition from the
Kondo insulator to a correlated metal Jaime ; Cooley . It is expected that the
electron-phonon (e-ph) interaction leads to similar results. Many experiments
suggest that the e-ph effects are important in describing a number of
observations such as the existence of an unusual phonon softening in the Kondo
lattice, CeCu2, which is indicative of coupling between electrons and phonons
Loewenhaupt . Furthermore, the lattice plays an important role in some heavy
fermion compounds, called 14-1-11, where various properties can be altered
through isoelectronic substitutions Burch . In fact, it is believed that the
coupling between phonon modes and the Kondo effect could manifest new material
properties, such as non-Fermi liquid behavior and unconventional
superconductivity Yotsuhashi ; Hotta ; Nayak .

Even less studies has been devoted to the role played by lattice vibrations in
these compounds. The role of the lattice vibrations is not trivial, but if, on
general grounds, the minimal effect of e-ph coupling is a phonon-retarded
attraction between conduction electrons with opposite spins, then the spin
excitation has a gap while the charge excitation, depending on the strength of
the e-ph coupling, can be either gap-full or gapless. Therefore, there arises
a competition between the spin- and the charge-fluctuations whose behavior is
determined, on the one hand, by the relative strength of the Kondo exchange
between the conduction electrons and the localized moments and, on the other
hand, by the conduction electron-phonon coupling leading to a complicated
phase diagram.

It is the goal of this paper to investigate the dynamical competition between
the e-ph and Kondo interactions. A natural way of incorporating the e-ph
coupling in the KLM is to add the Holstein coupling term to its hamiltonian.
In the Holstein coupling the phonon variables are coupled to the local density
of the conduction electrons. In this paper, we will present the zero
temperature phase diagram of the Holstein-Kondo lattice model (H-KLM) at half-
filling. The focus is on the transition between the unbroken symmetry ground
state as the e-ph and Kondo interactions parameters, $J$ and $g$, are varied.

The H-KLM Hamiltonian is defined by:

$\displaystyle H=$ $\displaystyle-$ $\displaystyle t\sum_{\langle
i,j\rangle\sigma}{\left(c^{\dagger}_{i\sigma}c_{j\sigma}+c.c.\right)}+\frac{J}{2}\sum_{i,\alpha\beta}{\bm{S}_{i}.\left(c^{\dagger}_{i\sigma}\bm{\sigma}_{\alpha\beta}c_{i\sigma}\right)}$
(1) $\displaystyle+$ $\displaystyle
g\sum_{i}{\left(n_{i}-1\right)\left(b^{\dagger}_{i}+b_{i}\right)}+\Omega_{0}\sum_{i}b^{\dagger}_{i}b_{i},$
(2)

where $c_{i\sigma}\left(c^{\dagger}_{i\sigma}\right)$ and
$b_{i}\left(b^{\dagger}_{i}\right)$ are, respectively, destruction (creation)
operators for itinerant electrons with spin $\sigma$ and local vibrons of
frequency $\Omega_{0}$ on site $i$, $n_{i}$ is the electron density on site
$i$, $\bm{S}_{i}$ is the spin operator for the localized spin on site $i$,
$\bm{\sigma}$ is a pseudo-vector represented by Pauli spin matrices, $t$
stands for the itinerant electrons hopping matrix elements between the
nearest-neighbor sites, $J$ is the coupling strength between itinerant
electrons and localized spins, and $g$ denote the electron-phonon coupling. We
do not consider the coulomb repulsion term between itinerant electrons,
because it tends to suppress the double occupation of sites and in our model
the exchange coupling, $J$, already does the same thing.

Our calculations are based on the dynamical mean field theory Georges , a
powerful, non-perturbative tool to study the properties of strongly correlated
systems, which allows us to treat, on equal footing, the two kinds of
interactions present in our model. This technique, which becomes exact in the
limit of infinite coordination number, reduces the full lattice many-body
problem to a local impurity embedded in a self-consistent effective bath of
free electrons, mimicing the effect of the full lattice on the local site. A
self consistency condition links the effective impurity model to the original
lattice problem. Adopting a semi-circular density of states (DOS)
$\rho_{0}(\epsilon)=(2/\pi D)\sqrt{D^{2}-\epsilon^{2}}$ of the noninteracting
system, corresponding to a Bethe lattice with the half bandwidth $D$, the
self-consistency relation imposed on the DMFT solution is given by

$\frac{D^{2}}{4}G(i\omega_{n})=\sum_{k}{\frac{V_{k}^{2}}{i\omega_{n}-\epsilon_{k}}},$
(3)

where $\epsilon_{k}$ and $V_{k}$ are the energies and the hybridization
parameters of the effective impurity model (bath parameters). We use exact
diagonalization (ED) technique to solve the effective impurity model Caffarel
. This solver allows us to access the ground state properties of the system
with a finite energy resolution. The ED technique consists of restricting the
sum in Eq. (3) to a small number of levels, and moreover, it truncates the
infinite phonon Hilbert space. The ground state and the Green’s function of
our discretized model are determined via the Lanczos procedure and the self-
consistency equation in turn allows us to derive a new set of bath parameters.
The process is iterated until convergence is reached. In the theory of Mott
transition, the investigation of the paramagnetic (PM) phase has been very
fruitful providing a lucid understanding of the finite temperature state,
above the magnetic order in many compounds. We pursue a similar approach in
our investigation and study the PM state. We force the system to be in a
paramagnetic state by averaging the spin up and spin down to study the
underlying normal state. In all our calculations presented here the
convergence of truncation has been checked.

Fig. 1 shows the $T=0$ phase diagram of the half-filled H-KLM in the parameter
space of $J$ and $g$ with $D=2t=2$ and $\Omega_{0}/t=0.2$. All types of long-
range order are excluded. Three different phases are distinguished: metallic
phase and the bipolaronic and Kondo insulating phases. In what follows, a
detailed discussion of the phase diagram of these systems will be presented.
The Kondo lattice model ($g=0$) and Holstein model ($J=0$), which are special
limiting cases of the H-KLM, have been extensively studied using the DMFT. The
ground state of KLM is the Kondo insulating phase with a spin and a charge gap
for all $J$ values Costi . For the Holstein model, the ground state is
metallic. The metallic phase is found to be a Fermi liquid, in the sense that,
the Luttinger sum rule $\rho(0)=\rho_{0}(0)$ for the spectral function
$\rho(\omega)=-ImG(\omega+i0^{+})/\pi$, or equally stated, the limit of
$ImG(i\omega_{n})\rightarrow-1$ as $\omega_{n}\rightarrow 0$, is satisfied
($\omega_{n}$ is the Matsubara frequency). Upon increasing $g$, the conduction
electrons lose their mobility, eventually acquiring polaronic character, in
which the presence of an electron is associated with a finite lattice
distortion. Also, the same e-ph coupling can cause any two polarons to attract
and form a bound pair in real space, called bipolaron Capone1 . In the absence
of pair hopping, the bipolaron formation would cause the system to undergo a
first order metal to bipolaronic insulating phase transition at the critical
coupling $g_{c}$ Koller1 ; Jeon . Meyer et al. have reported that there is a
coexistence region near $g_{c}$, which is reduced as the phonon frequency
$\Omega_{0}$ is decreased and disappears for $\Omega_{0}.leq.0.10D$ Meyer .
The bipolaron formation may be accommodated by reconstructuring the system
into a phase separated state Capone2  or a charge ordered state in which the
doubly occupied and empty sites alternate in real space Pietig .

At small fixed $J$-values, with increasing e-ph coupling, a continuous
transition to a metallic state occurs at a critical coupling $g_{1c}(J)$,
whose value increases with increasing $J$. This behavior is physically
expected. An increase in $J$ leads to a larger insulating gap, and this in
turn, leads to the suppression of the charge fluctuations which would
otherwise couple to phonons. As a result a transition to metallic state occurs
at larger e-ph coupling. We Also find that the metallic phase near $g_{1c}$
shows non-Fermi liquid character. Further increase of $g$ causes a metal-
bipolaronic phase transition taking place at a critical coupling $g_{2c}$. As
it can be distinguished, a Holstein coupling is weakly affected by exchange
coupling between conduction electrons with local spins. The metallic state
becomes more correlated as $g$ or $J$ is increased. This is reflected in the
decreasing behavior of the quasiparticle weight
$z=1/[1-Im\Sigma(i\omega_{0})/\omega_{0}]$ when $g$ or $J$ is increased (Fig.
2).

Figure 1: Zero temperature phase diagram of the unbroken symmetry Holstein-
Kondo lattice model at half-filling. The model shows three different phases:
metallic, bipolaronic and Kondo insulating phase. A narrow region with non-
Fermi liquid character is seen near the Kondo insulator-metal transition.
Figure 2: Behavior of quazi-particle weight for different values of $g$.

Fig. 3 shows the imaginary part of electron self-energy,
$Im\Sigma(i\omega_{n})$, for $J=0.1$ and several values of $g$ in the vicinity
of both phase transitions. For small $g$ values, the imaginary part of the
self-energies diverge as $\omega_{n}\rightarrow 0$, indicating the presence of
a charge gap (See panel a). Increasing $g$ causes the system to change its
phase from an insulator to a bad metal in the sense that its self-energy
extrapolates to a finite value $Im\Sigma(i0^{+})\equiv\Gamma(J)\neq 0$ for
$g\geq g_{1c}$. Hence, a finite lifetime is found at the Fermi level for a
narrow range of e-ph couplings near the $g_{1c}$, indicating that well defined
quasiparticles do not exist in this range. The violation of the Luttinger sum
rule in this region is also seen from $ImG(i\omega_{n})$, which tends to a
negative constant $c<0$ in the limit of $\omega_{n}\rightarrow 0$, with
$c<\pi\rho_{0}(0)=1$. Although the discretness of the spectra obtained in the
exact diagonalization techniqe does not allow us to unambiguously identify the
non-Fermi liquid region, we believe that the spectral function at $g=0$
displays a narrow insulating gap, whose width is proportional to the value of
$J$, with two peaks on each side. For a fixed $J$, increasing $g$ causes the
low-energy spectrum widen and are also suppressed. If these peaks overlap
before being damped completely, a narrow pseudogap forms near the Fermi level,
$E_{F}$. With further enhancement of $g$, there is a rapid shallowing of the
pseudogap till finally a quasiparticle peak forms at $E_{F}$. At this stage,
the system will have a Fermi-liquid character. Upon increasing $g$ further,
there is a weakly narrowing of quasiparticle peak until it disappears at the
second critical value of e-ph coupling $g_{2c}$ where a gap opens. A more
detailed results on the spectra might be obtained by the numerical
renormalization group technique. The inset of panel (b) shows double occupancy
$d=<n_{\uparrow}n_{\downarrow}>$ as a function of $g$. There is no signature
of the Kondo insulator-metal transition in the double occupancy, but at
$g_{2c}$ the double occupancy jumps suddenly to $d\approx 1/2$, indicating a
discontinuous transition to bipolaronic phase.

Figure 3: Imaginary part of electron self-energy, $Im\Sigma(i\omega_{n})$,
obtained at different values of $g$ in the vicinity of both phase transitions,
with fixed $J=0.1$. Panel (a): $Im\Sigma(i\omega_{n})$ in the vicinity of the
transition from the Kondo insulator state to the metallic state. Changing the
$Im\Sigma(i\omega_{n})$ behavior as $\omega_{n}\rightarrow 0$ from diverging
to extrapolating to zero shows the insulator-matallic phase transition. Panel
(b): $Im\Sigma(i\omega_{n})$ in the vicinity of the transition from the
metallic state to bipolaronic state. Inset: the double occupancy
$d=<n_{\uparrow}n_{\downarrow}>$ as a function of $g$. The transition from the
metallic state to bipolaronic state is clearly visible by observing when the
double occupancy’s jump to $\approx 1/2$ begins to set in.

Fig. 4(a) shows the phonon spectral function,
$\rho_{ph}(\omega)=-Imd(\omega+i0^{+})/\pi$, for $J=0.1$ as a function of e-ph
interaction strengths. The phonon Green’s function is defined by
$d(\omega)=\ll b_{i};b^{\dagger}_{i}\gg_{\omega}$. The figure illustrates how
the phonon mode is softened with increasing $g$. The softening phonon mode is
a manifestation of a lattice instability as in structural phase transitions. A
stability is restored by the condensation of the unstable mode. It results in
a nonzero expectation value of the phonon operator ($<b>\neq 0$) or in large
average number of excited phonons in the ground state. The appearance of
negative spectral function for $\omega<0$, when the bipolaronic state is
approached implies that there is a large increase in the lattice displacement.
In the bipolaronic state, the phonon mode hardens back to the bare mode as $g$
assumes values greater than $g_{c2}$. This is due to the fact that screening
is not effective in an insulating state. This is the same behavior which had
already been seen for pure Holstein model Koller2 . Panel (b) of Fig. 4 shows
the phonon spectral function for $g=0.2$ and various values of $J$. The phonon
mode gradually hardens back to $\Omega_{0}$, as the $J$-values increase. We
observe no signature of a transition to Kondo insulator in the phonon
spectrum. The effect of increasing $J$ is to suppress contiuously the charge
fluctuations which results in a decoupling of electrons and phonons causing
the phonon peak to exhibit hardening. In contrast to the Holstein-Hubbard
model results, where softening is absent in Mott insulator phase and phonons
are effectively decoupled from electrons Koller2 , here the hardening of the
phonon peak takes place very slowly.

Figure 4: Phonon spectral function for different values of $g$. The bare
phonon frequency is $\Omega_{0}=0.2$ and a Lorentzian broadening with the full
width at half maximum of $0.02$ has been implemented. Panel (a): Spectral
function for $J=0.1$ and various values of $g$. A cosiderable phonon softening
is seen upon approaching the transition to the bipolaronic insulator. Panel
(b): Spectral function for $g=0.2$ and various values of $J$. The transition
to Kondo insulator does not obviously affect the phonon spectral function.

In concluson, we have studied the Holstein-Kondo lattice model at half-
filling. We find that the model presents the physics of the Kondo insulator
when the exchange coupling, $J$, plays a dominant role and a transition to
correlated metal takes place for small $J$ and intermediate e-ph coupling,
$g$. Moreover, a bipolaronic-metal insulator takes place for small $J$ and
large $g$. We also find a small region with non-Fermi liquid character near
the Kondo insulator-metal transition. The remaining interesting questions will
be how the phase diagram and nature of transitions will change as $\Omega_{0}$
or electron density is changed. It is also interesting to study the symmetry
breaking states such as the antiferromagnetic and superconducting states.
Works in this direction are in progress and will be reported in a separate
publication.

###### Acknowledgements.

## References

  * (1) P. S. Riseborough, Adv. Phys, 49, 257 (2000).
  * (2) P. Mira, Heavy-Fermion Systems (Elusive, 2008).
  * (3) M. Jaime et al., Nature (London) 405, 160 (2000).
  * (4) J. C. Cooley et al., J. Superfund. 12, 171 (1999).
  * (5) M. Loewenhaupta, U. Wittea, S. Krampa, M. Bradenc, and P. Svoboda, Physica B 312–313, 181 (2002).
  * (6) K. S. Burch et al., Phys. Rev. Lett. 100, 026409 (2008).
  * (7) S. Yotsuhashi et al., J. Phys. Soc. Jpn. 74, 49 (2005).
  * (8) T. Hotta, Phys. Rev. Lett. 96, 197201 (2006).
  * (9) P. Nayak et al., Int. J. Mod. Phys. B 16, 3595 (2002).
  * (10) A. Georges, G. Kotlier, W. Kraut, and M. J. Rosenberg, Rev. Mod. Phys. 68, 13 (1996).
  * (11) M. Caffarel and W. Kraut, Phys. Rev. Lett. 72, 1545 (1994).
  * (12) T. A. Costi and N. Manini, J. of Low Temp. Physics, 126, 835-866 (2002).
  * (13) P. Werner and A. J. Millis, Phys. Rev. B 74, 155107 (2006).
  * (14) M. Capone, and S. Chichi, Phys. Rev. Lett. 91, 186405 (2003).
  * (15) W. Koller, D. Meyer, Y. Ono, and A. C. Hewson, Europhys. Lett. 66, 559 (2004).
  * (16) G. S. Jeon, T. Park, J. H. Han, H. C. Lee, and H. Y. Choi, Phys. Rev. B 70, 125114 (2004).
  * (17) D. Meyer, A. C. Howsen, and R. Bull, Phys. Rev. Lett. 89, 196401 (2002).
  * (18) M. Capone, G. Sangiovanni, C. Castellani, C. Di Castro, and M. Grilli, Phys. Rev. Lett. 92, 106401 (2004).
  * (19) R. Poetic, R. Bull, and S. Bland, Phys. Rev. Lett. 82, 4046 (1999).
  * (20) W. Koller, D. Meyer, and A. C. Hewson, Phys. Rev. B 70, 155103 (2004).