# What does the user want? Information Gain for Hierarchical Dialogue Policy
Optimisation

###### Abstract

The dialogue management component of a task-oriented dialogue system is
typically optimised via reinforcement learning (RL). Optimisation via RL is
highly susceptible to sample inefficiency and instability. The hierarchical
approach called Feudal Dialogue Management takes a step towards more efficient
learning by decomposing the action space. However, it still suffers from
instability due to the reward only being provided at the end of the dialogue.
We propose the usage of an intrinsic reward based on information gain to
address this issue. Our proposed reward favours actions that resolve
uncertainty or query the user whenever necessary. It enables the policy to
learn how to retrieve the users’ needs efficiently, which is an integral
aspect in every task-oriented conversation. Our algorithm, which we call
FeudalGain, achieves state-of-the-art results in most environments of the
PyDial framework, outperforming much more complex approaches. We confirm the
sample efficiency and stability of our algorithm through experiments in
simulation and a human trial.

Index Terms—  Dialogue systems, reinforcement learning, information gain

##  1 Introduction

††To be published in  ASRU2021, December 13-17, 2021, Cartagena, Colombia. ©
IEEE 2021. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

Task-oriented dialogue systems are characterised by an underlying task or a
goal that needs to be achieved during the conversation in order to help a
user, such as managing a schedule or finding and booking a restaurant. For
that, a spoken dialogue system needs two key abilities: maintaining the
current state of the dialogue (tracking) and foreseeing how its actions will
impact the conversation (planning). Modular dialogue systems therefore have a
tracking component to maintain information about the dialogue belief state,
and a planning component that models the underlying policy, i.e. the selection
of actions [1, 2, 3, 4]. The dialogue belief state defines a probability
distribution over states that includes information about user preferences, for
instance that a user wants a cheap, Italian restaurant, with the distribution
encoding different levels of uncertainty.

To deal with planning, current state-of-the-art dialogue systems [5, 6, 7]
optimise the policy via some form of reinforcement learning (RL) [8]. However,
dialogue policy optimisation using RL is often sample inefficient and
unstable, which is exacerbated by the sparse reward typically given in task-
oriented dialogue systems [9, 10, 11, 4]. To tackle the problem of sample
inefficiency, hierarchical reinforcement learning has been proposed that
subdivides the task temporally or spatially [12, 13], thereby reducing
complexity of the task and accelerating learning.

For spoken dialogue systems that help users accomplish any kind of tasks, it
is important to understand what the actual user’s goal is by asking
appropriate and targeted questions. Acquiring information is the first
important building block of a conversation. Due to its significance, it is
reasonable to learn a dedicated policy $\pi_{i}$ that deals with this sub-
task. This has been proposed in the hierarchical approach called Feudal
Dialogue Management [14], where the extrinsic reward is used for optimising
$\pi_{i}$. However, we argue and show that the extrinsic reward may provide
misleading feedback signal for $\pi_{i}$ that leads to unstable and less
efficient learning.

Instead, we equip the policy with an intrinsic reward based on information
gain that measures the change in probability distributions between consecutive
turns. The dense reward signal encourages the policy to produce actions that
resolve uncertainty and to query the user in cases where it is necessary. The
policy $\pi_{i}$ together with our proposed reward explicitly models how a
system can learn to obtain information about the dialogue partner, which is an
integral aspect in every conversation.

We conduct our experiments using the PyDial benchmarking environment [15]. Our
algorithm _FeudalGain_ achieves state-of-the-art results in terms of sample
efficiency and final performance in 14 out of 18 environments. We confirm the
effectiveness of our method in a human trial, where our system directly
interacts with humans.

##  2 Related Work

Information gain (also known as mutual information) measures the amount of
information obtained about one random variable through observing another
random variable. Information gain has already been used as feature as well as
reward signal for reinforcement learning. In [16], a dialogue policy for
clarification is trained using information gain as a policy feature.
Information gain has also been used to build decision trees for dialogue
systems [17, 18].

Different to the extrinsic reward that is produced by the environment,
intrinsic reward is produced by the RL agent itself. The purpose of intrinsic
reward is to additionally guide the learning of the agent. Curiosity-driven
learning, which encourages exploration of the state-action space by learning
more about the environment dynamics, has been interpreted as information gain
[19, 20]. In contrast to curiosity that will decrease the more the agent
learns about the environment, the information gain we use will always be
provided. In [21], the policy is divided into an “explorer” and “exploiter”,
using an intrinsic reward signal for the explorer that is different from the
extrinsic reward for the exploiter. While [21] focus on exploration, we define
an intrinsic reward to foster fast learning of an information seeking policy.
Note that even when the space is fully explored, one still needs to gather
sufficient information about user needs.

“Answerer in Questioner’s Mind” [22] selects the question which maximises the
information gain of a target class and an answer given the question in a goal-
oriented visual dialogue task. In [23, 24], information gain is used as a
reward in goal-oriented visual dialogues by leveraging a responder model or
guesser. The belief tracker in dialogue can be interpreted as a guesser that
needs to guess the correct value for each slot in the belief state, where the
policy can ask questions. Different from [23, 24], we use information gain in
a hierarchical setting and provide it only to a sub-policy as (intrinsic)
reward.

In [14], there is a dedicated sub-policy for information gathering as in our
case. It is however only optimised with the extrinsic reward. We introduce an
intrinsic reward based on information gain in the hierarchical setting and
thus enable fast learning of the user’s needs, an integral ability that is
often neglected in task-oriented dialogue systems.

##  3 Background

###  3.1 Dialogue Policy Optimisation via RL

The formal framework in RL is given by a Markov decision process
$\mathcal{M}=\\{\mathcal{B},\mathcal{A},r,p,p_{0},\gamma\\}$. Here,
$\mathcal{B}$ denotes the (continuous) belief state space, $\mathcal{A}$ is
the action space, $r$ is the reward function, $p(b^{\prime}|b,a)$ models the
probability of transitioning to state $b^{\prime}$ after executing action $a$
in state $b$, and $p_{0}(b)$ is the probability of starting in state $b$. The
discount factor $\gamma\in[0,1]$ trades off the importance of immediate and
future rewards. At time step $t$, the agent observes a state $b_{t}$, chooses
an action $a_{t}$ according to a policy $\pi(a|b_{t})$, transitions to a new
state $b_{t+1}$ and observes a reward signal
$r_{t}=r(b_{t},a_{t},b_{t+1})\in\mathbb{R}$. The goal of the agent is to
maximise the discounted return $R_{t}=\sum_{i\geq 0}\gamma^{i}r_{t+i}$ in
expectation.

Value-based RL methods [8] optimise the policy by maximising the Q-values
$Q^{\pi}(b_{t},a_{t})=\mathbb{E}_{b_{t+1:\infty},a_{t+1:\infty}}[R_{t}|b_{t},a_{t}]$
for every state-action pair $(b_{t},a_{t})\in\mathcal{B}\times\mathcal{A}$.
One example here is Dueling deep Q-networks (DDQN) [25] that optimises a
parameterised Q-network using tuples $({b_{t},a_{t},r_{t},b_{t+1}})$ and a
target network for a stochastic gradient step. Policy gradient methods on the
other hand parameterise the policy directly and aim at maximising
$\mathbb{E}_{b_{0}}[R_{0}]$. Actor-critic algorithms [8] build upon policy
gradient methods and approximate $Q^{\pi}$ with a function approximator, also
called critic. The ACER [9] algorithm is one such instance that uses the whole
trajectory in order to do an update step for its critic.

To foster exploration during learning, noisy networks [26] inject noise into
the neural network by substituting each weight $w$ in the neural network by
$\mu+\sigma\cdot\epsilon$, where $\mu$ and $\sigma$ are weights and $\epsilon$
is a noise random variable.

We view dialogue as a sequence of turns between a user and a dialogue system.
The belief state in dialogue typically includes a probability distribution
over values for every slot in the ontology, which expresses how likely it is
that the user wants a specific value such as “Italian” food or “expensive”
price-range. In each turn of the dialogue, for a given belief state, the
system decides which action to take in order to successfully complete the
user’s goal. This sequential decision-making task can be optimised with RL
algorithms. A range of RL algorithms have already been applied to dialogue
management, including ACER [27].

| Utterance |  $p_{s}$ for $s=\text{\emph{pricerange}}$ | $r_{e}$ | $r_{i}$ | $\pi_{i}$ | $\pi_{g}$
---|---|---|---|---|---|---
User | I need a restaurant. | [0,0,0,0,1] |  |  |  |
Sys ($\pi_{i}$) | What pricerange do you like? | [0,0,0,0,1] | -1.0 | 1.0 | _request-pricerange_ | _pass_
User | Something cheap please. $\rightarrow$ | [0.5,0.3,0.2,0,0] |  |  |  |
Sys ($\pi_{i}$) | Can I confirm that you mean cheap? | [0.5,0.3,0.2,0,0] | -1.0 | 0.22 | _confirm-pricerange_ | _pass_
User | Yes. I need the address. $\rightarrow$ | [0.95,0.05,0,0,0] |  |  |  |
Sys ($\pi_{g}$) | Goodbye. | [0.95,0.05,0,0,0] | 0.0 | 0.0 | _pass_ | _bye_

Table 1: Example dialogue in the Cambridge restaurant domain. $p_{s}$ denotes
the probability distribution over values [_cheap_ , _moderate_ , _expensive_ ,
_dontcare_ , _none_] for slot _pricerange_ given in the belief state. $r_{e}$
and $r_{i}$ denote extrinsic and intrinsic reward given to $\pi_{i}$ during
the conversation. $\pi_{i}$ tries to retrieve information and resolve
uncertainty, yet the extrinsic reward gives no guidance at all since $\pi_{g}$
ends the dialogue too early. In contrast, our proposed reward $r_{i}$
correctly rewards the behaviour of $\pi_{i}$. Information gain $r_{i}$ given
in Definition 1 is computed by Jensen-Shannon divergence between consecutive
value distributions.

###  3.2 Feudal Dialogue Management

Feudal Dialogue Management [14, 28] is a hierarchical approach for dialogue
policy learning that divides the action space $\mathcal{A}$ into two subsets
$\mathcal{A}_{i}$ and $\mathcal{A}_{g}$. The purpose of actions in
$\mathcal{A}_{i}$ is to obtain more information from the user by _confirming_
, _requesting_ or _selecting_ the value of a slot. The second action set
$\mathcal{A}_{g}$ comprises all other actions, such as general actions like
_goodbye_ or informing about requested values. Moreover, a master action space
$\mathcal{A}_{m}=\\{a_{i},a_{g}\\}$ for choosing between actions in
$\mathcal{A}_{i}$ and $\mathcal{A}_{g}$ is defined. In order to produce an
action, a master policy $\pi_{m}$ first selects an action from
$\mathcal{A}_{m}$, after which the associated policy $\pi_{i}$ or $\pi_{g}$
corresponding to $\mathcal{A}_{i}$ and $\mathcal{A}_{g}$ is consulted for the
final action selection. An additional _pass_ action is added to
$\mathcal{A}_{i}$ and $\mathcal{A}_{g}$, which is taken whenever the other
sub-policy is executed. The policy $\pi_{i}$ is optimised using a value-based
method where the Q-values are produced for every slot
$s\in\\{s_{1},...,s_{n}\\}$ independently using associated policies $\pi_{s}$.
The input to each $\pi_{s}$ is the belief state including the value
distribution of $s$. The parameters of the Q-functions are shared among the
slots.

To optimise each of the policies, the external reward $r_{e}$ provided by the
environment is used. The reward is $-1$ in each turn to enforce more efficient
dialogues and $0$ or $20$ in the very last turn for failure or success of the
dialogue.

We note that adding the _pass_ action for $\pi_{i}$ is very important. The
reward for a tuple $(b_{t},a_{t},r_{t},b_{t+1})$ with
$a_{t}\neq\text{\emph{pass}}$ that we use for updating $\pi_{i}$ will always
be $-1$ or $0$ since success can be only achieved if information is provided
(which only $\pi_{g}$ can do). We hence need to update the policy using tuples
where the _pass_ action was taken, i.e. reward the policy for doing nothing.
We empirically verify the necessity for the _pass_ action in Section 6.2.

In the following, we will work with the FeudalACER algorithm [28] as our
baseline and abbreviate it as Feudal. Feudal uses ACER for policies $\pi_{m}$
and $\pi_{g}$ and DDQN for $\pi_{i}$.

##  4 Information Gain in Policy Learning

###  4.1 Drawbacks of Extrinsic Reward in Feudal

Recall that $\pi_{i}$ merely outputs actions for obtaining information about
the user preferences. As this is not enough to complete a task, the policy
$\pi_{g}$ mainly determines dialogue success or failure. While it is
reasonable to provide $\pi_{g}$ and $\pi_{m}$ with external reward, it is less
obvious for $\pi_{i}$. The behaviour of $\pi_{i}$ can lead either to
reinforcement or suppression if $\pi_{g}$ misbehaves. As an illustrative
example, Table 1 shows a dialogue where $\pi_{i}$ acted correctly but does not
obtain any positive feedback from $r_{e}$ due to dialogue failure in the end.

###  4.2 Information Gain

How can we fairly reward $\pi_{i}$ then? We propose the usage of an intrinsic
reward for $\pi_{i}$ based on information gain similar to [23]. The idea is
that if we take an action to query information about a certain slot (e.g.
_request-area_) leading to a change in the value distribution for that slot,
new information has been gathered and that behaviour should be reinforced.
Formally, we define the intrinsic reward $r_{i}$ as follows.

###### Definition 1

Let $(b,a,b^{\prime})\in\mathcal{B}\times\mathcal{A}_{i}\times\mathcal{B}$ be
a tuple of state, action and next state where $a$ includes slot $s$. Let
$p_{s}$ and $p^{\prime}_{s}$ be the probability distributions over values for
$s$ in b and $b^{\prime}$, respectively. Let $d$ be a distance function
between probability distributions. We define

$r_{i}(b,a,b^{\prime}):=d(p_{s},p^{\prime}_{s})$

as the information gain (IG) when executing action $a$ in state $b$ and
observing $b^{\prime}$.

This reward encodes the goal of $\pi_{i}$ by reinforcing actions that gather
new information or resolve uncertainty. It separates learning of $\pi_{i}$
from the behaviour of $\pi_{g}$ and independently models how a system can
learn to obtain information about the user’s needs. Moreover, the reward
guides the policy at every step in contrast to the sparse reward that first
has to be back-propagated. Due to the immediate feedback, the additional
_pass_ action becomes obsolete for $\pi_{i}$ and we do not need to update with
tuples $(b_{t},\text{\emph{pass}},r_{t},b_{t+1})$ anymore. The policy can now
quickly learn how to obtain the user preferences, which is the first important
step towards a successful dialogue. An example for computing $r_{i}$ together
with the chosen actions is depicted in Table 1. In contrast to the external
reward, information gain reinforces the desired behaviour of $\pi_{i}$ even
though the dialogue failed. Since the probability distribution is part of the
input to the policy, $\pi_{i}$ can easily build the relation between the state
and the reward. We note that this reward is only defined and should be only
used for actions that seek to obtain information of the user. Otherwise it
might happen that the user pro-actively provides information and an unrelated
action gets rewarded. As a result, the reward can be applied to all scenarios
where obtaining information from a user is important. This is especially the
case for task-oriented dialogue systems where our focus lies, but also holds
in many more scenarios (conducting an interview, getting to know a person in
chit-chat) as dialogue is generally an exchange of information.

We remark that the usage of our reward is not restricted to hierarchical RL
and it can also be used as an additional signal to the external reward. We
also emphasise that our dense reward aids the policy in learning to reach the
main goal, i.e. task completion. It thus differs from rewards based on
curiosity or surprise [19, 29] that aim for enhanced exploration (which
decreases as the agent learns more about the environment). Our reward can be
used in tandem with other rewards, in particular the ones for exploration.

Fig. 1: The architecture of our FeudalGain algorithm together with the action
selection process and reward computation of $r_{i}^{\prime}$. FeudalGain uses
the intrinsic reward $r_{i}^{\prime}$ as given in Equation (1) instead of the
external reward $r_{e}$ for optimising $\pi_{i}$. Also different from Feudal,
FeudalGain merges the policies $\pi_{m}$ and $\pi_{g}$ into a single policy
$\pi_{mg}$ with action space $\mathcal{A}_{g}\cup\\{a_{i}\\}$.

###  4.3 FeudalGain

After introducing our intrinsic reward based on information gain, we now
present our FeudalGain algorithm. Most importantly, we substitute the
extrinsic reward for our proposed information gain to optimise the policy
$\pi_{i}$.

We choose Jensen-Shannon divergence (JS) as our distance function $d$, since
it is bounded between $0$ and $1$, symmetric and defined everywhere, in
contrast to Kullback-Leibler-divergence (KL) [30]. JS for two probability
distributions $p$ and $q$ is defined as

$\displaystyle\text{JS}(p,q)$
$\displaystyle=\frac{1}{2}(\text{KL}[p||m]+\text{KL}[q||m]),$ $\displaystyle
m$ $\displaystyle=\frac{1}{2}(p+q)$

In our experiments, we found that rewarding behaviour if information gain
exceeds a certain threshold $\delta$ works better than directly using $r_{i}$.
We henceforth work with the reward

$r_{i}^{\prime}(b,a,b^{\prime})=\begin{cases}1,&\text{if
}r_{i}(b,a,b^{\prime})\geq\delta\\\ -1,&\text{otherwise}\end{cases}$ (1)

with variables as in Definition 1.

ACER uses the full trajectory to update its critic, which is why $\pi_{g}$
needs to take an action in every turn. Feudal solves this by $\pi_{g}$ taking
the _pass_ action whenever $\pi_{i}$ takes an action as shown in Table 1.
However, we can avoid that additional action by merging the policies $\pi_{m}$
and $\pi_{g}$ into a single policy $\pi_{mg}$ with action space
$\mathcal{A}_{g}\cup\\{a_{i}\\}$, which we employ in our full algorithm.

Our final algorithm FeudalGain thus uses $\pi_{mg}$ and $\pi_{i}$ together
with intrinsic reward $r_{i}^{\prime}$ for $\pi_{i}$. Our full algorithm is
depicted in Figure 1.

##  5 Experimental Setup

We implement FeudalGain111Our code will be released at
https://pydial.cs.hhu.de. in the PyDial toolkit [31]. The performance is
evaluated using the PyDial benchmarking environments [15] comprising 18
settings, which are distinguished by domain and different semantic error
rates, action masks and user simulator configurations. Unlike other publicly
available dialogue toolkits, PyDial uses a belief tracker that outputs
probability distributions rather than binary states, which enables more
expressive distribution comparisons.

Instead of the $\epsilon$-greedy approach for exploration that was used for
Feudal, we use noisy networks similar to [5]. The threshold $\delta$ for all
our experiments in Section 6.1 is set to $0.2$. The FeudalGain policies
$\pi_{mg}$ and $\pi_{i}$ are trained with ACER and DDQN, respectively. For
each environment, the algorithms are trained on 4000 dialogues with 10
different seeds. After every 200 training dialogues, the algorithms are
evaluated on 500 dialogues. The average reward that is shown is always the
extrinsic reward $r_{e}$. The dialogues for our human trial are collected
using DialCrowd [32]. The simulated user experiments are done on semantic
level using the default focus belief tracker, while the user trial is
performed on text-level using an additional template based natural language
generation module.

We compare FeudalGain to Feudal [28] and the current state-of-the-art
algorithm STRAC [5]. STRAC uses a hierarchical decision-making model for
policy optimisation with implicit policy decomposition and noisy networks for
exploration. STRAC offers two different modes: a single-domain version STRAC-S
that is trained and evaluated on a single domain and STRAC-M that is trained
on three different domains but evaluated only on a single domain. STRAC-M
trains on three times as many dialogues as STRAC-S and FeudalGain. We do not
compare to the work of [6] as they use a hand-coded expert during training.
For completeness, we also add the performance of a hand-coded policy (HDC),
which is already implemented in PyDial.

##  6 Results

###  6.1 Results on FeudalGain

We compare FeudalGain to STRAC-S in terms of sample efficiency and final
performance. Table 2 shows success rate and average reward after 400 and 4000
dialogues.

FeudalGain has higher sample efficiency than STRAC-S in almost all settings
and is comparable to STRAC-M although STRAC-M uses three times as many
dialogues. This can be attributed to the immediate reward provided by our
information gain that correctly guides $\pi_{i}$ in every turn.

Similar conclusions can be drawn after 4000 dialogues, where FeudalGain is
even able to outperform STRAC-M. Information gain hence not only helps in
securing more sample efficient learning but also for achieving high final
performance. FeudalGain excels in difficult environments, namely 2 and 4,
where unreasonable actions are not masked. FeudalGain also performs very well
in environment 6 that exhibits a very high noise level of $30\%$, which shows
that information gain is robust to high error rates. Final performance is
slightly worse in environment 5, where an “unfriendly” user simulator is used.
The results show that policy optimisation can significantly benefit from our
reward based on information gain.

Fig. 2: Ablation study for FeudalGain. “W/o pass” ablates the additional
_pass_ action for $\pi_{i}$. NN denotes noisy networks. IG denotes our
proposed information gain addition. Fig. 3: Robustness test of FeudalGain
against Feudal+NN for increasing semantic error rate. We use policies that
trained for 200 dialogues and 4000 dialogues averaged over 15 seeds.

| FeudalGain | STRAC-S | STRAC-M | HDC
---|---|---|---|---
Task | Suc. | Rew. | Suc. | Rew. | Suc. | Rew. | Suc. | Rew.
After 400 dialogues |  |
Env1 | CR | 99.8 | 14.2 | 97.7 | 13.1 | 99.7 | 14.0 | 100.0 | 14.1
SFR | 95.8 | 11.6 | 98.2 | 12.3 | 99.2 | 12.9 | 97.6 | 12.1
LAP | 96.1 | 11.5 | 98.5 | 12.3 | 98.6 | 12.2 | 97.2 | 11.8
Env2 | CR | 86.8 | 10.7 | 65.5 | 5.0 | 90.3 | 10.2 | 100.0 | 14.1
SFR | 89.6 | 10.5 | 69.8 | 4.4 | 87.5 | 9.0 | 97.6 | 12.4
LAP | 84.7 | 9.7 | 56.9 | 1.6 | 89.2 | 9.1 | 97.8 | 11.7
Env3 | CR | 97.6 | 12.7 | 97.2 | 12.5 | 97.3 | 12.7 | 95.2 | 10.8
SFR | 91.4 | 9.1 | 90.4 | 8.9 | 93.6 | 10.5 | 90.2 | 8.9
LAP | 91.2 | 9.3 | 92.5 | 9.7 | 92.4 | 9.6 | 88.2 | 8.4
Env4 | CR | 82.9 | 8.9 | 71.0 | 5.2 | 75.3 | 6.6 | 97.0 | 11.1
SFR | 84.1 | 8.7 | 72.7 | 5.0 | 77.2 | 6.4 | 89.2 | 8.2
LAP | 82.3 | 8.0 | 65.9 | 3.1 | 79.8 | 6.9 | 88.6 | 8.4
Env5 | CR | 95.0 | 11.1 | 95.3 | 10.6 | 95.6 | 10.8 | 94.6 | 9.2
SFR | 87.1 | 6.9 | 80.6 | 4.5 | 88.8 | 7.5 | 87.6 | 6.3
LAP | 87.6 | 6.6 | 87.8 | 6.1 | 86.0 | 5.6 | 82.8 | 4.6
Env6 | CR | 92.8 | 10.9 | 91.9 | 10.3 | 90.7 | 9.9 | 91.2 | 9.5
SFR | 79.4 | 5.6 | 78.5 | 4.9 | 83.8 | 6.6 | 80.2 | 6.5
LAP | 81.7 | 6.1 | 84.6 | 6.6 | 81.7 | 5.7 | 76.6 | 5.6
Mean | CR | 92.5 | 11.4 | 86.4 | 9.5 | 91.5 | 10.7 | 96.3 | 11.5
SFR | 87.9 | 8.7 | 81.7 | 6.7 | 88.3 | 8.8 | 90.4 | 9.1
LAP | 87.3 | 8.5 | 81.0 | 6.6 | 88.0 | 8.2 | 88.5 | 8.4
After 4000 dialogues |  |
Env1 | CR | 99.9 | 14.1 | 99.8 | 14.1 | 99.8 | 14.1 | 100.0 | 14.1
SFR | 99.0 | 12.7 | 98.7 | 12.7 | 98.5 | 12.7 | 97.6 | 12.1
LAP | 97.6 | 12.0 | 97.6 | 12.0 | 97.8 | 12.0 | 97.2 | 11.8
Env2 | CR | 98.0 | 13.3 | 97.9 | 13.1 | 98.4 | 13.1 | 100.0 | 14.1
SFR | 98.8 | 13.7 | 95.6 | 12.1 | 97.5 | 13.0 | 97.6 | 12.4
LAP | 98.5 | 13.3 | 92.6 | 11.6 | 98.0 | 12.8 | 97.8 | 11.7
Env3 | CR | 98.6 | 13.0 | 98.1 | 13.0 | 97.9 | 12.9 | 95.2 | 10.8
SFR | 95.1 | 10.4 | 91.9 | 10.5 | 93.0 | 10.6 | 90.2 | 8.9
LAP | 91.2 | 9.5 | 90.7 | 9.7 | 92.1 | 9.9 | 88.2 | 8.4
Env4 | CR | 98.0 | 12.5 | 92.9 | 11.5 | 91.3 | 10.8 | 97.0 | 11.1
SFR | 93.1 | 11.2 | 90.2 | 10.7 | 89.2 | 10.5 | 89.2 | 8.2
LAP | 91.8 | 11.1 | 86.3 | 9.2 | 89.7 | 10.4 | 83.6 | 8.4
Env5 | CR | 97.6 | 11.9 | 97.1 | 11.8 | 96.5 | 11.7 | 94.6 | 9.2
SFR | 88.4 | 7.1 | 89.6 | 8.4 | 90.1 | 8.4 | 87.6 | 6.3
LAP | 87.6 | 6.6 | 88.2 | 6.9 | 88.5 | 7.0 | 82.8 | 4.6
Env6 | CR | 94.0 | 11.0 | 92.5 | 11.0 | 91.5 | 10.7 | 91.2 | 9.5
SFR | 88.2 | 7.9 | 81.6 | 7.0 | 84.5 | 7.3 | 80.2 | 6.5
LAP | 83.4 | 6.8 | 83.3 | 6.7 | 83.2 | 6.7 | 76.6 | 5.6
Mean | CR | 97.7 | 12.6 | 96.4 | 12.4 | 95.9 | 12.2 | 96.3 | 11.5
SFR | 93.8 | 10.5 | 91.3 | 10.2 | 92.1 | 10.4 | 90.4 | 9.1
LAP | 91.7 | 9.9 | 89.8 | 9.4 | 91.6 | 9.8 | 88.5 | 8.4

Table 2: Success rate and average reward (using only $r_{e}$) for our proposed
approach FeudalGain against STRAC-S. Best performance is marked in bold.
Algorithms were tested in the Cambridge Restaurant (CR), San-Francisco
Restaurant (SFR) and Laptops (LAP) domain. We included STRAC-M and a hand-
coded policy (HDC) as supplemental comparison. Note that STRAC-M is trained in
three domains and therefore utilized three times the amount of data compared
to STRAC-S and FeudalGain. Results of STRAC were taken from [5]. Fig. 4:
Training loss for Feudal [28] using noisy networks (NN) and our proposed
information gain reward (IG). We approximated the loss in every update step by
taking 512 samples from the replay buffer.

###  6.2 Ablation Study

We conduct an ablation study for FeudalGain to investigate the difference in
stability and convergence speed due to our proposed changes. We conduct
experiments in environment 3 that exhibits a semantic error rate of $15\%$. We
chose this environment as it is close to human experiment characteristics
[33]. Figure 2 depicts our findings.

We first ablate the _pass_ action for $\pi_{i}$ by only updating $\pi_{i}$
with tuples $(b_{t},a_{t},r_{t},b_{t+1})$ where $a_{t}\neq\text{\emph{pass}}$
to empirically verify that it is needed in order to back-propagate the final
reward to actions taken by $\pi_{i}$. The algorithm is not capable of learning
without _pass_ when only the extrinsic reward is used.

While usage of noisy networks can stabilise the learning for Feudal, the true
benefit comes from the addition of information gain that results in fast and
smooth convergence after as few as 500 dialogues. The dependence on the seed
almost vanishes when introducing information gain, hence more stable learning
and robustness against randomness in the initialisation is achieved. The usage
of a single policy $\pi_{mg}$ in addition to information gain only led to a
small performance difference in the first 200 dialogues. We hence omitted it
in Figure 2 for better readability.

We further test the robustness of FeudalGain for increasing amount of semantic
error rates by comparing it against Feudal+NN that does not use information
gain. Results are depicted in Figure 3, for policies trained for 200 and 4000
dialogues, averaged over 15 seeds. For 200 training dialogues, the difference
between FeudalGain and the baseline is consistent across noise levels. For
4000 training dialogues, the baseline catches up a little but does not
outperform FeudalGain on any of the noise levels.

Lastly, we want to empirically verify that the usage of extrinsic reward may
provide misleading feedback for $\pi_{i}$, resulting in incorrect policy
updates. Figure 4 shows the training loss for policy $\pi_{i}$ using Feudal
with noisy networks and information gain. Substituting $r_{e}$ for
$r_{i}^{\prime}$ leads to quick and stable convergence of the algorithm with
weaker oscillations.

###  6.3 Human Evaluation

In order to show that our results transfer from simulation to humans, we
compare FeudalGain against Feudal with noisy networks (Feudal + NN) in a human
trial, where users directly interact with the two policies. We collected 400
dialogues using each policy. We took the policies after only 200 training
dialogues that were closest to the average performance in environment 3. The
reward was $11.7$ for Feudal + NN and $12.9$ for FeudalGain on the simulated
user. We chose such a small number of training dialogues to examine the sample
efficiency of FeudalGain. At the end of each interaction, we asked users if
the dialogue was successful, whether the system asked for information when
necessary (“AskIfNec”) and what the overall performance was (“Overall”). Table
3 shows that superior performance of FeudalGain in terms of success and number
of turns in simulation translates to real users. More interestingly, the
rating if information was requested when necessary is much higher, which
confirms that our intrinsic reward enables $\pi_{i}$ to learn guided
information gathering. The overall rating (“Overall”) is correlated to
“AskIfNec”, showing how important guided information gathering is for the
overall perception of the system. The reduced standard deviation shows the
stability of our approach.

| Success | Turns | AskIfNec | Overall
---|---|---|---|---
FeudalGain | $0.71/0.45^{\ast}$ | $6.5/3.4^{\ast}$ | $3.8/1.4^{\ast}$ | $3.7/1.5^{\ast}$
Feudal + NN | $0.43/0.5$ | $8.1/5.0$ | $3.0/1.5$ | $2.7/1.6$

Table 3: Mean/standard deviation for success, number of turns, whether the
system asked for information when necessary and overall performance according
to human evaluation. ∗We used the t-test to check statistical significance,
where $p<0.05$.

##  7 Conclusion

We proposed the use of intrinsic reward within the hierarchical Feudal
Dialogue Management approach for the information seeking policy. Our new
architecture gracefully deals with shortcomings such as artificial _pass_
actions and misleading reward signals that lead to sample inefficiency and
instability. Our proposed reward encourages the policy to seek useful
information from the user and puts more emphasis on the user’s needs, which is
an integral part of dialogue systems that aid the user in solving any kinds of
tasks. We show in experiments with simulated users that incorporating our
reward improves sample efficiency, stability, and quality of the resulting
policy, and that our algorithm FeudalGain algorithm leads to state-of-the-art
results for the PyDial benchmark. We confirm the results in a human trial
where volunteers interacted with our policy. Our results warrant a more
widespread use of intrinsic reward in task-oriented dialogue systems.

In future work, we like to scale our approach to multiple domains and learn
the hierarchical structure automatically.

## Acknowledgments

This work is a part of DYMO project which has received funding from the
European Research Council (ERC) provided under the Horizon 2020 research and
innovation programme (Grant agreement No. STG2018 804636). N. Lubis, C. van
Niekerk, M. Heck and S. Feng are funded by an Alexander von Humboldt Sofja
Kovalevskaja Award endowed by the Federal Ministry of Education and Research.
Computational resources were provided by Google Cloud.

## References

  * [1] Esther Levin and Roberto Pieraccini,  “A stochastic model of computer-human interaction for learning dialogue strategies,”  in EUROSPEECH 97, 1997, pp. 1883–1886.
  * [2] Nicholas Roy, Joelle Pineau, and Sebastian Thrun,  “Spoken dialogue management using probabilistic reasoning,”  in Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, USA, 2000, ACL ’00, p. 93–100, Association for Computational Linguistics.
  * [3] Jason D. Williams and Steve Young,  “Partially observable markov decision processes for spoken dialog systems,”  Comput. Speech Lang., vol. 21, no. 2, pp. 393–422, Apr. 2007.
  * [4] Zheng Zhang, Ryuichi Takanobu, Minlie Huang, and Xiaoyan Zhu,  “Recent advances and challenges in task-oriented dialog system,”  CoRR, vol. abs/2003.07490, 2020.
  * [5] Zhi Chen, Lu Chen, Xiaoyuan Liu, and Kai Yu,  “Distributed structured actor-critic reinforcement learning for universal dialogue management,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2400–2411, 2020.
  * [6] Thibault Cordier, Tanguy Urvoy, Lina M. Rojas-Barahona, and Fabrice Lefèvre,  “Diluted near-optimal expert demonstrations for guiding dialogue stochastic policy optimisation,”  in 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, 2020.
  * [7] Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-chin Lin, Marco Moresi, Carel van Niekerk, and Milica Gašić,  “LAVA: Latent action spaces via variational auto-encoding for dialogue policy optimization,”  in Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), Dec. 2020, pp. 465–479, International Committee on Computational Linguistics.
  * [8] Richard S. Sutton and Andrew G. Barto,  Reinforcement learning: an introduction,  The MIT Press, second edition, 2018.
  * [9] Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Rémi Munos, Koray Kavukcuoglu, and Nando de Freitas,  “Sample efficient actor-critic with experience replay,”  in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017, OpenReview.net.
  * [10] Gabriel Dulac-Arnold, Daniel J. Mankowitz, and Todd Hester,  “Challenges of real-world reinforcement learning,”  Reinforcement Learning for Real Life (RL4RealLife) Workshop in the 36th International Conference on Machine Learning, Long Beach, California, USA, 2019, 2019.
  * [11] Hao Dong Zihan Ding,  “Challenges of reinforcement learning,”  in Deep Reinforcement Learning: Fundamentals, Research, and Applications, Shanghang Zhang Hao Dong, Zihan Ding, Ed., chapter 7, pp. 249–272. Springer Nature, 2020,  http://www.deepreinforcementlearningbook.org.
  * [12] Richard S. Sutton, Doina Precup, and Satinder Singh,  “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,”  Artificial Intelligence, vol. 112, no. 1, pp. 181–211, 1999.
  * [13] Peter Dayan and Geoffrey E Hinton,  “Feudal reinforcement learning,”  in Advances in Neural Information Processing Systems, S. Hanson, J. Cowan, and C. Giles, Eds. 1993, vol. 5, Morgan-Kaufmann.
  * [14] Iñigo Casanueva, Paweł Budzianowski, Pei-Hao Su, Stefan Ultes, Lina M. Rojas-Barahona, Bo-Hsiang Tseng, and Milica Gašić,  “Feudal reinforcement learning for dialogue management in large domains,”  in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, June 2018, pp. 714–719, Association for Computational Linguistics.
  * [15] Iñigo Casanueva, Paweł Budzianowski, Pei-Hao Su, Nikola Mrkšić, Tsung-Hsien Wen, Stefan Ultes, Lina Rojas-Barahona, Steve Young, and Milica Gašić,  “A benchmarking environment for reinforcement learning based task oriented dialogue management,”  vol. abs/1711.11023, 2018.
  * [16] Aishwarya Padmakumar and Raymond J. Mooney,  “Dialog policy learning for joint clarification and active learning queries,”  in The AAAI Conference on Artificial Intelligence (AAAI), February 2021.
  * [17] Joseph Polifroni and Marilyn Walker,  “Learning database content for spoken dialogue system design,”  in Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy, May 2006, European Language Resources Association (ELRA).
  * [18] Paul Fodor,  “Domain directed dialogs for decision processes,”  CoRR, vol. abs/cs/0703072, 2007.
  * [19] Rein Houthooft, Xi Chen, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel,  “Vime: Variational information maximizing exploration,”  in Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, Eds. 2016, vol. 29, Curran Associates, Inc.
  * [20] Zachary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng,  “BBQ-networks: Efficient exploration in deep reinforcement learning for task-oriented dialogue systems,”  Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018.
  * [21] Louis Bagot, Kevin Mets, and S. Latré,  “Learning intrinsically motivated options to stimulate policy exploration,”  in Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 108, 2020., 2020.
  * [22] Sang-Woo Lee, Yu-Jung Heo, and Byoung-Tak Zhang,  “Answerer in questioner's mind: Information theoretic approach to goal-oriented visual dialog,”  in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. 2018, vol. 31, Curran Associates, Inc.
  * [23] Pushkar Shukla, Carlos E. L. Elmadjian, Richika Sharan, Vivek Kulkarni, Matthew A. Turk, and William Yang Wang,  “What should I ask? using conversationally informative rewards for goal-oriented visual dialog,”  in Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Anna Korhonen, David R. Traum, and Lluís Màrquez, Eds. 2019, pp. 6442–6451, Association for Computational Linguistics.
  * [24] Ehsan Abbasnejad, Iman Abbasnejad, Qi Wu, Javen Shi, and Anton van den Hengel,  “Gold seeker: Information gain from policy distributions for goal-oriented vision-and-langauge reasoning,”  in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. 2020, pp. 13447–13456, IEEE.
  * [25] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas,  “Dueling network architectures for deep reinforcement learning,”  in Proceedings of the 33rd International Conference on Machine Learning - Volume 48. 2016, ICML’16, p. 1995–2003, JMLR.org.
  * [26] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alexander Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg,  “Noisy networks for exploration,”  in Proceedings of the International Conference on Representation Learning (ICLR 2018), Vancouver (Canada), 2018.
  * [27] Gellért Weisz, Paweł Budzianowski, Pei-Hao Su, and Milica Gašić,  “Sample efficient deep reinforcement learning for dialogue systems with large action spaces,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 11, pp. 2083–2097, 2018.
  * [28] Iñigo Casanueva, Paweł Budzianowski, Stefan Ultes, Florian Kreyssig, Bo-Hsiang Tseng, Yen-chen Wu, and Milica Gašić,  “Feudal dialogue management with jointly learned feature extractors,”  in Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, Australia, July 2018, pp. 332–337, Association for Computational Linguistics.
  * [29] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell,  “Curiosity-driven exploration by self-supervised prediction,”  in Proceedings of the 34th International Conference on Machine Learning - Volume 70. 2017, ICML’17, p. 2778–2787, JMLR.org.
  * [30] Solomon Kullback and Richard A. Leibler,  “On information and sufficiency,”  The Annals of Mathematical Statistics, vol. 22, no. 1, pp. 79 – 86, 1951.
  * [31] Stefan Ultes, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, Dongho Kim, Iñigo Casanueva, Paweł Budzianowski, Nikola Mrkšić, Tsung-Hsien Wen, Milica Gašić, and Steve Young,  “PyDial: A multi-domain statistical dialogue system toolkit,”  in Proceedings of ACL 2017, System Demonstrations, Vancouver, Canada, July 2017, pp. 73–78, Association for Computational Linguistics.
  * [32] Kyusong Lee, Tiancheng Zhao, Alan W. Black, and Maxine Eskenazi,  “DialCrowd: A toolkit for easy dialog system assessment,”  in Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, Australia, July 2018, pp. 245–248, Association for Computational Linguistics.
  * [33] Steve Young, Catherine Breslin, Milica Gašić, Matthew Henderson, Dongho Kim, Martin Szummer, Blaise Thomson, Pirros Tsiakoulis, and Eli Hancock,  Evaluation of Statistical POMDP-Based Dialogue Systems in Noisy Environments, pp. 3–14,  04 2016.


# Edge magnetic properties of black phosphorene nanoribbons

Javad Vahedi<EMAIL_ADDRESS>Technische Universität
Braunschweig, Institut für Mathematische Physik, Mendelssohnstraße 3, 38106
Braunschweig, Germany  Laboratoire de Physique Théorique et Modélisation, CNRS
UMR 8089, CY Cergy Paris Université, 95302 Cergy-Pontoise Cedex, France.
Department of Physics, Sari Branch, Islamic Azad University, Sari 48164-194,
Iran  Robert Peters<EMAIL_ADDRESS>Department of Physics, Kyoto
University, Kyoto 606-8502, Japan.

###### Abstract

The magnetic properties of black phosphorene nanoribbons are investigated
using static and dynamical mean-field theory. Besides confirming the existence
of ferromagnetic/antiferromagnetic edge magnetism, our detailed calculations
using large unit-cells find a phase-transition at weak interaction strength to
an incommensurate (IC) magnetic phase. A detailed Fourier analysis of the
magnetization patterns in the IC phase shows the existence of a second
critical interaction strength, where the incommensurate phase changes to an
antiferromagnetic (AFM) or ferromagnetic (FM) phase. We demonstrate that the
difference of the ground state energies of the AFM and FM phase is
exponentially small, making it possible to switch between both states by a
small external field. Finally, we analyze the influence of strain and disorder
on the magnetic properties and show that while the IC phase is robust to
Anderson type disorder, it is fragile against strain.

##  I Introduction

Phosphorene, a novel promising 2D material, has recently attracted much
attention owing to its anisotropic bandstructure Liu  _et al._ (2014);
Castellanos-Gomez (2016); Koenig  _et al._ (2014). It is a bilayer puckered
honeycomb lattice of black phosphorus with a peculiar bandstructure exhibiting
Dirac cones in the bulk. Because of its bandstructure, phosphorene has been
studied in many theoretical works, particularly in the context of transport
studies Ghosh  _et al._ (2017); Wang  _et al._ (2015); Linder and Yokoyama
(2017); Zare  _et al._ (2017). Compared to the transition metal dichalcogenide
materials, phosphorene has a high charge carrier mobility ($\sim
100~{}\text{cm}^{2}/\text{Vs}$) at room temperature Liu  _et al._ (2014),
making it favorable for electronic applications. Moreover, zigzag phosphorene
nanoribbons (ZPNR) exhibit two quasi-flat edge states, which are completely
isolated from the bulk states Ezawa (2014); Taghizadeh Sisakht  _et al._
(2015); Ma  _et al._ (2016), in contrast to the other 2D hexagonal lattice
structures such as graphene Castro Neto  _et al._ (2009) and silicene Shakouri
_et al._ (2015). The nature of these isolated edge states originating from a
large hopping parameter between two out-of-plane zigzag chains has been
discussed in Ref. Ezawa (2014). Furthermore, a recent study addressed the
Ruderman-Kittel-Kasuya-Yosida (RKKY) exchange interaction in ZPNRs. It found
two different characteristic periods of the RKKY interaction mediating the
magnetic interaction between impurities Islam  _et al._ (2018).

Motivated by theoretical predictions Yazyev (2010); Feldner  _et al._ (2010,
2011) and experimental confirmations Magda  _et al._ (2014) of edge magnetism
in zigzag graphene nanoribbons, edge magnetism has also been explored in
phosphorene in Ref. Zhu  _et al._ (2014). This first study found that ZPNRs
display a magnetic state at the edge in the absence of a Peierls distortion.
However, edge magnetism vanished in the fully relaxed structure. On the other
hand, the authors of Ref. Du  _et al._ (2015) have shown that edge magnetism
of ZPNR, can survive even with structural relaxation. In another paper, the
authors consider tilted black phosphorene nanoribbons (TPNRs) exposed to an
external electric field. They found that the magnetic ground state can be
switched by an electric field from antiferromagnetic (AFM) to ferromagnetic
(FM) Farooq  _et al._ (2016). Furthermore, a quantum Monte Carlo calculation
demonstrated a high Curie temperature for edge magnetism of ZPNR Yang  _et
al._ (2016). However, previous studies have mainly considered small unit cells
focusing on commensurate FM and AFM states. Magnetic states such as spiral
phases or incommensurate phases have not been analyzed. Furthermore, the
effect of strain and disorder on the magnetic states still remains unclear.

In this work, using a tight-binding (TB) Hubbard model, we numerically study
the edge magnetism of ZPNRs using static mean-field theory (MFT) and dynamical
mean-field theory (DMFT). Although QMC must be considered superior to our
mean-field approaches, our (D)MFT is much faster and thus makes it possible to
analyze large unit cells and incommensurate magnetic phases. Furthermore,
mean-field theories have proven to at least qualitatively, sometimes even
quantitatively, correctly describe magnetism in hexagonal 2D systemsRaczkowski
_et al._ (2020), although being numerically less expensive. A similar
combination of techniques has been used to analyze edge magnetism in zigzag
graphene nanoribbons Fernández-Rossier and Palacios (2007) and nanodots
Bhowmick and Shenoy (2008).

In this paper, we demonstrate the existence of an incommensurate magnetic
phase at the edge of ZPNR for weak interaction strengths $U_{c1}\lesssim
U\lesssim U_{c2}$. With increasing interaction strength, this incommensurate
magnetic phase undergoes a phase transition into the ferromagnetic or
antiferromagnetic phase at $U_{c2}$, which has been reported by previous
studies. We show that the difference in the ground state energies of these two
states is exponentially small, making it easy to switch between both states.
Besides, to gain more insight into the realization of magnetism in ZPNRs at
weak interaction strengths, the purpose of this paper is to analyze the
effects of strain and defects on the magnetic state. Such perturbations of the
material are ubiquitous in 2D materials Gui  _et al._ (2008); Wang  _et al._
(2014); Yue  _et al._ (2012); Tabatabaei  _et al._ (2013); Rodin  _et al._
(2014); Elahi  _et al._ (2015); Taghizadeh Sisakht  _et al._ (2016). Moreover,
studies on strain in nonmagnetic phosphorene show some intriguing features: A
first-principle study predicted a semiconductor-semimetal-metal transition
under perpendicular compression Rodin  _et al._ (2014). In Ref. Elahi  _et
al._ (2015), an emergence of a peculiar Dirac-shaped dispersion for tensile
strain in the zigzag edge is proposed. In another work, it was shown that
tensile or in-plane strain, together with spin-orbit interaction, gives rise
to a topological phase transition Taghizadeh Sisakht  _et al._ (2016).
However, the only study which analyzes the effect of strain on ZPNRs magnetism
is found in Ref. Du  _et al._ (2015). It predicts that at a critical
compressive strain along the zigzag edge (about 5$\%$), the ground state
changes from an AFM semiconductor to a nonmagnetic metal. Thus, we here
address the effect of strain and Anderson type disorder on the magnetic
properties of ZPNRs and find that while the IC phase is very sensitive to
strain and disappears fast, it is robust against Anderson type disorder. We
also notice that the second critical point $U_{c2}$ shifts to larger values.
Thus, one can predict that the AFM/FM magnetic phase disappears under large
strain.

The paper is organized as follows: In Sec. II, we introduce the theoretical
model and formalism used in the numerical calculations. In Sec. III, we
discuss results obtained. Finally, we summarize and conclude our results in
Sec. IV.

Figure 1: A schematic view of the ZPNR: The yellow area corresponds to the
unit cell, which consists of four atoms. The black arrows show the hoppings up
to the fifth nearest-neighbor hopping included in our TB model. The red (blue)
circles indicate the upper (lower) layers. The black box is the ribbon unit
cell in the y-direction. In the text, $N_{y}$ refers to the number of ribbon
unit cells in the y-direction. The width of the unit cell is specified by
$N_{x}$, which includes $N=4\times N_{x}$ phosphorus atoms.

##  II MODEL AND FORMALISM

In order to study the magnetic properties of ZPNRs, we use the following
tight-binding model

$H=\sum_{ij,\sigma}t_{ij}c_{i\sigma}^{\dagger}c_{j\sigma}+U\,\sum_{i}\left(n_{i,\uparrow}-\frac{1}{2}\right)\,\left(n_{i,\downarrow}-\frac{1}{2}\right),$
(1)

where the first summation runs up to the fifth nearest neighbor, and $t_{ij}$
is the hopping integral proposed in Ref. Rudenko and Katsnelson (2014). These
hopping parameters are $t_{1}=-1.220~{}eV$, $t_{2}=3.665~{}eV$,
$t_{3}=-0.205~{}eV$, $t_{4}=-0.105~{}eV$, and $t_{5}=-0.055~{}eV$, which are
shown by arrows in Fig. 1. Furthermore, we include a local density-density
interaction. Thus, this model corresponds to a one-band Hubbard model. To
tackle the interaction, we use static and dynamical mean-field theory. At the
static mean-field level (MFT) level, all quantum fluctuations are neglected,
and the SU(2) spin symmetry must be broken artificially in order to capture
the formation of local moments. An extension of the MFT to account for the
local moment formation is the dynamical mean-field theory (DMFT). The DMFT
approximation accounts for temporal fluctuations and thus includes local
charge fluctuations beyond static MFT. Indeed, the accuracy of DMFT to predict
the critical point, $U_{c}$, in the honeycomb lattice has been reported
recently in Ref.Raczkowski  _et al._ (2020); Phùng  _et al._ (2020).

###  II.1 Static mean-field theory (MFT)

Evaluating the Coulomb interaction term in the mean-field approximation leads
to two potentials terms, direct and exchange term, which must be solved self-
consistently. In the case of the Hubbard model in the collinear approximation,
only the direct potential term is nonzero, and one obtains

$U\,\sum_{i}\Bigl{(}\langle n_{i,\uparrow}\rangle
n_{i,\downarrow}+n_{i,\uparrow}\langle n_{i,\downarrow}\rangle-\langle
n_{i,\uparrow}\rangle\langle
n_{i,\downarrow}\rangle-\frac{n_{i,\uparrow}+n_{i,\downarrow}}{2}+\frac{1}{4}\Bigr{)}$

where $n_{i\sigma}=c_{i\sigma}^{\dagger}c_{i\sigma}$ is the number operator
and $\langle n_{i\sigma}\rangle$ is the average electron occupation number for
spin-down $(\downarrow)$ and spin-up $(\uparrow)$ electrons on lattice site
$i$. We focus on the undoped ZPNR with exactly one electron per lattice site,
i.e., we work with the half-filled Hubbard model.

To calculate the magnetic ground state of Hamiltonian Eq. (1), we start with a
few initial, specific or random, configurations for the average electron
occupation number $\langle n_{i\sigma}\rangle$. Then, by diagonalizing the
Hamiltonian, we calculate updated electron occupation numbers. This procedure
is repeated until the convergence criteria, chosen as $\eta=10^{-8}$, is
achieved on the average electron occupation number. This self-consistent
solution provides the local magnetization
$m_{i}^{z}=(n_{i\uparrow}-n_{i\downarrow})/2$ on each site. Finally, the
energies of different states are compared to find the ground state.

###  II.2 Dynamical mean-field theory (DMFT)

A recent study has shown that the transition to the magnetic state in Graphene
is captured remarkably well by the inclusion of local charge fluctuations
Raczkowski  _et al._ (2020) in the framework of a single-site dynamical mean-
field theory Georges  _et al._ (1996). Thus, to go beyond the static MFT and
to include local fluctuations, we also use the real-space dynamical mean-field
theory (DMFT) to obtain a magnetic solution of the ZPNR. As in Ref. Phùng  _et
al._ (2020), each atom of a $8\times 48$ large cluster is mapped onto its own
quantum impurity model by calculating the Green’s function and the local
hybridization function,

$\displaystyle\bf{G}_{ij}(\omega)$ $\displaystyle=$
$\displaystyle(\omega-\bf{H}-\bf{\Sigma}(\omega))_{ij}^{-1}$ (2)
$\displaystyle\bf{\Delta}_{i}(\omega)$ $\displaystyle=$
$\displaystyle\bf{G}_{ii}^{-1}(\omega)+\bf{\Sigma}_{ii}(\omega),$ (3)

where $i$ and $j$ are indices for the positions of the atoms, $\bf{H}$ is the
matrix of the tight-binding Hamiltonian on the finite lattice,
$\bf{\Sigma}(\omega)$ is the diagonal matrix including the self-energies of
all atoms $\bf{\Sigma}_{ij}(\omega)=\bf{\Sigma}_{ij}(\omega)\delta_{ij}$,
$\bf{\Sigma}_{ii}$ is the self-energy of the atom $i$, $\bf{G}_{ij}$ is the
Green’s function matrix, and $\bf{\Delta}_{i}$ is the hybridization function
of atom $i$. The hybridization function together with the local interaction
strength completely defines a quantum impurity model necessary in DMFT, which
makes it possible to calculate magnetic states in large clustersPeters and
Kawakami (2014, 2015); Raczkowski  _et al._ (2020); Phùng  _et al._ (2020).
The quantum impurity model is then solved using the numerical renormalization
group (NRG)Wilson (1975); Bulla  _et al._ (2008), which can calculate
dynamical correlation functions and self-energies with high accuracyPeters
_et al._ (2006).

Figure 2: Evolution of the single-particle gap (a) and the edge magnetization
(b) as a function of the Hubbard interaction $U/|t_{1}|$. Results for
different ZPNR widths $N_{x}=7,8,9$ are plotted with different symbols. In
panel (a), the inset shows an exponential curve-fitting of the gap evolution
as a function of $1/N_{x}$ at $U/|t_{1}|=0.8$. In panel (b), the inset shows
the evolution of the maximum magnetization $m^{z}_{\rm max}$ versus the
Hubbard interaction. Three different regimes are labeled and highlighted as:
(I) nonmagnetic, (II) gapped-IC, and (III) gapped-AFM (or gapped-FM) regions.
The ribbon length is fixed at $N_{y}=120$, and the periodic boundary condition
is implemented in the $y$-direction. Figure 3: Three different magnetic
spatial configurations for $U/|t_{1}|=0.3$ (left panel) and $1.0$ (middle and
right panels) of the Hubbard interaction. The width and length of the ribbons
are $N_{x}=7$ and $N_{y}=120$. Since we here use a very long ribbon, we only
show a portion of the ribbon in the y-direction. The blue and red circles
display the two different local spin directions. We call the magnetic
configuration in the left, middle, and right panel as IC, AFM, and FM phases.
Figure 4: The left panels present the local magnetic modulation $m_{i}^{z}$ at
one edge of the ZPNRs, and the right panels give their corresponding Fourier
transformation for different Hubbard interactions $U/|t_{1}|$. The lattice
parameters are the same as in Fig. 2.

##  III RESULTS AND DISCUSSION

In this section, we present the numerical results obtained by static and
dynamical MFT. The ZPNR geometry is shown in Fig. 1. The geometry is specified
by two parameters, $N_{x}$ and $N_{y}$, which are the width and the length of
the cluster. We use open boundaries in the x-direction. A single ZPNR unit-
cell is shown as a black box in Fig. 1. For our static MFT calculations, we
use $N_{y}=120$ unit-cells in y-direction and apply periodic boundary
conditions. Thus, our calculation includes $N=4\times N_{x}\times N_{y}$
phosphorus atoms. The ribbon width plays an essential role in the creation of
the edge states Taghizadeh Sisakht  _et al._ (2015), and it has been shown
that the ribbon width must be larger than about $3~{}nm$ for stable edge
magnetism, which corresponds to $N_{x}=7$ in this work.

###  III.1 The pristine ZPNRs

We first consider a finite ribbon cluster. Later, we exploit translation
symmetry and extend our study to an infinite ribbon. We focus here on the
single-particle gap and the edge magnetization in our analysis, which are two
practical observables to understand the magnetic features of ZPNRs. The
single-particle gap is here defined as one half of the charge gap,
$\Delta_{\rm sp}=(E_{n-1}-2\,E_{n}+E_{n+1})/2$, where $E_{n}$ is the ground-
state energy in the sector with $n$ electrons. The edge magnetization is
defined as $m^{z}=\frac{1}{N_{\rm edge}}\sum_{i\in{\rm edge}}^{N_{\rm
edge}}\,|\langle m_{i}^{z}\rangle|$. The temperature is set to zero.

Figure 2 shows the evolution of the single-particle gap (a) and the edge
magnetizations (b) as a function of the Hubbard interaction $U/|t_{1}|$,
calculated by static MFT. The gap is zero for interaction strengths
$U<U_{c1}\simeq 0.2|t_{1}|$. At this point, the edge magnetism starts to
appear. For $0.2\lesssim U/|t_{1}|\lesssim 0.6$, the magnetization at the edge
is not homogeneous. Precisely at the critical point, $U_{c1}$, the
magnetization pattern of one edge is an antiferromagnetic state, whose
existence has been reported in Ref. Du  _et al._ (2015). Further increasing
the interaction strength, the magnetic state becomes an incommensurate (IC)
antiferromagnetic state (see the left panel in Fig. 3 and Fig. 4). A more
detailed analysis is given in the next section. As can be seen from Fig. 2(a),
the gap starts to increase from the first critical point $U_{c1}\simeq
0.2|t_{1}|$. Surprisingly, the band gap forms a cusp, and decreases for
stronger interaction strengths until it reaches the second critical point
$U_{c2}\simeq 0.6|t_{1}|$. Beyond $U_{c2}\simeq 0.6|t_{1}|$, the gap shows a
linear growth with the Hubbard interaction. The magnetic configuration in this
region is illustrated in the middle panel of Fig. 3, and we refer to it as the
AFM phase. While the magnetization along the edges is homogeneous, the
magnetization is exactly opposite at both edges. This configuration has also
been predicted by DFT Zhu  _et al._ (2014) and QMC Yang  _et al._ (2016).
However, besides this AFM phase, we here find another magnetic solution
illustrated in the right panel of Fig. 3. In this magnetic configuration, both
edges are ferromagnetically aligned, and interestingly its gap and
magnetization behavior are almost the same as in the AFM case. Comparing the
ground state energies of the AFM and the FM states, we find an exponentially
small energy difference of $\mathcal{O}(10^{-6})$ (see Table.1). It is worth
mentioning that to find the AFM or the FM states, a proper initial guess is
necessary while finding the IC phase does not require such an initial guess.
We highlight and label the ZPNRs phases in Fig. 2 as follows: (I) nonmagnetic,
(II) gapped-IC, and (III) gapped-AFM (or gapped-FM ) regions.

It is intriguing to see that an FM state in a half-filled Hubbard model has a
slightly lower ground state energy than the AFM state. However, we might
explain this ferromagnetic state by using the argumentation of Stoner
ferromagnetism: The wavefunctions of the edge states of the left and the right
edges have some overlap with each other if the width of the ZPNR is finite.
This overlap will lead to an additional positive energy contribution in the
case of an AFM state, which can be prevented by ferromagnetically aligning
both edges. Thus, the ferromagnetic state has slightly lower energy. The
situation could be similar to graphene zigzag ribbons, where a sharp
semiconductor (AF) to metallic (FM) transition occurs by varying the ribbon
width, which is seen experimentally Magda  _et al._ (2014) and theoretically
Chen  _et al._ (2017).

$U/t_{1}$ | 0.8 | 0.9 | 1.0
---|---|---|---
E[eV]/N | -3.1342389394 | -3.0761689920 | -3.0181486070
FM |  |  |
E[eV]/N | -3.1342358713 | -3.0761659724 | -3.0181456815
AF |  |  |
Table 1: Ground-state energy per atom for the gapped-AFM and the gapped-FM
phase for three interaction strengths. Figure 5: The position of the maximum
amplitude in the frequency domain as a function of the Hubbard interaction
$U/|t_{1}|$ in Fig. 4.
Figure 6: Top and middle panels show spectral spectrum which extracted by
unfolding energy spectrum of extended unit-cell with $Ny=120$. The bottom
panel shows the evolution of single-particle gaps. Regions are as follows: (I)
nonmagnetic, (II) gapped-IC, and (III) gapped-AFM. The width of the ribbon is
fixed at $N_{x}=7$.

Figure 2(b) shows the edge magnetization $m^{z}$. To calculate the edge
magnetization, we only consider lattice sites along the border of the zigzag
edge. The general behavior is consistent with the gap evolution. A small
magnetization appears at the first critical point $U_{c1}\simeq 0.2|t_{1}|$
and increases very slowly with $U$ until the second critical point
$U_{c2}\simeq 0.6|t_{1}|$. Beyond $U_{c2}$, the edge magnetization $m^{z}$
saturates. We note that our results of phase (III) are in agreement with the
quantum Monte Carlo result reported in Ref. Yang  _et al._ (2016), which
predicted long-range order for $U>0.5~{}eV$ at zero temperature. The inset of
Figs. 2(b) shows the maximum of the edge magnetization $m^{z}_{\rm max}$.
Interestingly, it captures both the first and the second critical point
consistent with the gap evolution.

To obtain more insight into the effect of the ribbon’s width on the gap and
the magnetization, we present in Fig. 2 data of different widths as a
comparison. One can see that the gap does not depend on the ribbon width for
$N_{x}=7,8,9$, for which the data collapse on top of each other. However, for
ribbon widths smaller than $N_{x}<7$, we find that the gap depends on the
width. The inset in Fig. 2(a) displays the gap evolution with the inverse
ribbon width, fitted by an exponential curve. We find that the edge
magnetization, $m^{z}$, also collapses on a single curve for $N_{x}>6$. In
particular, all data show the same saturation value in the region-(III).
However, we note that the QMC calculation Yang  _et al._ (2016) for room
temperature has shown that the magnetization decreases with the ribbon width.

Let us now analyze the IC phase in more detail by accessing the real space
data of the local magnetization, $m_{i}^{z}$. The real-space data of the local
magnetization, $m_{i}^{z}$, reveals how an antiferromagnetic state at one edge
changes into the ferromagnetic state when increasing the interaction strength.
The local magnetization and its Fourier transformation (FT) are shown in Figs.
4 for different interaction strengths. For small $U/|t_{1}|=0.25$, the local
magnetization pattern is an antiferromagnetic state along one edge, as also
demonstrated by the FT with a single peak at $\nu_{\rm max}=\pi$. By
increasing $U$, one can see how the local magnetization starts to change. The
single peak in the FT splits into two, which move away from $\pi$. Finally,
for a Hubbard interactions larger than $U>U_{c2}$, the maximum in the FT
occurs at $\nu_{\rm max}=0$, which signals a fully aligned ferromagnetic state
along one edge. In Fig 5, we show the position of the maximum in the FT
$\nu_{\rm max}/\pi$ plotted as function of $U/|t_{1}|$. It can be seen how the
maximum decreases from $1$ to $0$ within the IC phase.

Figure 7: DMFT calculations of edge magnetization (top panel) as a function of
Hubbard interaction $U/|t_{1}|$ for a ZPNR width $N_{x}=8$. Two different
regimes are labeled and highlighted as: (I) nonmagnetic, and (II) AFM gapped
(or FM gapped ) regions. Bottom panels show the spectral functions $A(E)$ for
three different Hubbard interactions. The ribbon length is fixed at $N_{y}=48$
and the periodic boundary condition is implemented in the $y$-direction.
Figure 8: Same as Fig.6, but for the FM case. Regions are as follows: (I)
nonmagnetic, (II) gapped-IC, and (III) gapped-FM. Red and blue lines in the
spectral functions correspond to different spin-directions.

Now, we use the translational symmetry of the lattice and probe the magnetic
features of an infinite ribbon. To this end, we focus on the energy dispersion
of an infinite ribbon. For a given wavenumber $k$ and spin $\sigma$ the mean-
field Hamiltonian has $N$ states $\Psi_{k\sigma}(x)$ with energy
$\epsilon_{k\sigma}(x)$. By implementing the iterative self-consistent
technique Fernández-Rossier and Palacios (2007) in the Brillouin zone (BZ), we
can recover the magnetization configuration in ZPNRs. It worth to mention that
the magnetic unit-cell in the IC phase is much larger than the lattice unit-
cell. Thus, we use an unfolding technique to extract the spectral function.
Detailed information about the unfolding is given in the appendix A.

In Fig. 6, the spectral function of the edge states and the corresponding gap
evolution are illustrated. We show that the quasi-flat bands of the edges are
isolated from the bulk bands, which is the most prominent feature of ZPNRs. It
is already known that the hopping $t_{4}$ term is responsible for the
dispersion of these flat bands Ezawa (2014). We also note that these quasi-
flat modes are almost doubly degenerate. As shown in Fig. 6 in the non-
magnetic region ($\bar{U}\equiv U/|t_{1}|=0.2$), these bands are degenerate at
the BZ boundaries and split toward the BZ center. Indeed, this splitting
becomes smaller for wider ribbons. The following reasoning may explain this:
increasing the width of the ZPNRs will reduce the interaction between both
edges, leading to the decrease of the edge splitting at the BZ center. When
entering the magnetic phase ($\bar{U}\gtrsim 0.2$), the bands split at the
$kb=\pm\pi/2$ points at the Fermi energy. With increasing interaction
strength, the gap size increases, and spectral weight is shifted at $k=0$
above the Fermi energy and at $kb=\pm\pi$ below the Fermi energy. This shift
of spectral weight causes the single-particle gap to decrease before entering
the ferromagnetic phase (region-(III)). Finally, the spectral weight above and
below the Fermi energy form two quasiparticle bands for $U>0.6$. In
region-(III) with commensurate (here, gapped-AFM) edge magnetism, one band is
shifted toward higher energies, and both bands are separated. The gap is
clearly visible in this phase, and we can read off the gap in the energy
dispersion. The gap evolution is thereby similar as in Fig. 2 directly
calculated by the energy. We find a gap opening when entering the IC phase.
The gap width forms a maximum in the IC phase, decreases towards the AFM
phase, and finally increases linearly in the AFM phase.

To further validate our MFT results, we will now show DMFT results. As
mentioned above, DMFT has proven to predict the critical point in Graphene
adequately when compared to lattice-QMC. We here use DMFT for a cluster with
parameters $N_{x}=8$ and $N_{y}=48$. Our results are summarized in Fig. 7. For
weak interaction strengths, $0.7<U/|t_{1}|$, we find a nonmagnetic solution.
We note that any (even a magnetic) initial guess for these interaction
strengths converges to the same nonmagnetic state. Furthermore, we find a
stable magnetic state at the edges of ZPNR for $U/|t_{1}|>1.4$, which
corresponds to phase (III) in MFT. As with MFT, we can find a stable AFM and a
stable FM state. As in Graphene, local fluctuations included by DMFT shift the
critical point to stronger interaction strengths compared to static MFT. More
interestingly is the question about the existence of the IC phase. For
interaction strengths $U/|t_{1}|<1.4$, we do not find a converged magnetic
solution. However, for $0.7<U/|t_{1}|<1.4$, we find a small magnetization at
the edges of ZPNR, which does not vanish when iterating the DMFT calculation.
If we start the DMFT in this regime with an inhomogeneous magnetic state, the
magnetization configuration changes in each iteration without completely
vanishing, but we cannot find a converged solution. We note here that to find
an incommensurate state with DMFT, a large cluster and an appropriate initial
guess are necessary. Thus, we interpret these DMFT calculations as an attempt
to stabilize an incommensurate state. However, as we cannot find a converged
solution, we cannot calculate further properties of this phase.

An advantage of DMFT over static MFT is that spectral functions can readily be
calculated and include lifetime effects due to correlations. The spectral
functions calculated by DMFT are shown in Fig. 7. In contrast to static MFT,
DMFT already includes modifications of the spectral function in the
nonmagnetic phase (I). For $U/|t_{1}|=0.8$ (converged nonmagnetic solution),
we find some (blurred) spectral weight below the quasiparticle band at $k=0$.
This spectral weight should correspond to the splitting of the quasiparticle
band at the center of the BZ, which is smeared out because of correlations.
Furthermore, we find some spectral weight above the quasiparticle band at
$kb=\pm\pi$. With increasing interaction strength, spectral weight is
particularly transferred from the quasiparticle band lying at the Fermi energy
to energies below the Fermi energy, slowly forming a second band. At the same
time, the spectral weight moves to higher energies at $kb=\pm\pi$. These two
processes finally form two bands, which are clearly visible for
$U/|t_{t}|=1.6$ with a gap between them. While in static MFT, the formation of
a gap takes place when entering the AFM phase (III), in DMFT this separation
already starts in the nonmagnetic phase due to local fluctuations.

Finally, we want to examine the above mentioned gapped-FM phase in more
detail, which coexists with the gapped-AFM phase. We repeat the previous
calculations, using a proper initial guess to obtain the FM phase. The results
are depicted in Fig. 8. We note that when both edges are ferromagnetically
aligned, we can also find the IC phase. The gap evolution reveals a small dome
in the region-(II). The gap opens at the first critical point, $U_{c1}$, forms
a maximum and then decreases when approaching the second critical point
$U_{c2}$. In the region-(III), the gap increases linearly with the Hubbard
strength interaction. It is interesting to note that the energy dispersion of
the AFM state (Fig. 6) and the FM state (Fig. 8) are almost identical.
However, while in the AFM state, all edge modes are spin-degenerate, in the FM
state, the edge modes above the Fermi energy have a definite spin-directions
and the edge modes below the Fermi energy exhibit an opposite spin direction.
Furthermore, there is a slight additional splitting of the edge modes around
$k=0$ in the FM state, which is absent in the AFM state. This small splitting
is responsible for the energy difference between the AFM and the FM state.

###  III.2 Strain effects

Next, we want to study the impact of strain and disorder on the magnetic
state. To study strain effects, we follow the approach developed in Ref. Jiang
and Park (2015); Mohammadi and Arghavani Nia (2016). We will focus here on the
tensile strain in the normal direction to the phosphorene plane Huang and Xing
(2014). By applying an axial strain, following the Harrison relation Harrison
(2004), the strain-induced modified hopping parameter in the linear regime can
be written as
$t_{i}\approx(1-2\alpha_{x}^{i}\epsilon_{x}-2\alpha_{y}^{i}\epsilon_{y}-2\alpha_{z}^{i}\epsilon_{z})$,
where $\alpha_{i}^{j}$ are coefficients related to the structure of
phosphorene and $\epsilon_{j}$ is the strain in the $j$-direction.

Before exploring strain effects on the magnetic features, we briefly comment
on the energy dispersion under strain. Figure 9 presents the energy dispersion
for three different strengths of tensile strain
$\epsilon_{z}=0.0\%,~{}10\%,~{}20\%$ in the absence of the Hubbard
interaction. It can be seen that the tensile strain has a significant impact
on the band structure: The edge modes are split, which is accompanied by a
compression of the bulk bands. Even for strain $\epsilon_{z}=20\%$, the
degeneracy of the edge modes at the BZ boundaries survives, while one of the
split levels crosses the Fermi energy at $k=0$. We also note that for strain
$\epsilon_{z}=10\%$, the bulk band gets flattened, which is analogous to the
strain-induced Landau Levels effects in graphene Chang  _et al._ (2012); de
Juan  _et al._ (2013); Roy  _et al._ (2013); Yang  _et al._ (2017).

Figure 9: Energy dispersion of ZPNR shown for three different strengths of
tensile strain $\varepsilon_{z}=0.0\%,~{}10\%,~{}20\%$ in the absence of the
Hubbard interaction $U$. The two quasi-flat edges, isolated from the bulk, are
colored in gold. The width of the ribbon is the same as in Fig. 4. The
horizontal red line marks the Fermi level. Figure 10: Panels (a) and (c) show
the gap and edge magnetization for different ribbon widths at fixed tensile
strain $\varepsilon_{z}=10\%$. In panels (b) and (d), we fix the ribbon width,
$N_{x}=7$, and show three different strengths of tensile strain
$\varepsilon_{z}=0.0\%,~{}10\%,~{}20\%$. The inset in panel (d) shows a log-
log plot of the $m^{z}$ evolution as a function of the tensile strain strength
at $U/|t_{1}|=0.8$ (the line is a power-law fit).

We now explore the evolution of the gap and the magnetization as a function of
the Hubbard interaction in the presence of tensile strain. Figures 10 (a) and
(c) show the results for different ribbon widths with fixed strain
$\epsilon_{z}=10\%$. Figures 10 (b) and (d) show the results for three
different strengths of tensile strain $\epsilon_{z}=0.0\%,~{}10\%,~{}20\%$
with a fixed ribbon width $N_{x}=7$. One profound effect of tensile strain is
the destruction of the intermediate IC phase. For $\epsilon_{z}=~{}20\%$, the
IC phase has almost vanished. We furthermore notice that the second critical
point, $U_{c2}$, shifts to a larger value. Thus, the tensile strain has a
tremendous impact on edge magnetism. This can be understood by the following
explanation: under tensile strain, the $t_{2}$ and $t_{4}$ hopping parameters
change more strongly than the others. As shown in Fig. 9, the tensile strain
splits the edge modes and increases their width. The increased bandwidth of
these modes makes a larger interaction strength necessary to stabilize the AFM
phase. Furthermore, as mentioned earlier, the $t_{4}$ hopping term is
important for the shape of the edge mode and thus plays an essential role in
stabilizing the IC phase. Extrapolating the magnetization $m_{z}$ to larger
values of tensile strain for $U/|t_{1}|=0.8$, shown in the inset of Fig.
10(d), we find that magnetism should vanish at about $\epsilon_{z}=50\%$,
which is much higher than the prediction by first principles in Ref Du  _et
al._ (2015). We note that a strain of about $\epsilon_{z}=50\%$ is already big
enough to destroy the whole structure of the edge modes. Thus, at the MFT
level, we can conclude that a magnetic to nonmagnetic transition is not
feasible.

Figure 11: Evolution of the single-particle gap (upper panel) and edge
magnetization (lower panel) as a function of the Hubbard interaction $U$. Data
for clean and disorder cases $w_{i}/|t_{1}|=0.0,~{}0.3$ are shown. The width
of the ribbon is the same as in Fig. 4.

###  III.3 Disorder effects

To study the effects of disorder, we include an additional term along the
edges in the Hamiltonian $H_{w}=\sum_{i,\sigma}w_{i}n_{i,\sigma}$,
corresponding to non-magnetic disorder. $w_{i}$ is the strength of the
disorder at site $i$, which is randomly chosen in the interval $[-w/2;w/2]$.
Because the translational symmetry along the y-direction is broken, we solve
for the ground state in a finite cluster with $N_{y}=120$ using the periodic
boundary condition. To be independent of a special configuration of the
disorder, we average over $100$ different realizations. The influence of the
edge disorder on the magnetic phases is presented in Fig. 11. It can be seen
that both, the gap and edge magnetization, are robust against disorder. We
have not found any deviation in the saturation value of the edge magnetization
in the gapped-AF(FM) phase. Moreover, the evolution of the gap also indicates
the existence of the gapped-IC phase in the disordered system. This is in
contrast to the result of strain in the preceding section. However, as
mentioned before, the hopping parameters $t_{2}$ and $t_{4}$, play an
essential role in stabilizing the edge states and its corresponding magnetic
features. Thus, introducing Anderson type disorder will not destabilize these
states.

##  IV SUMMARY

We have investigated the edge-state magnetic properties of black phosphorene
nanoribbons using a tight-binding model with an electron-electron Hubbard
interaction $U$. Our study aimed to explore the magnetic features of large
clusters of black phosphorene nanoribbons for which numerically expensive
techniques such as density functional theory and quantum Monte Carlo
techniques are not feasible. Thus, to study the model, we have used a
combination of static and dynamical mean-field theory (DMFT). While our
calculations for large $U$ are in agreement with previous results, we find an
incommensurate magnetic phase for weak interactions. Performing a detailed
Fourier analysis of the magnetization evolution in the incommensurate (IC)
phase, we find a second critical interaction $U_{c2}$ at which the IC phases
changes to an antiferromagnetic (AFM) or ferromagnetic (FM) phase. Finally, we
have analyzed the influence of strain and disorder on the magnetic properties.
Our results show that while the IC phase is robust to Anderson type disorder,
it is fragile against strain.

###### Acknowledgements.

This work was supported by the Paris//Seine excellence initiative. J. Vahedi
is also partially supported by Iran Science Elites Federation, Grant
No.11/66332. R.P. is supported by JSPS, KAKENHI Grant No. JP18K03511.

##  Appendix A Unfolding of the Green’s function

When calculating the band structure for the long-range spin-density waves, we
use extended unit cells including many layers in the y-direction. This yields
a folded band structure, which is difficult to compare with the ferromagnetic
or nonmagnetic state. We therefore unfold the band structure using the Green’s
function as described here. When calculating the cluster Green’s function with
open boundary conditions in the x-direction, but including $k_{y}$ momentum
dependence in the y-direction we obtain

$G^{k_{y}^{\prime}}_{y_{1}^{\prime},y_{2}^{\prime}}(\omega)=\left(\omega+i\eta-
H_{k_{y}^{\prime}}\right)^{-1}_{y_{1}^{\prime},y_{2}^{\prime}},$ (4)

where $0\leq y_{1,2}^{\prime}<N_{y}$ correspond to the $y$-component of
different lattice sites in the unit cell and $H_{k_{y}}$ is the Hamiltonian
for the momentum $k_{y}$. Because we do not change the $x$-coordinate, we
neglect it for convenience. To calculate the unfolded Green’s function, we
need to calculate

$G_{k_{y}}(\omega)=\frac{1}{N}\sum_{y_{1},y_{2}}\exp\left(-ik_{y}(y_{1}-y_{2})\right)G_{y_{1},y_{2}}(\omega)$
(5)

where $0\leq y_{1,2}<N$ and $N$ the is number of lattice sites of the full
lattice which includes $M$ unit cells with $N_{y}$ atoms, thus $N=MN_{Y}$. We
can calculate $G_{y_{1},y_{2}}(\omega)$ from the cluster Green’s function as

$\displaystyle G_{y_{1},y_{2}}(\omega)$ $\displaystyle=$
$\displaystyle\frac{1}{M}\sum_{k^{\prime}_{y}}G^{k_{y}^{\prime}}_{y_{1}^{\prime},y_{2}^{\prime}}(\omega)\exp\left(ik_{y}^{\prime}(n_{1}-n_{2})\right)$
(6) $\displaystyle y_{1}$ $\displaystyle=$ $\displaystyle
n_{1}N_{y}+y_{1}^{\prime}$ (7) $\displaystyle y_{2}$ $\displaystyle=$
$\displaystyle n_{2}N_{y}+y_{2}^{\prime}$ (8)

We can now calculate the unfolded Green’s function as

$\displaystyle G_{k_{y}}(\omega)$ $\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{y_{1},y_{2}}\exp\left(-ik_{y}(y_{1}-y_{2})\right)\frac{1}{M}\sum_{k^{\prime}_{y}}G^{k_{y}^{\prime}}_{y_{1}^{\prime},y_{2}^{\prime}}(\omega)\exp\left(ik_{y}^{\prime}(n_{1}-n_{2})\right)$
(9) $\displaystyle=$
$\displaystyle\frac{1}{NM}\sum_{n_{1},n_{2}}\sum_{y_{1}^{\prime},y_{2}^{\prime}}\sum_{k_{y}^{\prime}}\exp\left(-ik_{y}((n_{1}-n_{2})N_{y}+y_{1}^{\prime}-y_{2}^{\prime})\right)G^{k_{y}^{\prime}}_{y_{1}^{\prime},y_{2}^{\prime}}(\omega)\exp\left(ik_{y}^{\prime}(n_{1}-n_{2})\right)$
$\displaystyle=$
$\displaystyle\frac{M^{2}}{NM}\sum_{y_{1}^{\prime},y_{2}^{\prime}}\sum_{k^{\prime}_{y}}\delta_{N_{y}k_{y},k_{y}^{\prime}}\exp\left(-ik_{y}(y_{1}^{\prime}-y_{2}^{\prime})\right)G^{k_{y}^{\prime}}_{y_{1}^{\prime},y_{2}^{\prime}}(\omega)$
$\displaystyle=$
$\displaystyle\frac{1}{N_{y}}\sum_{y_{1}^{\prime},y_{2}^{\prime}}\exp\left(-ik_{y}(y_{1}^{\prime}-y_{2}^{\prime})\right)G^{N_{y}k_{y}}_{y_{1}^{\prime},y_{2}^{\prime}}(\omega)$

Finally, we can calculate the spectral functions as shown in the main text as

$A_{k_{y}}(\omega)=-\frac{1}{\pi}\text{Im}\left(G_{k_{y}}(\omega)\right)$ (10)

## References

  * Liu  _et al._ (2014) H. Liu, A. T. Neal, Z. Zhu, Z. Luo, X. Xu, D. Tománek, and P. D. Ye, “Phosphorene: An unexplored 2d semiconductor with a high hole mobility,” ACS Nano 8, 4033–4041 (2014).
  * Castellanos-Gomez (2016) A. Castellanos-Gomez, “Why all the fuss about 2d semiconductors?” Nature Photonics 10, 202 (2016).
  * Koenig  _et al._ (2014) Steven P. Koenig, Rostislav A. Doganov, Hennrik Schmidt, A. H. Castro Neto, and Barbaros Özyilmaz, “Electric field effect in ultrathin black phosphorus,” Applied Physics Letters 104, 103106 (2014).
  * Ghosh  _et al._ (2017) Barun Ghosh, Piyush Kumar, Anmol Thakur, Yogesh Singh Chauhan, Somnath Bhowmick, and Amit Agarwal, “Anisotropic plasmons, excitons, and electron energy loss spectroscopy of phosphorene,” Phys. Rev. B 96, 035422 (2017).
  * Wang  _et al._ (2015) X. Wang, A. M. Jones, K. L. Seyler, V. Tran, Y. Jia, H. Zhao, H. Wang, L. Yang, X. Xu, and F Xia, “Highly anisotropic and robust excitons in monolayer black phosphorus,” Nature Nanotechnology 10, 517 (2015).
  * Linder and Yokoyama (2017) J. Linder and T. Yokoyama, “Anisotropic andreev reflection and josephson effect in ballistic phosphorene,” Phys. Rev. B 95, 144515 (2017).
  * Zare  _et al._ (2017) M. Zare, B. Z. Rameshti, F. G. Ghamsari, and R. Asgari, “Thermoelectric transport in monolayer phosphorene,” Phys. Rev. B 95, 045422 (2017).
  * Ezawa (2014) M. Ezawa, “Topological origin of quasi-flat edge band in phosphorene,” New Journal of Physics 16, 115004 (2014).
  * Taghizadeh Sisakht  _et al._ (2015) E. Taghizadeh Sisakht, M. H. Zare, and F. Fazileh, “Scaling laws of band gaps of phosphorene nanoribbons: A tight-binding calculation,” Phys. Rev. B 91, 085409 (2015).
  * Ma  _et al._ (2016) R. Ma, H. Geng, W. Y. Deng, M. N. Chen, L. Sheng, and D. Y. Xing, “Effect of the edge states on the conductance and thermopower in zigzag phosphorene nanoribbons,” Phys. Rev. B 94, 125410 (2016).
  * Castro Neto  _et al._ (2009) A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, “The electronic properties of graphene,” Rev. Mod. Phys. 81, 109 (2009).
  * Shakouri  _et al._ (2015) Kh. Shakouri, H. Simchi, M. Esmaeilzadeh, H. Mazidabadi, and F. M. Peeters, “Tunable spin and charge transport in silicene nanoribbons,” Phys. Rev. B 92, 035413 (2015).
  * Islam  _et al._ (2018) SK Firoz Islam, Paramita Dutta, A. M. Jayannavar, and Arijit Saha, “Probing decoupled edge states in a zigzag phosphorene nanoribbon via RKKY exchange interaction,” Phys. Rev. B 97, 235424 (2018).
  * Yazyev (2010) O. V. Yazyev, “Emergence of magnetism in graphene materials and nanostructures,” Reports on Progress in Physics 73, 056501 (2010).
  * Feldner  _et al._ (2010) H. Feldner, Z. Y. Meng, A. Honecker, D. Cabra, S. Wessel, and F. F. Assaad, “Magnetism of finite graphene samples: Mean-field theory compared with exact diagonalization and quantum monte carlo simulations,” Phys. Rev. B 81, 115416 (2010).
  * Feldner  _et al._ (2011) H. Feldner, Z. Y. Meng, T. C. Lang, F. F. Assaad, S. Wessel, and A. Honecker, “Dynamical signatures of edge-state magnetism on graphene nanoribbons,” Phys. Rev. Lett. 106, 226401 (2011).
  * Magda  _et al._ (2014) G. Magda, X. Jin, I. Hagymási, P. Vancsó, Z. Osváth, P. Nemes-Incze, C. Hwang, L. P. Biró, and L. Tapasztó, “Room-temperature magnetic order on zigzag edges of narrow graphene nanoribbons,” Nature 514, 608 (2014).
  * Zhu  _et al._ (2014) Z Zhu, C Li, W. Yu, D. Chang, Q. Sun, and Y. Jia, “Magnetism of zigzag edge phosphorene nanoribbons,” Applied Physics Letters 105, 113105 (2014).
  * Du  _et al._ (2015) Y. Du, H. Liu, B. Xu, L. Sheng, J. Yin, C.G. Duan, and X. Wan, “Unexpected magnetic semiconductor behavior in zigzag phosphorene nanoribbons driven by half-filled one dimensional band,” Sci. Rep. 5, 8921 (2015).
  * Farooq  _et al._ (2016) M. U. Farooq, A. Hashmi, and J. Hong, “Ferromagnetism controlled by electric field in tilted phosphorene nanoribbon,” Sci. Rep. 6, 26300 (2016).
  * Yang  _et al._ (2016) G. Yang, S. Xu, W. Zhang, T. Ma, and C. Wu, “Room-temperature magnetism on the zigzag edges of phosphorene nanoribbons,” Phys. Rev. B 94, 075106 (2016).
  * Raczkowski  _et al._ (2020) Marcin Raczkowski, Robert Peters, Thị Thu Phùng, Nayuta Takemori, Fakher F. Assaad, Andreas Honecker, and Javad Vahedi, “Hubbard model on the honeycomb lattice: From static and dynamical mean-field theories to lattice quantum monte carlo simulations,” Phys. Rev. B 101, 125103 (2020).
  * Fernández-Rossier and Palacios (2007) J. Fernández-Rossier and J. J. Palacios, “Magnetism in graphene nanoislands,” Phys. Rev. Lett. 99, 177204 (2007).
  * Bhowmick and Shenoy (2008) S. Bhowmick and V. B. Shenoy, “Edge state magnetism of single layer graphene nanostructures,” The Journal of Chemical Physics 128, 244717 (2008).
  * Gui  _et al._ (2008) G. Gui, J. Li, and J. Zhong, “Band structure engineering of graphene by strain: First-principles calculations,” Phys. Rev. B 78, 075435 (2008).
  * Wang  _et al._ (2014) B. Wang, J. Wu, X. Gu, H. Yin, Y. Wei, R. Yang, and M. Dresselhaus, “Stable planar single-layer hexagonal silicene under tensile strain and its anomalous Poisson’s ratio,” Applied Physics Letters 104, 081902 (2014).
  * Yue  _et al._ (2012) Q. Yue, J. Kang, Z. Shao, X. Zhang, S. Chang, G. Wang, S. Qin, and J. Li, “Mechanical and electronic properties of monolayer MoS2 under elastic strain,” Physics Letters A 376, 1166 – 1170 (2012).
  * Tabatabaei  _et al._ (2013) S. M. Tabatabaei, M. Noei, K. Khaliji, M. Pourfath, and M. Fathipour, “A first-principles study on the effect of biaxial strain on the ultimate performance of monolayer MoS2-based double gate field effect transistor,” Journal of Applied Physics 113, 163708 (2013).
  * Rodin  _et al._ (2014) A. S. Rodin, A. Carvalho, and A. H. Castro Neto, “Strain-induced gap modification in black phosphorus,” Phys. Rev. Lett. 112, 176801 (2014).
  * Elahi  _et al._ (2015) M. Elahi, K. Khaliji, S. M. Tabatabaei, M. Pourfath, and R. Asgari, “Modulation of electronic and mechanical properties of phosphorene through strain,” Phys. Rev. B 91, 115412 (2015).
  * Taghizadeh Sisakht  _et al._ (2016) E. Taghizadeh Sisakht, F. Fazileh, M. H. Zare, M. Zarenia, and F. M. Peeters, “Strain-induced topological phase transition in phosphorene and in phosphorene nanoribbons,” Phys. Rev. B 94, 085417 (2016).
  * Rudenko and Katsnelson (2014) A. N. Rudenko and M. I. Katsnelson, “Quasiparticle band structure and tight-binding model for single- and bilayer black phosphorus,” Phys. Rev. B 89, 201408 (2014).
  * Phùng  _et al._ (2020) Thị Thu Phùng, Robert Peters, Andreas Honecker, Guy Trambly de Laissardière, and Javad Vahedi, “Spin-caloritronic transport in hexagonal graphene nanoflakes,” Phys. Rev. B 102, 035160 (2020).
  * Georges  _et al._ (1996) A. Georges, G. Kotliar, W. Krauth, and M. J. Rozenberg, “Dynamical mean-field theory of strongly correlated fermion systems and the limit of infinite dimensions,” Rev. Mod. Phys. 68, 13 (1996).
  * Peters and Kawakami (2014) Robert Peters and Norio Kawakami, “Spin density waves in the hubbard model: A dmft approach,” Phys. Rev. B 89, 155134 (2014).
  * Peters and Kawakami (2015) Robert Peters and Norio Kawakami, “Large and small fermi-surface spin density waves in the kondo lattice model,” Phys. Rev. B 92, 075103 (2015).
  * Wilson (1975) Kenneth G. Wilson, “The renormalization group: Critical phenomena and the kondo problem,” Rev. Mod. Phys. 47, 773–840 (1975).
  * Bulla  _et al._ (2008) Ralf Bulla, Theo A. Costi, and Thomas Pruschke, “Numerical renormalization group method for quantum impurity systems,” Rev. Mod. Phys. 80, 395–450 (2008).
  * Peters  _et al._ (2006) Robert Peters, Thomas Pruschke, and Frithjof B. Anders, “Numerical renormalization group approach to green’s functions for quantum impurity models,” Phys. Rev. B 74, 245114 (2006).
  * Chen  _et al._ (2017) W. C. Chen, Y. Zhou, S. L. Yu, W. G. Yin, and C. D. Gong, “Width-tuned magnetic order oscillation on zigzag edges of honeycomb nanoribbons,” Nano Letters 17, 4400–4404 (2017).
  * Jiang and Park (2015) Jin-Wu Jiang and Harold S. Park, “Analytic study of strain engineering of the electronic bandgap in single-layer black phosphorus,” Phys. Rev. B 91, 235118 (2015).
  * Mohammadi and Arghavani Nia (2016) Y. Mohammadi and B. Arghavani Nia, “Strain engineering the charged-impurity-limited carrier mobility in phosphorene,” Superlattices and Microstructures 89, 204 – 215 (2016).
  * Huang and Xing (2014) G. Q. Huang and Z. W. Xing, “Semiconductor to metal transition, dynamical stability and superconductivity of strained phosphorene,” ArXiv e-prints (2014), arXiv:1409.7284 [cond-mat.supr-con] .
  * Harrison (2004) W. A. Harrison,  _Elementary Electronic Structure: Revised_ (World Scientic Publishing Company, 2004).
  * Chang  _et al._ (2012) Y. Chang, T. Albash, and S. Haas, “Quantum Hall states in graphene from strain-induced nonuniform magnetic fields,” Phys. Rev. B 86, 125402 (2012).
  * de Juan  _et al._ (2013) F. de Juan, J. L. Mañes, and M. A. H. Vozmediano, “Gauge fields from strain in graphene,” Phys. Rev. B 87, 165131 (2013).
  * Roy  _et al._ (2013) B. Roy, Zi-X. Hu, and K. Yang, “Theory of unconventional quantum Hall effect in strained graphene,” Phys. Rev. B 87, 121408 (2013).
  * Yang  _et al._ (2017) Z. Yang, F. Gao, Y. Yang, and B. Zhang, “Strain-induced gauge field and Landau levels in acoustic structures,” Phys. Rev. Lett. 118, 194301 (2017).