# Multi-Objective Coordination Graphs for the
Expected Scalarised Returns with Generative Flow Models

Conor F. Hayes
National University of Ireland Galway
<EMAIL_ADDRESS>Timothy Verstraeten
Vrije Universiteit Brussel, Belgium
Diederik M. Roijers
HU University of Applied Sciences Utrecht, the Netherlands
Vrije Universiteit Brussel, Belgium Enda Howley
National University of Ireland Galway
Patrick Mannion
National University of Ireland Galway

###### Abstract

Many real-world problems contain both multiple objectives and agents, where a
trade-off exists between objectives. Key to solving such problems is to
exploit sparse dependency structures that exist between agents. For example in
wind farm control a trade-off exists between maximising power and minimising
stress on the systems components. Dependencies between turbines arise due to
the wake effect. We model such sparse dependencies between agents as a multi-
objective coordination graph (MO-CoG). In multi-objective reinforcement
learning (MORL) a utility function is typically used to model a user’s
preferences over objectives. However, a utility function may be unknown a
priori. In such settings a set of possibly optimal policies must be computed.
Which policies are optimal depends on which optimality criterion applies. If
the utility function of a user is derived from multiple executions of a
policy, the scalarised expected returns (SER) must be optimised. If the
utility of a user is derived from the single execution of a policy, the
expected scalarised returns (ESR) criterion must be optimised. For example,
wind farms are subjected to constraints and regulations that must be adhered
to at all times, therefore the ESR criterion must be optimised. For MO-CoGs,
the state-of-the-art algorithms can only compute a set of optimal policies for
the SER criterion, leaving the ESR criterion understudied. To compute a set of
optimal polices under the ESR criterion, also known as the _ESR set_ ,
distributions over the returns must be maintained. Therefore, to compute a set
of optimal policies under the ESR criterion for MO-CoGs, we present a novel
_distributional multi-objective variable elimination_ (DMOVE) algorithm. We
evaluate DMOVE in realistic wind farm simulations. Given the returns in real-
world wind farm settings are continuous, such as generated power, we utilise a
generative flow model known as _real-NVP_ to learn the continuous return
distributions to calculate the _ESR set_.

_K_ eywords multi-objective $\cdot$ multi-agent $\cdot$ coordination graphs
$\cdot$ (nonserial) dynamic programming $\cdot$ expected scalarised returns
$\cdot$ generative flow models

##  1 Introduction

In many real-world problems a trade-off must be made between multiple, often
conflicting objectives (Vamplew et al., 2021; Reymond et al., 2022). For
example, learning a wind farm control policy is a multi-objective problem
(Padullaparthi et al., 2022). Generally, a wind turbine aims to maximise its
power production; however, this can lead to higher stress being induced on the
turbine’s components. Therefore, there is a trade-off between power production
and induced damage. However, real-world problems can also have multiple agents
(Rădulescu et al., 2020; Hayes et al., 2020). In this work, we focus on multi-
objective multi-agent settings.

Many multi-agent settings have a sparse dependency structure between the
different agents, which can be exploited. For example, a phenomenon known as
the wake effect occurs when upstream turbines extract energy from the wind,
which increases the turbulence intensity of the wind flow and reduces the
available energy for downstream turbines (González-Longatt et al., 2012). In
order to compensate for this effect, upstream turbines may aim to change their
yaw angle, i.e., the direction that they are facing w.r.t. the incoming wind
vector. This leads to a deflection of the wake, which reduces the power
production of the misaligned turbines, but increases the energy production of
the downstream turbines, which may lead to a higher energy production over the
entire wind farm (van Dijk et al., 2016). Therefore, it is crucial to consider
the actions taken by different agents jointly to guarantee optimality.
However, deriving a joint action control policy by directly considering the
actions jointly is intractable given the number of possible solutions scales
exponentially with the number of agents in the system. Fortunately, the
dependency structure between agents is often sparse. In the case of the wind
farm control setting, we can exploit the wake effect and the resulting
dependencies, given that the decisions made by upstream turbines only affect
neighboring downstream turbines (Verstraeten et al., 2020). To this end, we
aim to model a wind farm as a multi-objective coordination graph (MO-CoG)
(Rollón and Larrosa, 2006; Marinescu et al., 2012; Delle Fave et al., 2011b).

In the multi-objective reinforcement learning (MORL) literature a utility
function is typically used to model a user’s preferences over objectives
(Hayes et al., 2022c). In the real-world a user’s utility function may be
unknown. In this case, a set of optimal policies must be computed (Roijers et
al., 2013). In MORL multiple optimality criteria exist. Which criteria to
optimise is determined by how the utility of a user is derived. If the utility
of a user is derived from multiple executions of a policy (Roijers et al.,
2013), the scalarised expected returns (SER) criterion must be used to learn a
set of optimal policies (for example the Pareto front (Wang and Sebag, 2012)).
However, if the utility of a user is derived from a single execution of a
policy (Roijers et al., 2018; Hayes et al., 2021a), then the expected
scalarised returns (ESR) criterion should be optimised. The set of optimal
policies under the ESR criterion is known as the _ESR set_ which must be
computed using return distributions (Hayes et al., 2021c, 2022d). It is
important that the correct optimality criteria is selected, given the
policies, and set of optimal policies, computed under the SER criterion and
ESR criterion can be different (Rădulescu et al., 2019; Hayes et al., 2022b,
a). For example, a wind farm controller is subjected to various restrictions,
such as contractual constraints or grid regulations. It is important to adhere
to these at all times. Therefore, the utility should be defined on the basis
of a single policy execution, and thus the ESR criterion should be optimised.

Given return distributions must be utilised to compute the ESR set, a
distributional approach to MORL must be taken. Recently, a number of
distributional MORL algorithms, which utilise categorical distributions, have
been deployed to learn policies (Hayes et al., 2022a; Reymond et al., 2021;
Hayes et al., 2021a). In real-world problems the returns are often continuous.
For example, the power and turbulence values for a wind turbine are
continuous. While categorical distributions can be used, they cannot
accurately and efficiently represent return distributions with continuous
returns. Therefore, methods which can compute return distributions when the
returns are continuous must be utilised in real-world settings like wind farm
control.

The state-of-the-art MO-CoG algorithms cannot compute sets of optimal policies
for the ESR criterion (Hayes et al., 2022a, b), because they use expected
value vectors (Rollón, 2008; Delle Fave et al., 2011a; Marinescu et al., 2012;
Roijers et al., 2015). Therefore, we propose a new nonserial dynamic
programming (Rosenthal, 1977) algorithm for MO-CoGs that we call
_distributional multi-objective variable elimination_ (DMOVE). DMOVE maintains
set of return distributions to compute a set of optimal policies under the ESR
criterion. Given the returns for wind farm control are continuous we utilise a
generative flow model (Durkan et al., 2019; Ho et al., 2019) known as _real-
NVP_ (Dinh et al., 2016) to learn return distributions for each policy. We
evaluate DMOVE using the state-of-the-art FLORIS wake simulator (NREL, 2021)
with different realistic wind farm configurations of varying sizes and
dependencies. Specifically, we first learn a model of the vector-valued local
reward functions in the MO-CoG from interacting with FLORIS, and then employ
DMOVE to extract the _ESR set_.

##  2 Background

In this section we introduce the relevant background information on multi-
objective coordination graphs, dominance criteria in multi-objective
reinforcement learning, and generative flow models.

###  2.1 Multi-Objective Coordination Graphs

In the context of multi-objective reinforcement learning (MORL) (Hayes et al.,
2022c; Roijers et al., 2013), it is possible to utilise multi-objective
coordination graphs (MO-CoG) (Rollón, 2008) to compute a set of optimal
policies. A MO-CoG is a multi-objective extension to coordination graphs and
is a tuple $(\mathcal{D,\mathcal{A},\mathcal{U}})$.
$\mathcal{D}=\\{1,...,n\\}$ is a set of $n$ agents. $\mathcal{D}$ is
factorised into $p$, possibly overlapping, groups of agents $\mathcal{D}^{e}$.
$\mathcal{A}=\mathcal{A}_{i},...,\mathcal{A}_{n}$ is the set of joint actions.
$\mathcal{A}^{e}$ denotes the set of local joint actions for the group
$\mathcal{D}^{e}$. $\mathcal{U}={\textbf{u}^{1},...,\textbf{u}^{p}}$ is a set
of $p$, $d$-dimensional local payoff functions. The joint payoff for all
agents is the sum of local payoff functions:
$\textbf{u}(\textbf{a})=\sum_{e=1}^{p}\textbf{u}^{e}(\textbf{a}^{e})$. The
dependencies between the local reward functions and agents can be represented
by a bipartite graph with a set of nodes $\mathcal{D}$ and a set of edges,
$\mathcal{E}$. In this setting the nodes, $\mathcal{D}$, are agents and
components of a factored payoff function, and an edge
$(i,\textbf{u}^{e})\in\mathcal{E}$ exists if and only if agent $i$ influences
component $\textbf{u}^{e}$. The set of all possible join action value vectors
is denoted by the set $\mathcal{V}$.

Multi-objective variable elimination (MOVE) extends traditional variable
elimination (Koller and Friedman, 2009) to problems with multiple-objectives.
MOVE methods have been used to compute sets of optimal policies for MO-CoGs.
However, the present MOVE methods compute solution sets for expected (or
deterministic) reward vectors (Roijers et al., 2015; Rollón and Larrosa, 2006)
and do not take the underlying distributions over the rewards into account.
Therefore, they are suitable for the scalarised expected returns (SER)
criterion, but not for the expected scalarised returns (ESR) criterion.

###  2.2 Dominance Criteria in Multi-Objective Reinforcement Learning

In multi-objective reinforcement learning (MORL) (Hayes et al., 2022c), a
user’s preferences over objectives are represented by a utility function. When
applying a user’s utility function, the MORL literature generally focuses on
two optimality criteria (Roijers et al., 2013). Calculating the expected value
of the return of a policy before applying the utility function leads to the
scalarised expected returns (SER) optimisation criterion:

$V_{u}^{\pi}=u\left(\mathbb{E}\left[\sum\limits^{\infty}_{t=0}\gamma^{t}{\bf
r}_{t}\>\middle|\>\pi,\mu_{0}\right]\right).$ (1)

The SER criterion is the preferred optimality criteria is scenarios where the
user aims to maximise their utility over multiple policies executions (Hayes
et al., 2022c). The majority of MORL literature focuses on the SER criterion
(Wang and Sebag, 2012; White, 1982). Applying the utility function to the
returns and then calculating the expected value leads to the ESR optimisation
criterion:

$V_{u}^{\pi}=\mathbb{E}\left[u\left(\sum\limits^{\infty}_{t=0}\gamma^{t}{\bf
r}_{t}\right)\>\middle|\>\pi,\mu_{0}\right].$ (2)

In certain settings a user’s utility may be derived from a single execution of
a policy, therefore the expected scalarised returns criterion should be
optimised (Hayes et al., 2022d, 2021a, 2021b; Roijers et al., 2018; Malerba
and Mannion, 2021).

In many scenarios the utility function of a user is unknown a priori, and set
of optimal policies must be computed (Roijers et al., 2013). To compute a set
of optimal policies under the ESR criterion, a distribution over the returns
must be maintained (Hayes et al., 2021c, 2022d, 2022b). By utilising the
distribution over the returns a dominance criterion known as ESR dominance can
be used to determine a partial ordering over policies (Hayes et al., 2021c,
2022d). To calculate ESR dominance, the cumulative distribution function (CDF)
of the each return distribution under consideration must be calculated. For a
return distribution $\textbf{z}^{\pi}$, the CDF of $\textbf{z}^{\pi}$ is
denoted by $F_{\textbf{z}^{\pi}}$. A return distribution $\textbf{z}^{\pi}$
ESR dominates a return distribution $\textbf{z}^{\pi^{\prime}}$ if the
following is true:

$\textbf{z}^{\pi}>_{ESR}\textbf{z}^{\pi^{\prime}}\Leftrightarrow\\\
\forall\textbf{v}:F_{\textbf{z}^{\pi}}(\textbf{v})\leq
F_{\textbf{z}^{\pi^{\prime}}}(\textbf{v})\wedge\exists{\bf
v}:F_{\textbf{z}^{\pi}}(\textbf{v})<F_{\textbf{z}^{\pi^{\prime}}}(\textbf{v}).$
(3)

###  2.3 Generative Flow Models

To model return distributions where the returns are continuous we utilise
generative flow models (Kobyzev et al., 2020; Papamakarios et al., 2021) which
are a class of probabilistic generative models (Tomczak, 2022) that learn non-
linear models in continuous spaces through maximum likelihood. Generative flow
models utilise bijective functions which enable exact and tractable density
evaluation and inference on continuous data using the change of variables
formula (Kingma and Dhariwal, 2018).

Consider an observed data point $x$ generated from sampling a random variable
$X$, a latent variable $Z$ and a bijective function $f:X\rightarrow Z$ (and
thus $f^{-1}:Z\rightarrow X$ exists). Therefore, the change of variables
formula can be used to define a model distribution on the random variable $X$
as follows:

$p_{X}(x)=p_{Z}(f(x))\>\left|\textit{det}\left(\triangledown
f(x)\right)\right|,$ (4)

where $\triangledown f(x)$ is the Jacobian of $f$ at $x$. To generate samples
from the original sample space, $X$, a random sample, $z$, is drawn from the
latent space, $Z$, according to $p_{Z}$, and the inverse, $x=f^{-1}(z)$ is
computed. In this work we use a model known as _real-valued non-volume
preserving transformations_ (real-NVP) (Dinh et al., 2016) which utilises the
fact the determinant of a triangular matrix can be efficiently computed as the
product of its diagonal terms to learn a tractable and flexible model.

Dinh et al. (2016) stack a sequence of simple bijections, where each simple
bijection is updated using only part of the input vector with a function which
is simple to invert. Dinh et al. (2016) refer to each simple bijection in the
sequence as an _affine coupling layer_. Each _affine coupling layer_ takes a
$D$ dimensional input $x$ and $d$, where $d<D$ and outputs $y$. The output $y$
can be defined as follows:

$y_{1:d}=x_{1:d}$ (5)
$y_{(d+1):D}=x_{(d+1):D}\odot\exp\left(s(x_{1:d}))+t(x_{1:d}\right),$ (6)

where $s$ is a scale function, $t$ transformation function and are functions
from $\mathcal{R}^{d}\rightarrow\mathcal{R}^{D-d}$ and $\odot$ is the element-
wise product. Dinh et al. (2016) show that the Jacobian of the transformation
is triangular and therefore it is efficient to compute the determinant. A
limitation of utilising _affine coupling layers_ is during the forward
transformation some components are left unchanged. Dinh et al. (2016) overcome
this issue by combining _affine coupling layers_ in an alternating pattern,
therefore elements which are left unchanged in one transformation are modified
in the next (see supplementary information (SI) for more details).

##  3 Multi-Objective Coordination Graphs for the Expected Scalarised Returns

The current literature on multi-objective coordination graphs (MO-CoGs)
focuses exclusively on the SER criterion (Rollón and Larrosa, 2006; Roijers et
al., 2015). As shown by Hayes et al. (2022b), methods that compute solution
sets for the SER criterion cannot be used under the ESR criterion. Before we
outline our new method to compute solution sets for the ESR criterion, we must
define a set of optimal policies under the ESR criterion for MO-CoGs.

To determine a partial ordering over policies under the ESR criterion, a
dominance relation known as ESR dominance can be used (Hayes et al., 2022d).
To calculate ESR dominance a return distribution for each local payoff
function must be calculated. For MO-CoGs we define a return distribution, z,
as the distribution over the returns of a local payoff function. Therefore, we
define $\mathcal{Z}$ where $\mathcal{Z}={\textbf{z}^{1},...,\textbf{z}^{p}}$
is a set of $p$, $d$-dimensional return distributions of local payoff
functions. The joint payoff for all agents is the sum of local payoff return
distributions:
$\textbf{z}(\textbf{a})=\sum_{e=1}^{p}\textbf{z}^{e}(\textbf{a}^{e})$. The set
of all possible joint action return distributions is denoted by $\mathcal{V}$.

By utilising ESR dominance (Equation 3) we can define a set of optimal
policies under the ESR criterion for a MO-CoG as a set of ESR non-dominated
global joint actions a and associated return distributions of local payoff
functions, $\textbf{z}(\textbf{a})$, known as the ESR set:

###### Definition 1

The ESR set (ESR) of a MO-CoG, is the set of all joint actions and associated
payoff return distributions that are ESR non-dominated,

$ESR(\mathcal{V})=\left\\{\textbf{z}(\textbf{a})\in\mathcal{V}\ \middle|\
\nexists\
\textbf{z}(\textbf{a})^{{}^{\prime}}\in\mathcal{V}:\textbf{z}(\textbf{a})^{{}^{\prime}}\succ_{\text{ESR}}\textbf{z}(\textbf{a})\right\\}.$
(7)

##  4 Distributional Multi-Objective Variable Elimination

To solve multi-objective coordination graphs (MO-CoGs) for the expected
scalarised returns (ESR) criterion we define a novel _distributional multi-
objective variable elimination_ (DMOVE) algorithm. DMOVE utilises return
distributions and ESR dominance to compute the _ESR set_.

Generally multi-objective variable elimination (MOVE) methods translate the
problem to a set of value set factors (Roijers et al., 2013, 2015). Given
under the ESR criterion return distributions must be utilised, the problem
must first be translated to a set of return distribution set factors (RSFs),
$f$, where each RSF $f^{e}$ is a function mapping local joint actions to set
of payoff return distributions. On initialisation each RSF is a singleton set
containing a local actions payoff return distribution and is defined as
follows:

$f^{e}(\textbf{a}^{e})=\\{\textbf{z}^{e}(\textbf{a}^{e})\\}$ (8)

It is possible to describe the coordination graph as a bipartite graph whose
nodes, $\mathcal{D}$, are both agents and components of a factored RSF, and an
edge $(i,f^{e})\in\mathcal{E}$ if and only if agent $i$ influences component
$f^{e}$. It is important to note an agent node is joined by an edge to a
factored RSF component if the agent influences the RSF. Therefore, the
dependencies can be described by setting
$\mathcal{E}=\\{(i,f^{e})|i\in\mathcal{D}^{e}\\}$. To compute an _ESR set_ ,
DMOVE treats a MO-CoG as a series of local sub-problems. DMOVE manipulates the
set of RSFs by computing local _ESR sets_ (LESRs) when eliminating agents. To
calculate LESRs we first must define a set of neighbouring RSFs, $f_{i}$.

###### Definition 2

The set of neighbouring RSFs $f_{i}$ of agent $i$ is the subset of all RSFs
which agent $i$ influences.

Each agent $i$ has a set of neighbour agents, $n_{i}$, where each agent in
$n_{i}$ influences one or more RSFs in $f_{i}$ (see SI for a visualisation).
To compute a LESR, all return distributions for the sub-problem must first be
considered, $\mathcal{V}_{i}$, by calculating the following:

$\mathcal{V}_{i}(f_{i},\textbf{a}^{n_{i}})=\bigcup_{a^{i}}\bigoplus_{f^{e}\in
f_{i}}f^{e}(\textbf{a}^{e}),$ (9)

where $\odot$ is the cross sum of sets of return distributions. In Equation 9
all actions in $\textbf{a}^{n_{i}}$ are fixed however, not $a^{i}$, and
$\textbf{a}^{e}$ is formed from $a^{i}$ and the appropriate part of
$\textbf{a}^{n_{i}}$. Once $\mathcal{V}_{i}(f_{i},\textbf{a}^{n_{i}})$ has
been computed for agent $i$, a LESR can be calculated by applying an ESR
pruning operator. Therefore, a LESR is defined as follows:

###### Definition 3

A local ESR set, a LESR, is the ESR non-dominated subset of $\
\mathcal{V}_{i}(f_{i},\textbf{a}^{n_{i}})$:

$LESR_{i}(f_{i},\textbf{a}^{n_{i}})=ESR\ (\
\mathcal{V}_{i}(f_{i},\textbf{a}^{n_{i}})\ ),$ (10)

When calculating a LESR a new RSF, $f_{new}$, is generated which is
conditioned on the actions of the agents in $n_{i}$:

$\forall\ \textbf{a}^{n_{i}}\
f_{new}(\textbf{a}^{n_{i}})=LESR_{i}(f_{i},\textbf{a}^{n_{i}}).$ (11)

The set of RSFs, $f$, must then be updated with $f_{new}$. Therefore, we
remove the RSFs in $f_{i}$ from $f$ and update $f$ with $f_{new}$. To do so,
we define a new operator, known as the eliminate operator:

$\texttt{eliminate}(f,i)=(f\setminus
f_{i})\cup\\{f_{new}(\textbf{a}^{n_{i}})\\}.$ (12)

Computing Equation 11 and Equation 12 removes agent $i$ from the coordination
graph. Therefore, the nodes and edges of the coordination graph are updated,
where the edges for each agent in $n_{i}$ are now connected to the new RSF,
$f_{new}$.

Utilising the steps outlined above, we can now define the DMOVE algorithm
(Algorithm 1). DMOVE first translates the problem into a set of return
distribution set factors (RSFs) and removes agents in a predefined order, q.
DMOVE calls the algorithm eliminate which computes local _ESR sets_ by
pruning, and updates the set of RSFs. Once all agents have been eliminated the
final factor from the set of RSFs, $f$, is retrieved, pruned and returned. The
resulting set, $\mathcal{S}$, contains ESR non-dominated return distributions,
known as the _ESR set_ , and the associated joint actions. DMOVE only executes
a forward pass, and calculates joint actions using a tagging scheme (Roijers
et al., 2015).

1Input: $\mathcal{U}\ \leftarrow$ A set of local payoff functions;
$\textbf{q}\ \leftarrow\text{an elimination order}$

2 $f\rightarrow$ create one RSF for every local payoff function in
$\mathcal{U}$

3 while  _$\textbf{a}^{n_{i}}\in\mathcal{A}^{n_{i}}$_ do

4 $i$ $\leftarrow$ q.dequeue()

5 $f$ $\leftarrow$ $\texttt{eliminate}(f,i,\texttt{prune1, prune2})$

6 end while

7$f$ $\leftarrow$ retrieve final factor from $f$

8 $\mathcal{S}$ $\leftarrow$ prune3($f(a_{\emptyset})$)

Return $\mathcal{S}$

Algorithm 1 DMOVE ($\mathcal{U}$, prune1, prune2, prune3, q)

The eliminate algorithm (Algorithm 2) calculates the LESRs and updates the set
of RSFs, $f$. To calculate the LESRs the function CalculateLESR is utilised.
We define $\texttt{CalculateLESR}_{i}$ as follows:

$\texttt{CalculateLESR}_{i}(f_{i},\textbf{a}^{n_{i}},\texttt{prune1,prune2})=\texttt{prune2}\left(\bigcup_{a^{i}}\bigoplus^{*}_{f^{e}\in
f_{i}}f^{e}(\textbf{a}^{e})\right),$ (13)

where $\bigoplus\limits^{*}$ is the prune and cross-sum operator defined by
Roijers et al. (2015). The $\texttt{CalculateLESR}_{i}$ function prunes at two
different stages; prune1 is applied after the cross-sum has been computed and
prune2 is applied after the union over all $a^{i}$, this results in
incremental pruning (Cassandra et al., 1997).

1Input: $f\ \leftarrow$ A set of RSFs; $i\ \leftarrow$ an agent

2 $n_{i}\ \leftarrow\text{the set of neighbouring agents of $i$;}$
$f_{i}\leftarrow$ the subset of f that $i$ influences;
$f_{new}(a^{n_{i}})\leftarrow$ a new RSF

3 for  _$\textbf{a}^{n_{i}}\in\mathcal{A}^{n_{i}}$_ do

4 $f_{new}(a^{n_{i}})\leftarrow$
$\texttt{CalculateLESR}_{i}(f_{i},a^{n_{i}},\texttt{prune1,prune2})$

5 end for

6$f\leftarrow\ f\setminus\ f_{i}\cup\\{f^{new}\\}$

Return $f$

Algorithm 2 eliminate($f$, $i$, prune1, prune2)

Both DMOVE (Algorithm 1) and eliminate (Algorithm 2) are paramaterised by
pruning operators. To compute the _ESR set_ , the pruning algorithm ESRPrune
(see SI, Hayes et al. (2022b, a)) is used as each pruning operator.

The state-of-the-art MORL algorithms that compute sets of optimal solutions
for the ESR criterion focus on utilising categorical distributions to
represent return distributions (Hayes et al., 2022b, d). However, categorical
distributions are discretised and therefore cannot fully represent the
distribution of observations for continuous data. Therefore, we utilise a
generative flow model (Tomczak, 2022) known as _real non-volume preserving
transformations_ (_real-NVP, Dinh et al. (2016)_, see Section 2.3), to model
return distributions with continuous returns.

As already mentioned to compute DMOVE, the problem must first be decomposed
into a set of RSFs, $f$. To calculate $f$ a set of _real-NVP_ models,
$\mathcal{X}$ is used, where a _real-NVP_ model exists per payoff function,
$x^{e}\in\mathcal{X}$. To compute a return distribution for each local action,
$\textbf{a}^{e}$, for each payoff function each _real-NVP_ model is
conditioned on the appropriate local action $\textbf{a}^{e}$ during training.
The local action, $\textbf{a}^{e}$, is then used to condition the scale, $s$,
and transformation, $t$, functions for each individual _affine coupling layer_
as follows:

$y_{1:d}=x_{1:d}$ (14) $y_{(d+1):D}=x_{(d+1):D}\odot
exp(s(x_{1:d},\textbf{a}^{e}))+t(x_{1:d},\textbf{a}^{e}),$ (15)

where both $s$ and $t$ are represented by a neural network. Figure 1 displays
a computational graph of the forward propagation of a conditioned _real-NVP_
instance for a single _affine coupling layer_. As previously highlighted in
Section 2.3, during the forward transformation some components are left
unchanged, therefore each _real-NVP_ model must have multiple _affine coupling
layers_ to ensure each component is modified during the training phase.

$x_{1}$$=$$y_{1}$$x_{2}$$\odot$$+$$y_{2}$$\textbf{a}^{e}$$s$$t$ Figure 1:
Computational graph for the forward propagation of a _real-NVP_ instance,
conditioned on a local action, $\textbf{a}^{e}$.

To train a _real-NVP_ instance, samples from the underlying return
distributions for each payoff function must be gathered. To do so, we call an
algorithm learn which randomly executes a global joint action, a, in the
environment. Once a has been executed a set of vectorial returns r, is
received and is decomposed into local returns, $\textbf{r}^{e}$, for each
group. Each local return, $\textbf{r}^{e}$, and joint local action,
$\textbf{a}^{e}$ are then pushed to a replay buffer, where each _real-NVP_
instance has its own replay buffer. Using the data stored in the replay
buffer, it is possible to train each _real-NVP_ at different increments,
$t_{inc}$, using a number of random samples from the replay buffer.

Once each _real-NVP_ model has been trained the set of RSFs $f$ can be
calculated. To generate the required return distributions for each local
action, $\textbf{a}^{e}$, for each payoff function, $\textbf{u}^{e}$,
$\textbf{a}^{e}$ is passed as input to the corresponding _real-NVP_ instance,
$x^{e}$. A random sample is taken from the latent space of $x^{e}$ and the
inverse is then computed which generates an observation from the observed data
space. To generate a return distribution for $\textbf{a}^{e}$ this process is
repeated $n$ times. The constructed return distribution for each local action,
$\textbf{a}^{e}$, can be used to construct the singleton set outlined in
Equation 8 for each RSF in $f$.

The following algorithm learn (Algorithm 3) is defined which outlines how each
_real-NVP_ instance is trained and calls DMOVE, which returns a set of optimal
return distributions and corresponding global joint actions.

1Input: $steps$ $\leftarrow$ a number of random global joint actions to
execute; $\mathcal{U}$ $\leftarrow$ a set of payoff functions

2 $\mathcal{X}$ $\leftarrow$ $\emptyset;$ $\mathcal{F}$ $\leftarrow$
$\emptyset;$ $t_{inc}\leftarrow\text{increments for real-NVP training}$

3 for  _$u$ $\in$ $\mathcal{U}$_ do

4 $\mathcal{X}$ .append(real-NVP())

5 end for

6for  _$step\in steps$_ do

7 a $\leftarrow$ random global joint action

8 r = env.execute(a)

9 for  _$x^{e}$ $\in$ $\mathcal{X}$_ do

10 $x^{e}$.push($\textbf{r}^{e},\textbf{a}^{e}$)

11 if  _$step=t_{inc}$_ then

12 $x^{e}$.train()

13 end if

14

15 end for

16

17 end for

$\mathcal{S}$ $\leftarrow$ DMOVE()

Algorithm 3 learn()

##  5 Experiments

Generally, offshore wind farms are designed in a symmetric grid-like
structure, given this approach is advantageous for planning and construction
of the wind farms (Tao et al., 2020). Utilising a grid-like structure can
cause wake given the turbines in the farm are in close proximity (Tao et al.,
2020). To construct the dependencies required to model the problem as a MO-
CoG, we utilise a geometric approach to analyse the wake field for a given
wind direction as proposed by Verstraeten et al. (2021). To do so, a
downstream turbine, $t$, is considered to be dependent on a reference turbine,
$t_{ref}$ if and only if $t$ is geographically positioned within a specified
radius and is within a specified angle from the incoming wind vector. We
consider that downstream turbines located at an angle of $22.5^{\circ}$ and a
radius of $1$km apart from a given reference turbine lie outside the wake
field of that reference turbine (Verstraeten et al., 2021). Using the
identified dependencies it is possible to model the problem as a MO-CoG. We
deploy DMOVE to compute the _ESR set_ using several wind farms with different
numbers of turbines and different grid-like structures.

Each wind farm has specific wind conditions. For each wind farm we investigate
the incoming wind vector at $30^{\circ}$ with a wind speed of $11$ m/s
(Verstraeten et al., 2021). Under the conditions presented the wake effect is
strong and the dependencies for each farm created by the wake effect are shown
in Figure 2. For each wind farm, each turbine is placed $500$m apart along the
x-axis and $400$m apart on the y-axis (Verstraeten et al., 2021).

Figure 2: The dependency graphs for wind farms with 4, 8, 9 and 10 turbines.

To simulate each wind farm at the specified conditions, a state-of-the-art
simulator known as FLORIS is used (NREL, 2021). A reference turbine model for
the LW-8MW turbine is also used to model the turbines dynamics (Desmond et
al., 2016). These models can accurately represent the scale and conditions of
offshore wind farms.

We aim to use DMOVE to compute a set of optimal policies under the ESR
criterion where the turbulence intensity and power for each turbine is
maximised by optimising the yaw for each turbine. We choose to minimize
turbulence intensity, as it is a direct contributor to fatigue loads, and thus
to the overall lifetime of the turbine (Frandsen and Thøgersen, 1999). DMOVE
considers each turbine as an agent. Each turbine has the following possible
yaw angles, $y$:

$y=[-10,-5,\ 0,\ 5,\ 10]$ (16)

Therefore, the possible yaw angles can be considered as the action space for
each agent (Verstraeten et al., 2020). For each joint action, a, a reward r is
received where the reward can be decomposed into reward vectors for each
group, $\textbf{r}^{e}$, as follows:

$\textbf{r}^{e}=[-\text{turbulence},\ \text{power}].$ (17)

Therefore, DMOVE aims to find a set of optimal return distributions over the
turbulence and power returns, and the associated global yaw angles. All
required parameters for DMOVE are presented in SI.

$1.26$$1.28$$1.3$$1.32$$1.34$$\cdot
10^{7}$$-0.57$$-0.56$$-0.55$$-0.54$$-0.53$powerturbulence$2.4$$2.42$$2.44$$\cdot
10^{7}$$-1.35$$-1.3$power$2.72$$2.74$$2.76$$2.78$$2.8$$\cdot
10^{7}$$-1.42$$-1.4$$-1.38$$-1.36$power$3.15$$3.2$$3.25$$\cdot
10^{7}$$-1.45$$-1.4$$-1.35$power Figure 3: The expected value vector for each
policy in _ESR set_ for wind farms with 4, 8, 9 and 10 turbines.
$2.5$$3$$3.5$$4$$\cdot
10^{6}$$-0.1$$-0.098$$-0.096$powerturbulence$2$$3$$4$$\cdot
10^{6}$$-0.204$$-0.202$$-0.2$$-0.198$power$2$$3$$4$$\cdot
10^{6}$$-0.169$$-0.168$$-0.167$$-0.166$power$2$$3$$4$$\cdot
10^{6}$$-0.101$$-0.1$$-0.099$$-0.098$power Figure 4: The learned distributions
(learned by real-NVP using DMOVE, red) and the underlying distributions (blue)
of power and turbulence for each group in a 4 turbine wind farm.

Figure 3 presents the expected value vectors of ESR non-dominated return
distributions of the _ESR sets_ computed by DMOVE for wind farms with 4, 8, 9
and 10 turbines. Given the large number of policies in the _ESR set_ it is not
possible to present the distributions for each individual policy. However,
Figure 4 presents the return distributions for a policy in the _ESR set_
computed by DMOVE for each group in a 4 turbine wind farm. Figure 4 contrasts
the learned distributions (red) with the underlying return distributions
(blue). It is clear from Figure 4 the _real-NVP_ models utilised by DMOVE can
accurately learn the underlying return distributions for each specific
turbine.

The learned distributions presented in Figure 4 highlight the effectiveness
_real-NVP_ models have in learning continuous return distributions. To date
only categorical distributions have been used to compute policies for the ESR
criterion. As highlighted by Hayes et al. (2022b), categorical distributions
have limitations when dealing with continuous data. Therefore the return
distributions learned when using multi-objective categorical methods cannot
accurately represent return distributions when the returns are continuous. The
results presented in Figure 4 show that _real-NVP_ models can be used to
replace categorical distributions and learn good representations of return
distributions in real-world settings with continuous returns.

Given that wind farm control is subjected to strict regulations and
constraints, it is crucial that an operator has sufficient information about
the potential outcomes that a policy execution may have and the likelihood of
each outcome. Therefore, using the distributions presented in Figure 4 an
operator can perform a more informed analysis of potential policies. Given
that in wind farm control regulations and constraints need to be adhered to at
any moment, and thus each policy execution is important, having the outlined
distributions available can help the operator avoid selecting policies which
may have some likelihood of violating these regulations or constraints.

Given the rise in deployment of artificial intelligence systems in the real-
world, it is important such systems are explainable, trustworthy and safe
(Mannion et al., 2021). Distributional methods, like DMOVE, can be used to
provide explainability for multi-objective decision making processes. In the
context of wind farm control, risk assessment is crucial when developing
operation & maintenance strategies. Having a distribution over the returns
available can aid operators in understanding how and why an adverse event
occurred as such an event will be present in the distribution of returns.
Additionally, distributional methods can also bring outlier events to the
attention of the operator before a decision is made, which can help the
operator to take decisions to avoid unsafe events. Therefore, it may be
possible to utilise distributional multi-objective approaches for expanding
the field of trustworthy and safe artificial intelligence (Mannion et al.,
2021). As already highlighted, the state-of-the-art MORL approaches focuses on
computing policies using expected value vectors (Roijers et al., 2015; Bryce
et al., 2007). Making decisions based on expected value vectors reduces the
explainability and safety of such algorithms given the distribution over the
potential outcomes is lost when the expectation in computed. For example,
risks in wind farm control depend on short-term events that may occur (e.g.,
unexpected downtime or grid curtailment). An expected value vector may not
effectively capture uncommon events given their probability of occurring may
be small. Expected value vectors can also be difficult to interpret especially
in environments where outcomes can be stochastic, like wind farm control. In
contrast, return distributions can easily be interpreted by a system expert.
Therefore, distributional methods could be utilised as a decision making aid
in complex real world problem domains.

##  6 Related Work

Multi-objective variable elimination focuses on computing sets of optimal
policies for the SER criterion (Rollón and Larrosa, 2006; Roijers et al.,
2015; Marinescu et al., 2012). For example, Pareto multi-objective variable
elimination (PMOVE) (Rollón and Larrosa, 2006), and convex multi-objective
variable elimination (CMOVE) (Roijers et al., 2015) are used to compute the
Pareto front and convex coverage set respectively. The outlined methods all
utilise expected value vectors to determine a partial ordering over policies
and therefore cannot be used to learn optimal policies for the ESR criterion
(Hayes et al., 2022b, a).

The majority of research for the expected scalarised returns criterion largely
focuses on utilising categorical distributions to compute return
distributions. Hayes et al. (2022b) compute the _ESR set_ using multi-variate
categorical distributions in sequential decision making settings when the
utility function is unknown. Reymond et al. (2021) utilise a multi-variate
categorical distribution to learn a return distribution in MORL settings where
the utility function of a user is known. However, in this work we utilise
generative flow models, which is a class of generative modelling.

Generative modelling (Tomczak, 2022) has become a very popular area of
research where models such as general-adversarial networks (Goodfellow et al.,
2014) (GANs), variational auto-encoders (Kingma and Welling, 2013) (VAEs) and
generative flow models (Kingma and Dhariwal, 2018; Dinh et al., 2016) have
made significant advancements in artificial intelligence specifically in the
area of supervised learning. Recently, generative flow models have been used
in sequential decision making settings to model complex distributions. Ma et
al. (2020) utilise a _real-NVP_ model conditioned on a state in reinforcement
learning settings to model the complex distribution of a policy.

##  7 Conclusion & Future Work

In this work, we present a novel _distributional multi-objective variable
elimination_ (DMOVE) algorithm that can compute the _ESR set_ for real-world
problem domains where the returns are continuous. We show that DMOVE can
compute the _ESR set_ in real-world wind farm settings and also highlight how
the _real-NVP_ models can accurately learn the continuous return distributions
by interacting with the environment.

In a future publication we aim to perform a computational analysis on DMOVE to
calculate the computational implications of maintaining return distributions
to calculate policies. We also aim to utilise generative flow models in multi-
objective sequential decision making settings, like multi-objective Markov
decision processes (MOMDPs), to compute the _ESR set_ in MOMDPs with
continuous state-action space and continuous returns.

## References

  * Boersma et al. [2017] S. Boersma, B. Doekemeijer, P. M. Gebraad, P. A. Fleming, J. Annoni, A. K. Scholbrock, J. Frederik, and J.-W. van Wingerden.  A tutorial on control-oriented modeling and control of wind farms.  In _2017 American Control Conference (ACC)_ , pages 1–18. IEEE, 2017\.
  * Bryce et al. [2007] D. Bryce, W. Cushing, and S. Kambhampati.  Probabilistic planning is multi-objective.  _Arizona State University, Tech. Rep. ASU-CSE-07-006_ , 2007.
  * Cassandra et al. [1997] A. Cassandra, M. L. Littman, and N. L. Zhang.  Incremental pruning: A simple, fast, exact method for partially observable markov decision processes.  In _Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence_ , UAI’97, page 54–61, San Francisco, CA, USA, 1997. Morgan Kaufmann Publishers Inc.  ISBN 1558604855.
  * Delle Fave et al. [2011a] F. Delle Fave, R. Stranders, A. Rogers, and N. R. Jennings.  Bounded decentralised coordination over multiple objectives.  In _Proceedings of the Tenth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS’11)_ , pages 371–378, 2011a.
  * Delle Fave et al. [2011b] F. M. Delle Fave, R. Stranders, A. Rogers, and N. R. Jennings.  Bounded decentralised coordination over multiple objectives.  In _The 10th International Conference on Autonomous Agents and Multiagent Systems - Volume 1_ , AAMAS ’11, page 371–378, Richland, SC, 2011b. International Foundation for Autonomous Agents and Multiagent Systems.  ISBN 0982657153.
  * Desmond et al. [2016] C. Desmond, J. Murphy, L. Blonk, and W. Haans.  Description of an 8 mw reference wind turbine.  In _Journal of Physics: Conference Series_ , volume 753, page 092013\. IOP Publishing, 2016.
  * Dinh et al. [2016] L. Dinh, J. Sohl-Dickstein, and S. Bengio.  Density estimation using real nvp.  _arXiv preprint arXiv:1605.08803_ , 2016.
  * Durkan et al. [2019] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios.  Neural spline flows.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/7ac71d433f282034e088473244df8c02-Paper.pdf.
  * Frandsen and Thøgersen [1999] S. Frandsen and M. L. Thøgersen.  Integrated fatigue loading for wind turbines in wind farms by combining ambient turbulence and wakes.  _Wind Engineering_ , pages 327–339, 1999.
  * González-Longatt et al. [2012] F. González-Longatt, P. Wall, and V. Terzija.  Wake effect in wind farm performance: Steady-state and dynamic behavior.  _Renewable Energy_ , 39(1):329–338, 2012.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Hayes et al. [2020] C. F. Hayes, E. Howley, and P. Mannion.  Dynamic thresholded lexicograpic ordering.  In _Adaptive and Learning Agents Workshop (at AAMAS 2020)_ , May 2020\.
  * Hayes et al. [2021a] C. F. Hayes, M. Reymond, D. M. Roijers, E. Howley, and P. Mannion.  Distributional monte carlo tree search for risk-aware and multi-objective reinforcement learning.  In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , pages 1530–1532, 2021a.
  * Hayes et al. [2021b] C. F. Hayes, M. Reymond, D. M. Roijers, E. Howley, and P. Mannion.  Risk-aware and multi-objective decision making with distributional monte carlo tree search.  _In: Proceedings of the Adaptive and Learning Agents workshop at AAMAS 2021)_ , 2021b.
  * Hayes et al. [2021c] C. F. Hayes, T. Verstraeten, D. M. Roijers, E. Howley, and P. Mannion.  Dominance criteria and solution sets for the expected scalarised returns.  In _Proceedings of the Adaptive and Learning Agents workshop at AAMAS 2021_ , 2021c.
  * Hayes et al. [2022a] C. F. Hayes, D. M. Roijers, E. Howley, and P. Mannion.  Decision-theoretic planning for the expected scalarised returns.  In _Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems_ , pages 1621–1623, 2022a.
  * Hayes et al. [2022b] C. F. Hayes, D. M. Roijers, E. Howley, and M. Patrick.  Multi-objective distributional value iteration.  In _Adaptive and Learning Agents Workshop (AAMAS 2022)_ , 2022b.
  * Hayes et al. [2022c] C. F. Hayes, R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, E. Howley, A. A. Irissappane, P. Mannion, A. Nowé, G. Ramos, M. Restelli, P. Vamplew, and D. M. Roijers.  A practical guide to multi-objective reinforcement learning and planning.  _Autonomous Agents and Multi-Agent Systems_ , 36(1):26, 2022c.  ISSN 1573-7454.  doi: 10.1007/s10458-022-09552-y.  URL https://doi.org/10.1007/s10458-022-09552-y.
  * Hayes et al. [2022d] C. F. Hayes, T. Verstraeten, D. M. Roijers, E. Howley, and P. Mannion.  Expected scalarised returns dominance: A new solution concept for multi-objective decision making.  _Neural Computing & Applications_, 2022d.
  * Ho et al. [2019] J. Ho, X. Chen, A. Srinivas, Y. Duan, and P. Abbeel.  Flow++: Improving flow-based generative models with variational dequantization and architecture design.  In _International Conference on Machine Learning_ , pages 2722–2730. PMLR, 2019.
  * Kingma and Dhariwal [2018] D. P. Kingma and P. Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  _Advances in neural information processing systems_ , 31, 2018.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kobyzev et al. [2020] I. Kobyzev, S. J. Prince, and M. A. Brubaker.  Normalizing flows: An introduction and review of current methods.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(11):3964–3979, 2020.
  * Koller and Friedman [2009] D. Koller and N. Friedman.  _Probabilistic graphical models: principles and techniques_.  MIT press, 2009.
  * Ma et al. [2020] X. Ma, J. K. Gupta, and M. J. Kochenderfer.  Normalizing flow policies for multi-agent systems.  In _International Conference on Decision and Game Theory for Security_ , pages 277–296. Springer, 2020.
  * Malerba and Mannion [2021] F. Malerba and P. Mannion.  Evaluating tunable agents with non-linear utility functions under expected scalarised returns.  In _Multi-Objective Decision Making Workshop (MODeM 2021)_ , 2021\.
  * Mannion et al. [2021] P. Mannion, F. Heinz, T. G. Karimpanal, and P. Vamplew.  Multi-objective decision making for trustworthy ai.  _Multi-Objective Decision Making Workshop (MODeM 2021)_ , 2021.
  * Marinescu et al. [2012] R. Marinescu, A. Razak, and N. Wilson.  Multi-objective influence diagrams.  _arXiv preprint arXiv:1210.4911_ , 2012.
  * NREL [2021] NREL.  Floris. version 2.4, 2021.  URL https://github.com/NREL/floris.
  * Padullaparthi et al. [2022] V. R. Padullaparthi, S. Nagarathinam, A. Vasan, V. Menon, and D. Sudarsanam.  Falcon-farm level control for wind turbines using multi-agent deep reinforcement learning.  _Renewable Energy_ , 181:445–456, 2022.
  * Papamakarios et al. [2021] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan.  Normalizing flows for probabilistic modeling and inference.  _Journal of Machine Learning Research_ , 22(57):1–64, 2021.
  * Rădulescu et al. [2020] R. Rădulescu, P. Mannion, D. M. Roijers, and A. Nowé.  Multi-objective multi-agent decision making: a utility-based analysis and survey.  _Autonomous Agents and Multi-Agent Systems_ , 34(10), 2020\.
  * Reymond et al. [2021] M. Reymond, C. F. Hayes, D. M. Roijers, D. Steckelmacher, and A. Nowé.  Actor-critic multi-objective reinforcement learning for non-linear utility functions.  _Multi-Objective Decision Making Workshop (MODeM 2021)_ , 2021.
  * Reymond et al. [2022] M. Reymond, C. F. Hayes, L. Willem, R. Rădulescu, S. Abrams, D. M. Roijers, E. Howley, P. Mannion, N. Hens, A. Nowé, et al.  Exploring the pareto front of multi-objective covid-19 mitigation policies using reinforcement learning.  _arXiv preprint arXiv:2204.05027_ , 2022.
  * Roijers et al. [2013] D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley.  A survey of multi-objective sequential decision-making.  _Journal of Artificial Intelligence Research_ , 48:67–113, 2013.
  * Roijers et al. [2015] D. M. Roijers, S. Whiteson, and F. A. Oliehoek.  Computing convex coverage sets for faster multi-objective coordination.  _Journal of Artificial Intelligence Research_ , 52:399–443, 2015.
  * Roijers et al. [2018] D. M. Roijers, D. Steckelmacher, and A. Nowé.  Multi-objective reinforcement learning for the expected utility of the return.  In _Proceedings of the Adaptive and Learning Agents workshop at FAIM 2018_ , 2018.
  * Rollón [2008] E. Rollón.  _Multi-objective optimization in graphical models_.  PhD thesis, Universitat Politècnica de Catalunya, 2008.
  * Rollón and Larrosa [2006] E. Rollón and J. Larrosa.  Bucket elimination for multiobjective optimization problems.  _Journal of Heuristics_ , 12(4):307–328, 2006\.
  * Rosenthal [1977] A. Rosenthal.  Nonserial dynamic programming is optimal.  In _Proceedings of the ninth annual ACM symposium on Theory of computing_ , pages 98–105, 1977.
  * Rădulescu et al. [2019] R. Rădulescu, P. Mannion, D. Roijers, and A. Nowé.  Equilibria in multi-objective games: A utility-based perspective.  In _Adaptive and Learning Agents workshop (at AAMAS 2019)_ , May 2019\.
  * Tao et al. [2020] S. Tao, A. Feijóo, J. Zhou, and G. Zheng.  Topology design of an offshore wind farm with multiple types of wind turbines in a circular layout.  _Energies_ , 13(3):556, 2020.
  * Tomczak [2022] J. M. Tomczak.  _Deep Generative Modeling_.  Springer Nature, 2022.
  * Vamplew et al. [2021] P. Vamplew, B. J. Smith, J. Kallstrom, G. Ramos, R. Radulescu, D. M. Roijers, C. F. Hayes, F. Heintz, P. Mannion, P. J. Libin, et al.  Scalar reward is not enough: A response to silver, singh, precup and sutton (2021).  _arXiv preprint arXiv:2112.15422_ , 2021.
  * van Dijk et al. [2016] M. T. van Dijk, J.-W. Wingerden, T. Ashuri, Y. Li, and M. A. Rotea.  Yaw-misalignment and its impact on wind turbine loads and wind farm power output.  _Journal of Physics: Conference Series_ , 753(6), 2016\.
  * Verstraeten et al. [2020] T. Verstraeten, E. Bargiacchi, P. J. Libin, J. Helsen, D. M. Roijers, and A. Nowé.  Multi-agent thompson sampling for bandit applications with sparse neighbourhood structures.  _Scientific reports_ , 10(1):1–13, 2020.
  * Verstraeten et al. [2021] T. Verstraeten, P.-J. Daems, E. Bargiacchi, D. M. Roijers, P. J. Libin, and J. Helsen.  Scalable optimization for wind farm control using coordination graphs.  In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , pages 1362–1370, 2021.
  * Wagenaar et al. [2012] J. Wagenaar, L. Machielse, and J. Schepers.  Controlling wind in ECN’s scaled wind farm.  _Proc. of the Europe Premier Wind Energy Event_ , pages 685–694, 2012\.
  * Wang and Sebag [2012] W. Wang and M. Sebag.  Multi-objective Monte-Carlo tree search.  volume 25 of _Proceedings of Machine Learning Research_ , pages 507–522, Singapore Management University, Singapore, 04–06 Nov 2012. PMLR.
  * White [1982] D. White.  Multi-objective infinite-horizon discounted markov decision processes.  _Journal of mathematical analysis and applications_ , 89(2):639–647, 1982.

##  Appendix A Background

###  A.1 Real-NVP

Figure 5 presents a computational graph for a _real-NVP_ model for a single
_affine coupling layer_. However, to ensure that all components are changed
during training an alternating pattern for multiple _affine coupling layers_
must be used for each _real-NVP_ model. Figure 6 outlines an alternating
patter for multiple _affine coupling layers_ , therefore if some components
are unchanged in one layer, the components will be changed in the next layer
Dinh et al. [2016].

$x_{1}$$=$$y_{1}$$x_{2}$$\odot$$+$$y_{2}$$s$$t$ Figure 5: Computational graph
for the forward propagation of a _real-NVP_ instance. $+$ $\odot$=$+$
$\odot$=$+$ $\odot$= Figure 6: Alternating pattern for updating _affine
coupling layers_.

##  Appendix B DMOVE

###  B.1 Coordination Graph

Figure 7 outlines for agent $i$ the neighbour nodes, $n_{i}$ and the set of
RSFs, $f_{i}$, which agent $i$ influences.

$f_{i}$$f_{i}$$n_{i}$$i$ Figure 7: Coordination graph showing the graphical
relationship between agent $i$ and neighbouring RSFs $f_{i}$ and neighbouring
agents $n_{i}$.

###  B.2 ESR Prune

Algorithm 4 outlines the ESRPrune algorithm [Hayes et al., 2022a] which is
used as prune1, prune2 and prune3 in DMOVE.

1 Input: $\textbf{Z}\ \leftarrow$ A set of return distributions

2 $\textbf{Z}^{*}\ \leftarrow\emptyset$

3 while  _$\textbf{Z}\neq\emptyset$_ do

4 $\textbf{z}\leftarrow\text{the first element of $\textbf{Z}$}$

5 for  _$\textbf{z}^{\prime}\in\textbf{Z}$_ do

6 if  _$\textbf{z}^{\prime} >_{ESR}\textbf{z}$_ then

7 ${\bf{z}}\leftarrow\textbf{z}^{\prime}$

8 end if

9

10 end for

11 Remove z and all return distributions ESR-dominated by z from Z

12 Add z to $\textbf{Z}^{*}$

13 end while

Return $\textbf{Z}^{*}$

Algorithm 4 ESRPrune

##  Appendix C Experiments

###  C.1 Real-NVP

Table 1 outlines the parameters used by DMOVE, each _real-NVP_ and the learn
algorithm for each wind farm used during experimentation. The parameter
$flows$ determines the number of _affine coupling layers_ used in each _real-
NVP_ model. While the optimiser and lr parameters determine which optimiser is
used and what the learning rate is set to for each _real-NVP_ model. To
compute the scale, s, and transformation, t, functions of a _real-NVP_ model
we represent both s and t with a neural network where the hidden layers
parameter determines the number of hidden layers in the network. We note that
$r_{max}$, $r_{min}$, $n_{samples}$ and $n_{bins}$ are used to compute the
cumulative distribution function (CDF) of each return distribution. The $n$
parameter determines the number of samples which are taken from the replay
buffer during training, the size of the replay buffer size is determined by
the replay buffer size parameter. Samples from the environment are gathered by
executing a random action a number of times determined by the parameter steps.
Finally, we train each _real-NVP_ model in increments, $t_{inc}$. Therefore
training takes places every $t_{inc}$ steps.

| 4 Turbines | 8 Turbines | 9 Turbines | 10 Turbines
---|---|---|---|---
$flows$ | $8$ | $8$ | $8$ | $8$
$optimiser$ | Adam | Adam | Adam | Adam
$lr$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$ | $1e^{-3}$
$hidden$ $layers$ | $30$ | $30$ | $30$ | $30$
$\textbf{r}_{min}$ | [$-1.0,0.0$] | [$-2.0,0.0$] | [$-2.0,0.0$] | [$-2.0,0.0$]
$\textbf{r}_{max}$ | [$0.0,2.0e^{7}$] | [$0.0,3.5e^{7}$] | [$0.0,4.0e^{7}$] | [$0.0,4.2e^{7}$]
$n_{bins}$ | $2,000$ | $2,000$ | $2,000$ | $2,000$
n | $500$ | $500$ | $500$ | $500$
$n_{samples}$ | $2,000$ | $2,000$ | $2,000$ | $2,000$
$t_{inc}$ | $1,000$ | $1,000$ | $1,000$ | $1,000$
replay buffer size | $5,000,000$ | $5,000,000$ | $5,000,000$ | $5,000,000$
steps | $300,000$ | $300,000$ | $300,000$ | $300,000$
Table 1: Hyperparameters for the experiments for DMOVE and each corresponding
wind farm.

##  Appendix D Related Work on Wind Farm Control

Wind farm control research mainly focuses on the maximization of generated
power and minimization of fatigue loads (i.e., stress induced on the
mechanical components during steady-state operation) [Boersma et al., 2017].
Data-driven optimization approaches are often used to optimize control
parameters in a wake simulator [Verstraeten et al., 2021]. This is generally
achieved by reducing the wake effect. In our work, we focus on wake
redirection control, which aims at finding a joint orientation of the wind
turbines’ nacelles to redirect wake away from the wind farm [van Dijk et al.,
2016, Wagenaar et al., 2012]. State-of-the-art methods focus on performing
analysis of wake redirection strategies on small-sized wind farms. Moreover,
although both power and fatigue load are considered as optimization criteria,
a simple linear scalarization of the objectives is typically used [van Dijk et
al., 2016]. In our work, we demonstrate the benefits of using our a multi-
objective approach in the context of wake redirection control to provide a
more flexible decision framework.


!TeX document-id = 9c0ae04a-bcf3-4a29-a29d-35e056ca3658 !TEX TS-program =
pdflatexmk

††thanks: These authors contributed equally in this work.††thanks: These
authors contributed equally in this work.††thanks: corresponding author:
<EMAIL_ADDRESS>
# Complex Fermiology and Electronic Structure of Antiferromagnet EuSnP

Milo Sprague  Department of Physics, University of Central Florida, Orlando,
Florida 32816, USA  Anup Pradhan Sakhya  Department of Physics, University of
Central Florida, Orlando, Florida 32816, USA  Sabin Regmi  Department of
Physics, University of Central Florida, Orlando, Florida 32816, USA  Mazharul
Islam Mondal  Department of Physics, University of Central Florida, Orlando,
Florida 32816, USA  Iftakhar Bin Elius  Department of Physics, University of
Central Florida, Orlando, Florida 32816, USA  Nathan Valadez  Department of
Physics, University of Central Florida, Orlando, Florida 32816, USA  Kali
Booth  Department of Physics, University of Central Florida, Orlando, Florida
32816, USA  Tetiana Romanova  Institute of Low Temperature and Structure
Research, Polish Academy of Sciences, ul. Okólna 2, 50-422 Wrocław, Poland
Andrzej Ptok  Institute of Nuclear Physics, Polish Academy of Sciences, W. E.
Radzikowskiego 152, PL-31342 Kraków, Poland  Dariusz Kaczorowski  Institute of
Low Temperature and Structure Research, Polish Academy of Sciences, ul. Okólna
2, 50-422 Wrocław, Poland  Madhab Neupane  Department of Physics, University
of Central Florida, Orlando, Florida 32816, USA

###### Abstract

We studied the electronic structure of a layered antiferromagnetic metal,
EuSnP, in the paramagnetic and in the antiferromagnetic phase using angle-
resolved photoemission spectroscopy (ARPES) alongside density functional
theory (DFT) based first-principles calculations. The temperature dependence
of the magnetic susceptibility measurements exhibits an antiferromagnetic
transition at a Néel temperature of 21 K. Employing high-resolution ARPES, the
valence band structure was measured at several temperatures above and below
the Néel temperature, which produced identical spectra independent of
temperature. Through analysis of the ARPES results presented here, we
attribute the temperature-independent spectra to the weak coupling of the Sn,
and P conduction electrons with Eu 4f states.

The involvement of antiferromagnetic (AFM) orderings on the electronic band
structure of materials have seen increased interest recently [1, 2, 3, 4, 5,
6, 7, 8, 9]. Antiferromagnetism, where the net magnetization vanishes due to
compensation of magnetic moments, is capable of producing a variety of
ordering forms that modify the crystal symmetry experienced by itinerant
electrons [10]. This can include a redefining of the unit-cell periodicity,
which results in the reduction of the Brillouin zone (BZ) in reciprocal space.
Such re-scaling of the BZ induces band-folding, from which band-hybridization
may follow, dramatically reconstructing the electronic structure [1, 2, 3, 4,
7]. Another way in which AFM ordering can modify the system is through
changing the crystal space-group [11, 12]. Rotational and mirror symmetries
may be broken by the alignment of magnetic moments, which may remove
degeneracies and forced band dispersions along high symmetry directions of the
BZ [13].
EuSnP is an AFM material which crystallizes in the NbCrN-type primitive
tetragonal crystal structure (P4/nmm, No. 129) with six atoms in each unit
cell and lattice constants of a = b= 4.27 Å and c = 8.76 Å [14, 15]. It is a
layered compound with Eu-Sn-P and P-Sn-Eu linear chains along the tetragonal
[001] axis [15]. Magnetic susceptibility measurements on single crystals of
EuSnP as a function of temperature indicate a paramagnetic (PM) to AFM
transition with a Néel temperature of 21 K [14]. It is observed that the
magnetic susceptiblity along the c-axis ($\chi_{c}$) is suppressed below the
Néel temperature when compared with the magnetic susceptibility along the
a-axis ($\chi_{a}$), exhibiting an anisotropic behavior [16]. The PM Curie
temperatures estimated from $\chi_{c}^{-1}(T)$ and $\chi_{a}^{-1}(T)$ curves
respectively are 13.3 K and 18.2 K which suggests that the exchange coupling
between the Eu sublayer is ferromagnetic and these ferromagnetic sublayers
couple antiferromagnetically along the c-axis [16]. Pressure studies on EuSnP
observed no structural phase transition up to $\sim$ 6.2 GPa [17]. High-
pressure resistivity measurements suggested that the Néel temperature is
significantly enhanced under high pressure up to 2.15 GPa [17]. Most relevant
to this work, angular-dependent de-Haas van-Alphen (dHvA) measurements of the
valence electronic structure for EuSnP within the AFM phase have been
performed, which presents the low-temperatue bulk Fermi pockets [15]. The
sample conditions under dHvA measurements produce the bulk Fermi pockets of
EuSnP within the magnetically ordered phase, extending beyond the metamagnetic
transitions. However, due to the presence of both a PM-AFM phase transition at
21 K and metamagnetic transitions at 1.9 T and 6.7 T [16], field-free
temperature dependent study of the electronic structure is necessary.

In this Letter, we have performed a detailed study of EuSnP using high-
resolution angle-resolved photoemission spectroscopy (ARPES) in both the PM
and the AFM phase to understand the electronic structure of this material and
the influence of AFM ordering on the electronic structure below and above the
Néel temperature. Our temperature dependent ARPES data suggest no significant
electronic reconstruction across the AFM transition thus suggesting weak
coupling of Eu 4f electrons with Sn and P conduction electrons. Our density
functional theory (DFT)-based first principles calculations show excellent
agreement with the observed ARPES spectra.

Fig. 1: Crystal structure and DFT calculations of EuSnP. (a) Tetragonal unit
cell of EuSnP. Magenta, gray, and green colored solid spheres denote the Eu,
Sn and P atoms, respectively. (b) Temperature dependence of the inverse
magnetic susceptibility of EuSnP measured in a magnetic field of 0.1 T applied
perpendicular to the tetragonal axis. Solid straight line represents the
Curie-Weiss fit with the parameters given in the panel. Inset: low-temperature
magnetic susceptibility data collected in a small field of 10 mT. Arrow marks
the AFM transition at $T_{N}$. (c) Magnetic field variation of the
magnetization in EuSnP taken at 2 K in magnetic fields directed perpendicular
to the tetragonal axis. Arrows indicate a metamagnetic transition at
$\mu_{0}H_{m}$ and the onset of spin-polarized state above $H_{sat}$. (d)
Temperature dependence of the electrical resistivity measured within the
tetragonal plane. Solid straight line emphasizes a linear-in-T behavior in the
PM phase. The measured crystal was characterized by a large value of the
residual resistivity ratio (RRR) given in the panel. (e) Bulk Brillouin zone
reciprocal to the tetragonal lattice of EuSnP. High symmetry points and
reciprocal lattice vectors are indicated. (f) Electronic band structure
calculations along various high-symmetry directions without (yellow curves)
and with (blue curves) the inclusion of spin-orbit coupling.

High-quality single crystals of EuSnP were grown by Sn-flux method [15], and
characterized by x-ray diffraction (Oxford Diffraction X’calibur four-circle
diffractometer with a CCD Atlas detector) and energy dispersive x-ray analysis
(FEI scanning electron microscope with an EDAX Genesis XM4 spectrometer) to
have the expected crystal structure and chemical composition. Their magnetic
and electrical properties were verified by magnetic susceptibility (Quantum
Design MPMS-XL magnetometer) and electrical resistivity (Quantum Design
PPMS-14 platform) measurements. The ARPES measurements were performed at
Stanford Synchrotron Radiation Lightsource (SSRL) endstation 5-2 and the ALS
beamline 4.0.3. Measurements were carried out at temperatures of 18.5 K, 30 K,
and 50 K. The pressure in the UHV chamber was maintained better than
1$\times$10-10 torr. The angular and energy resolution were set better than
0.2∘ and 15 meV, respectively. Measurements were performed employing the
photon energies in the range of 45 eV - 60 eV. DFT calculations were performed
using the projector augmented-wave (PAW) potentials [18] implemented in the
Vienna Ab initio Simulation Package [19, 20, 21]. Calculations are made within
the generalized gradient approximation (GGA) in the Perdew, Burke, and
Ernzerhof (PBE) parameterization [22]. The energy cutoff for the plane-wave
expansion was set to $400$ eV. In calculation we used the experimental lattice
vectors as well as atomic positions [14]. The electronic band structure was
also evaluated within Quantum ESPRESSO [23, 24, 25] with PsLibrary [26]. Exact
results of the electronic band structure calculation, performed for primitive
unit cell, was used to find the tight binding model in the basis of the
maximally localized Wannier orbitals [27, 28, 29]. It was performed using the
Wannier90 software [30, 31, 32]. In our calculations, we used the $8\times
8\times 8$ full ${\bm{k}}$-point mesh. Finally, $24$-orbital tight binding
model of EuSnP, was used to investigate the surface Green’s function for semi-
infinite system [33], using WannierTools [34] software.

The crystal structure of EuSnP is shown in Fig 1(a). The compound crystallizes
in the tetragonal structure (P4/nmm, space group No. 129), which is in
agreement with previous reports [15, 14]. The primitive unit cell contains Eu
atoms at the $2c$ ($0$,$0.5$,$0.169$), Sn atoms at $4d$ ($0.5$,$0$,$0.45$),
and P atoms at $4e$ ($0$,$0.5$,$0.84$) [15] Wyckoff positions. The crystal
structure is composed of three layers which are stacked along the c-axis where
each Sn atom shares four equal bonds with other Sn atoms. The top and the
bottom layer is connected via P and Eu atoms. Along the c-direction, Eu atoms
are arranged linearly with Eu-P-Sn-Eu order. Fig. 1(b) displays the inverse
magnetic susceptibility of single-crystalline EuSnP measured in a magnetic
field oriented perpendicular to the crystallographic c-axis. Above 40 K, the
$\chi^{-1}(T)$ dependence can be described by the Curie-Weiss law with the
effective magnetic moment $\mu_{eff}$ = 7.84(2) $\mu_{B}$ and the PM Curie
temperature of $\theta_{p}$ = 18.9(4) K. The value of $\mu_{eff}$ is close to
the theoretical value g[J(J+1)] 1/2 = 7.94 (Landé factor g = 2, total angular
momentum J = 7/2) expected within Russell-Saunders coupling scheme for a
divalent Eu ion. In turn, the positive value of $\theta_{p}$ hints at
predominance of ferromagnetic exchange interactions. It is worth noting that
similar values of the Curie-Weiss parameters were found in the previous
studies on EuSnP [14, 15, 16]. As visualized in the inset to Fig. 1(b),
$\chi(T)$ measured in a small magnetic field shows a very sharp peak at
$T_{N}$ = 21(1) K that manifests the AFM phase transition. Another indication
of the AFM character of the magnetic ground state in this compound comes from
a fairly complex behavior of the magnetization isotherm taken at T = 2 K with
magnetic field confined in the tetragonal plane. In concert with the
literature data [15, 16], there occurs in $\sigma(H)$ a clear inflection near
$\mu_{0}H_{m}$ = 3.5(3) T, marking a metamagnetic-like transition, and another
distinct anomaly in a field $\mu_{0}H_{sat}$ = 6.1(1) T, above which a fully
polarized spin state is gradually approached on further increase of the field
strength (Fig. 1(c)).

The temperature dependence of the electrical resistivity of single-crystalline
EuSnP, measured with electric current flowing within the tetragonal plane, is
shown in Fig. 1(d). Worth noting is small magnitude of the resistivity in the
entire temperature range covered, yet especially the residual resistivity of
only 0.12 mWcm that gives rise to the residual resistivity ratio RRR =
$\rho(300K)$/$\rho(2K)$ = 270, which is a value much larger than those
reported before [14, 15]. Both parameters reflect very high crystalline
quality of the EuSnP sample investigated. As can be inferred from Fig. 1(d),
the onset of the AFM state manifests itself as a distinct kink in $\rho(T)$.
Below $T_{N}$, the resistivity sharply decreases due to suppression of
scattering conduction electrons on disordered magnetic moments. The most
striking feature of $\rho(T)$ of EuSnP is its perfectly linear behavior in the
entire PM state. The linear-in-T electrical transport was recognized as a
characteristic property of strange metals (cuprates, heavy-fermion systems,
etc.), in their quantum critical regimes, but also observed for some
conventional metals (e.g., Li, Mg, Ag), where scattering from phonons is
limited by the Plankian dissipation rate [35]. In order to elucidate the
actual origin of the T-linear resistivity of EuSnP, further experimental and
theoretical studies aimed at gaining detailed information about the charge
carriers in this material are compulsory.
Given the presence of superconductivity in sister compound SrSnP [36], our
finding of linear-in-T resistivity behavior, and the presence of an AFM ground
state, the electronic structure of EuSnP needs to be elucidated in order to
determine the band structure’s role in these phenomenon. To gain insight into
the electronic structure of this material, we have presented the bulk
Brillouin zone (Fig. 1(e)) and the electronic band dispersion along the
corresponding high symmetry directions, as shown in Fig. 1(f). The yellow
lines indicate the band dispersion without the inclusion of spin-orbit
coupling (SOC), while the blue lines represent the dispersion with SOC. The
calculated band structure shows that there are multiple bands which crosses
the Fermi level (EF) suggesting it to be a metal, in accordance with our
transport measurements. In the absence of SOC, we can observe multiple band
crossings along the $\Gamma$-X, M-$\Gamma$, $\Gamma$-Z-R, and A-Z direction.
With the inclusion of SOC these band crossings become gapped. One seemingly
important feature of the band structure occurs at the $\Gamma$-point, where a
hole-like curvature of the valence band is seen along the $\Gamma$-X direction
and the $\Gamma$-M direction, however, moving along $k_{z}$ in the $\Gamma$-Z
direction shows electron-like curvature. Previous calculations on a related
material, SrSnP, have attributed these features to the presence of a saddle-
point at $\Gamma$ close to the Fermi level [36]. To evaluate the validity of
this description in EuSnP, we have performed density of states calculations,
which shows no pronounced peak near the Fermi level (see the Supplemental
Material [37]). Absence of a saddle point in EuSnP could explain the lack of
low-temperature superconductivity in this material, contrasting with SrSnP in
this regard.

Fig. 2: Fermi surface and constant energy contours. (a) ARPES measured Fermi
surface (first panel) and constant energy contours measured using a photon
energy of 100 eV at various binding energies as indicated on top of each plot.
(b) Respective Fermi surface and constant energy contours obtained from DFT
calculations. The experimental data were taken at the SSRL beamline 5-2 at a
temperature of 18.5 K using LH polarization.

We have measured the Fermi surface (FS) and constant energy contours (CECs) at
various binding energies using a photon energy of 100 eV as shown in Fig. 2 in
order to disclose the electronic structure of EuSnP. Fig. 2(a) shows the
experimental FS maps and CECs, while Fig. 2(b) shows the DFT-calculated FS and
CECs. Theoretical and experimental FSs are provided with BZs that have high-
symmetry points marked on them. At the FS, several pockets are seen, pointing
to the metallic band structure of the material. The experimental FS and the
CECs of the material are quite well reproduced in the calculated plots shown
in Fig. 2(b). The Fermi surface consists of two large concentric elliptical
electron pockets surrounding the BZ corners. The four elliptical pockets
oriented with their eccentric axis along the
$\overline{\Gamma}$-$\overline{\text{M}}$ direction, which we label as
$\alpha$, coincide at the $\overline{\Gamma}$ point. The interaction between
these pockets produce a complex electronic structure which shows hole-like
behavior along the $\overline{\Gamma}$-$\overline{\text{M}}$ direction,
however which produces four electron-like pockets along the
$\overline{\Gamma}$-$\overline{\text{X}}$ direction. Increasing the binding
energy to 0.4 eV below the Fermi level shows the presence of a circular hole
pocket surrounding the $\overline{\Gamma}$ point. Increasing the binding
energy further adds another small circular pocket surrounding
$\overline{\Gamma}$.

Fig. 3: Band dispersion along the
$\overline{\text{M}}$–$\overline{\Gamma}$–$\overline{\text{M}}$ and
$\overline{\text{X}}$–$\overline{\Gamma}$–$\overline{\text{X}}$ directions.
(a, b) Experimentally measured band dispersions along the
$\overline{\text{M}}$–$\overline{\Gamma}$–$\overline{\text{M}}$ and
$\overline{\text{X}}$–$\overline{\Gamma}$–$\overline{\text{X}}$ directions
using photon energies of 45 eV, 50 eV, 55 eV, and 60 eV, respectively. The
experimental data were taken at the SSRL beamline 5-2 at a temperature of 18.5
K using LH polarization.

In order to understand the electronic structure of EuSnP at low temperatures,
we have analyzed the band dispersion along various high-symmetry directions.
Fig. 3(a) presents ARPES-obtained band dispersions along the
$\overline{\text{M}}$-$\overline{\Gamma}$-$\overline{\text{M}}$ direction for
photon energies ranging from 45 eV to 60 eV. A sharp steeply dispersing band
is observed, crossing the Fermi level at about 0.5 $\text{\AA}^{-1}$, which is
attributed to the co-vertex of the $\beta$ elliptical pocket. This band
appears as a sharp, intense spectral feature, which does not appear to
disperse over various photon energies (Fig. 3(a)). A second feature observed
near the Fermi level is a hole-like band with relatively broad spectral
features. Comparing with the Fermi surface, we attribute this hole pocket
around $\overline{\Gamma}$ to the vertex of ellipse $\alpha$. This pocket
disperses strongly as a function of photon energy. Another pocket below 0.2 eV
at the $\overline{\Gamma}$ point strongly disperses with photon energy
indicating it to be of bulk 3D origin. Next, we discuss the band dispersion
measured along the
$\overline{\text{X}}$-$\overline{\Gamma}$-$\overline{\text{X}}$ direction. The
measured band dispersion with a photon energy of 45 eV suggests that there is
no pocket at the $\overline{\Gamma}$ point. However an electron-like pocket
exists along the $\overline{\Gamma}$-$\overline{\text{X}}$ direction. This
pocket is highly dispersive as the photon energy is changed from 45 eV to 60
eV, suggesting its bulk origin. The pocket below 0.2 eV is strongly photon
energy dependent indicating its bulk nature.

Fig. 4: Temperature dependent band dispersion of EuSnP along various high-
symmetry directions. (a, b) Experimentally measured band dispersions along the
$\overline{\text{X}}$–$\overline{\Gamma}$–$\overline{\text{X}}$, (c, d)
$\overline{\text{M}}$–$\overline{\Gamma}$–$\overline{\text{M}}$, and (e,f)
$\overline{\text{M}}$–$\overline{\text{X}}$–$\overline{\text{M}}$ directions
at 50 K and 18.5 K, respectively. (g-i) Momentum distribution curves (MDCs) at
EF as a function of temperature. The high-symmetry directions are indicated in
the captions. Comparison of the Fermi surface measured at (j) 50 K, (k) 30 K,
and (l) 18.5 K, respectively. The experimental data were taken at the SSRL
beamline 5-2 using LH polarization.

To understand the effect of antiferromagnetism on the electronic band
structure of EuSnP we have performed temperature dependent ARPES measurements
along various high-symmetry directions as shown in Fig. 4. The measurements
have been performed at 50 K, 30 K, and 18.5 K which is above and below the
Néel temperature. Fig. 4(a-f) presents the ARPES measured band dispersion
along the $\overline{\text{X}}$-$\overline{\Gamma}$-$\overline{\text{X}}$,
$\overline{\text{M}}$-$\overline{\Gamma}$-$\overline{\text{M}}$, and
$\overline{\text{M}}$-$\overline{\text{X}}$-$\overline{\text{M}}$ directions
at temperatures of 50 K and 18.5 K, respectively. The FS measured above and
below the Néel temperature is presented in Fig. 4(j-l). As can be seen on
comparing Fig. 4(a,b), Fig. 4(c,d), and Fig. 4(e,f), we do not observe any
significant change in the electronic structure across the magnetic transition
above and below the Néel temperature (see the Supplemental Material for
further temperature dependent measurements [37]). In order to quantitavely
analyze any change in the electronic structure as a function of temperature we
have performed a close inspection of the momentum distribution curves (MDCs)
at the Fermi level along the several high-symmetry directions, as shown in
Fig. 4(g-i) which reveals the ARPES spectra does not demonstrate significant
modification across the magnetic transition.

Moving on to the interpretation of these results; we first note that the
coupling between the magnetic f electrons and conduction electrons define the
modification to the electronic structure across the magnetic phase transition.
Our present temperature dependent ARPES results suggest that f electrons do
not strongly interact with the valence bands and thus does not have any
significant effect on the electronic structure of this material [1, 38]. As
discussed in ref [1], one may justify several reasons for the lack of observed
AFM reconstruction via ARPES. Firstly, the AFM ordering could conceivably be a
strong bulk effect and may not extend to the surface. We note that EuSnP is
revealed to show good agreement between the AFM-phase dHvA measurements [15]
and the obtained ARPES results, indicating that both the bulk and surface
electronic structures see no reconstruction. The more likely scenario is where
there is very weak coupling between the Eu moments and the Sn-P conduction
electrons. This is understood to indicate that, while the AFM ordering does
redefine the scale of the unit cell, the spectral weight associated with the
AFM band folding is directly related to the coupling strength of the itinerant
electrons to the Eu moments [39]. Given the overall consistency between the
previously reported bulk-sensitive dHvA measurements and the surface-sensitive
ARPES measurements both above and below TN, we associate the lack of observed
band-folding, band-splitting, or band-shifts as an indication of the weak
interaction strength between Eu 4$f$ electrons and the valence electronic
states.

In conclusion, we have successfully grew high-quality single crystals of
EuSnP, which possess a tetragonal space group of P4/nmm. Our investigations
revealed an antiferromagnetic (AFM) transition in this material, with a Néel
temperature of 21 K, as evidenced by magnetic measurements. Resistivity
measurements indicated EuSnP to be a good electrical conductor, showing a
striking linear temperature dependence of the resistivity in the PM phase. In
order to explore the electronic structure of this compound, we employed ARPES
(angle-resolved photoemission spectroscopy). Our ARPES measurements unveiled
significant elliptical pockets centered around the $\overline{\text{M}}$
point, corresponding to the conduction electrons. Additionally, we observed
small bulk electron pockets along the
$\overline{\Gamma}$–$\overline{\text{X}}$ direction, as well as small hole-
pockets along the $\overline{\Gamma}$–$\overline{\text{M}}$ direction.
Importantly, our experimental results were well reproduced by DFT-based first
principles calculations. Surprisingly, upon cooling the sample below the Néel
temperature, we observed no electronic reconstruction. This suggests that the
Sn and P electrons weakly couple to the magnetic order of the Eu spins. These
techniques will shed light on the interplay between the Sn and P conduction
electrons and the Eu spins. Our work not only provides a comprehensive
framework for understanding the electronic structure of this new material both
above and below the Néel temperature but also serves as a valuable resource
for future researchers interested in exploring the effects of
antiferromagnetism in EuSnP.

M.N. is supported by the Air Force Office of Scientific Research MURI (Grant
No. FA9550-20-1-0322), and the National Science Foundation (NSF) CAREER award
DMR-1847962. D.K. and T.R. were supported by the National Science Centre (NCN,
Poland) under research grant 2021/41/B/ST3/01141. A.P. acknowledges the
support by National Science Centre (NCN, Poland) under Projects No.
2021/43/B/ST3/02166. The use of Stanford Synchrotron Radiation Lightsource
(SSRL) in SLAC National Accelerator Laboratory is supported by the U.S.
Department of Energy, Office of Science, Office of Basic Energy Sciences under
Contract No. DE-AC02-76SF00515. We thank Makoto Hashimoto and Donghui Lu for
the beamline assistance at SSRL endstation 5-2. This research also used
resources of the ALS at the Lawrence Berkeley National Laboratory, a US
Department of Energy Office of Science User Facility, under Contract No. DE-
AC02-05CH11231. We thank Jonathan Denlinger for beamline assistance at the ALS
Beamline 4.0.3.

## References

  * [1] J. A. Sobota, K. Kim, H. Takatsu, M. Hashimoto, S.-K. Mo, Z. Hussain, T. Oguchi, T. Shishidou, Y. Maeno, B. I. Min, and Z.-X. Shen, Electronic structure of the metallic antiferromagnet PdCrO2 measured by angle-resolved photoemission spectroscopy, Phys. Rev. B 88, 125109 (2013).
  * [2] N. Hao, Y. Wang and J. Hu, Oriented gap opening in the magnetically ordered state of iron pnictides: An impact of intrinsic unit cell doubling on the Fe square lattice by As atoms, Europhys. Lett. 104.5, 57007 (2013).
  * [3] H.-J. Noh, J. Jeong, B. Chang, D. Jeong, H. S. Moon, E.-J. Cho, J. M. Ok, J. S. Kim, K. Kim, B. I. Min, H.-K. Lee, J.-Y. Kim, B.-G. Park, H.-D. Kim, and S. Lee, Direct Observation of Localized Spin Antiferromagnetic Transition in PdCrO2 by Angle-Resolved Photoemission Spectroscopy, Sci. Rep. 4, 3680 (2014).
  * [4] K. Kuroda, Y. Arai, N. Rezaei, S. Kunisada, S. Sakuragi, M. Alaei, Y. Kinoshita, C. Bareille, R. Noguchi, M. Nakayama, S. Akebi, M. Sakano, K. Kawaguchi, M. Arita, S. Ideta, K. Tanaka, H. Kitazawa, K. Okazaki, M. Tokunaga, Y. Haga, S. Shin, H. S. Suzuki, R. Arita, and Takeshi Kondo, Devil’s staircase transition of the electronic structures in CeSb, Nat. Commun. 11, 2888 (2020).
  * [5] A. P. Sakhya, S. Kumar, A. Pramanik, R. P. Pandeya, R. Verma, B. Singh, S. Datta, S. Sasmal, R. Mondal, E. F. Schwier, K. Shimada, A. Thamizhavel, and K. Maiti, Behavior of gapped and ungapped Dirac cones in the antiferromagnetic topological metal SmBi, Phys. Rev. B 106, 085132 (2022).
  * [6] B. Schrunk, Y. Kushnirenko, B. Kuthanazhi, J. Ahn, L.-L. Wang, E. O’Leary, K. Lee, A. Eaton, A. Fedorov, R. Lou, V. Voroshnin, O. J. Clark, J. Sánchez-Barriga, S. L. Bud’ko, R.-J. Slager, P. C. Canfield, and A. Kaminski, Emergence of Fermi arcs due to magnetic splitting in an antiferromagnet, Nature 603, 610 (2022).
  * [7] A. P. Sakhya, B. Wang, F. Kabir, C. Y. Huang, M. M. Hosen, B. Singh, S. Regmi, G. Dhakal, K. Dimitri, M. Sprague, R. Smith, E. D. Bauer, F. Ronning, A. Bansil, and M. Neupane, Complex electronic structure evolution of NdSb across the magnetic transition, Phys. Rev. B 106, 235119 (2022).
  * [8] Y. Kushnirenko, B. Schrunk, B. Kuthanazhi, L.-L. Wang, J. Ahn, E. O’Leary, A. Eaton, S. L. Bud’ko, R.-J. Slager, P. C. Canfield, and A. Kaminski, Rare-earth monopnictides: Family of antiferromagnets hosting magnetic Fermi arcs, Phys. Rev. B 106, 115112 (2022).
  * [9] S. Regmi, M. M. Hosen, B. Ghosh, B. Singh, G. Dhakal, C. Sims, B. Wang, F. Kabir, K. Dimitri, Y. Liu, A. Agarwal, H. Lin, D. Kaczorowski, A. Bansil, and M. Neupane, Temperature-dependent electronic structure in a higher-order topological insulator candidate EuIn2As2, Phys. Rev. B 102.16 (2020): 165153.
  * [10] M. S. Dresselhaus, G. Dresselhaus, and A. Jorio. Group theory: application to the physics of condensed matter. Springer Science & Business Media, 2007.
  * [11] A. P. Cracknell, Magnetism in Crystalline Materials , Oxford: Pergamon (1975)
  * [12] G. Burns and A. M. Glazer, Space Groups for Solid State Scientists, New York: Academic (2013)
  * [13] Birss R R 1964 Symmetry and Magnetism (Amsterdam: North-Holland)
  * [14] A. C. Payne, A. E. Sprauve, A. P. Holm, M. M. Olmstead, S. M. Kauzlarich, and P. Klavins, EuSnP: a novel antiferromagnet with two-dimensional, corrugated Sn sheets, J. Alloys Compd. 338, 229 (2002).
  * [15] W. Iha et al., in Proceedings of J-Physics 2019: International Conference on Multipole Physics and Related Phenomena (2020), p. 012002.
  * [16] T. Fujiwara, Y. Saiga, T. J. Sato, M. Hedo, and Y. Uwatoko, Magnetic properties of EuSnP Physica B Condens. Matter 378-380, 1122 (2006).
  * [17] X. Gui, G. J. Finkelstein, D. E. Graf, K. Wei, D. Zhang, R. E. Baumbach, P. Dera, and W. Xie, Enhanced Néel temperature in EuSnP under pressure Dalton Trans 48, 5327 (2019).
  * [18] P. E. Blöchl, Projector augmented-wave method, Phys. Rev. B 50, 17953-17979 (1994).
  * [19] G. Kresse & J. Hafner, Ab initio molecular-dynamics simulation of the liquid-metal–amorphous-semiconductor transition in germanium Phys. Rev. B 49, 14251-14269 (1994).
  * [20] G. Kresse, & J. Furthmüller, Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set Phys. Rev. B 54, 11169-11186 (1996).
  * [21] G. Kresse, & D. Joubert, From ultrasoft pseudopotentials to the projector augmented-wave method Phys. Rev. B 59, 1758-1775 (1999).
  * [22] J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized Gradient Approximation Made Simple, Phys. Rev. Lett. 77, 3865-3868 (1996).
  * [23] P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials, J. Phys. Condens. Matter. 21, 395502 (2009).
  * [24] P Giannozzi, O Andreussi, T Brumme, O Bunau, M Buongiorno Nardelli, M Calandra, R Car, C Cavazzoni, D Ceresoli, M Cococcioni, N Colonna, I Carnimeo, A Dal Corso, S de Gironcoli, P Delugas, R A DiStasio Jr, A Ferretti, A Floris, G Fratesi, G Fugallo, R Gebauer, U Gerstmann, F Giustino, T Gorni, J Jia, M Kawamura, H-Y Ko, A Kokalj, E Küçükbenli, M Lazzeri, M Marsili, N Marzari, F Mauri, N L Nguyen, H-V Nguyen, A Otero-de-la-Roza, L Paulatto, S Poncé, D Rocca, R Sabatini, B Santra, M Schlipf, A P Seitsonen, A Smogunov, I Timrov, T Thonhauser, P Umari, N Vast, X Wu and S Baroni, Advanced capabilities for materials modelling with Quantum ESPRESSO J. Phys. Condens. Matter. 29, 465901 (2017).
  * [25] P. Giannozzi, O. Baseggio, P. Bonfà, D. Brunato; R. Car, I. Carnimeo; C. Cavazzoni, S. Gironcoli, P. Delugas, F. F. Ruffino, A. Ferretti, N. Marzari, I. Timrov, A. Urru, S. Baroni, Quantum ESPRESSO toward the exascale J. Chem. Phys. 152, 154105 (2020).
  * [26] A. Dal Corso, Pseudopotentials periodic table: From H to Pu, Comput. Mater. Sci. 95, 337-350 (2014).
  * [27] N. Marzari, A. A. Mostofi, J. R. Yates, I. Souza, & D. Vanderbilt, Maximally localized Wannier functions: Theory and applications, Rev. Mod. Phys. 84, 1419-1475 (2012).
  * [28] N. Marzari & D. Vanderbilt, Maximally localized generalized Wannier functions for composite energy bands, Phys. Rev. B 56, 12847-12865 (1997).
  * [29] I. Souza, N. Marzari, & D. Vanderbilt, Maximally localized Wannier functions for entangled energy bands, Phys. Rev. B 65, 035109 (2001).
  * [30] A. A. Mostofi, J. R. Yates, Y. Lee, I. Souza, D. Vanderbilt, N. Marzari, wannier90: A tool for obtaining maximally-localised Wannier functions, Comput. Phys. Commun. 178, 685-699 (2008).
  * [31] A. A. Mostofi, J. R. Yates, G. Pizzi, Y. Lee, I. Souza, D. Vanderbilt, N. Marzari, An updated version of wannier90: A tool for obtaining maximally-localised Wannier functions, Comput. Phys. Commun. 185, 2309-2310 (2014).
  * [32] G. Pizzi, V. Vitale, R. Arita, S. Blüge, F. Freimuth, G. Géranton, M. Gibertini, D. Gresch, C. Johnson, T. Koretsune, J. Ibañez-Azpiroz, H. Lee, J. Lihm, D. Marchand, A. Marrazzo, Y. Mokrousov, J. I. Mustafa, Y. Nohara, Y. Nomura, L. Paulatto, S. Poncé, T. Ponweiser, J. Qiao, F. Thöle, S. S Tsirkin, M. Wierzbowska, N. Marzari, D. Vanderbilt, I. Souza, A. A. Mostofi, and J. R. Yates, Wannier90 as a community code: new features and applications, J. Phys. Condens. Matter. 32, 165902 (2020).
  * [33] M. L. Sancho, J. L. Sancho, J. L. Sancho & J. Rubio, Highly convergent schemes for the calculation of bulk and surface Green functions, Journal of Physics F: Metal Physics 15, 851 (1985).
  * [34] Q. Wu, S. Zhang, H. F. Song, M. Troyer, & A. A. Soluyanov, WannierTools: An open-source software package for novel topological materials, Comput. Phys. Commun. 224, 405-416 (2018).
  * [35] S. A. Hartnoll, and A. P. Mackenzie, Colloquium: Planckian dissipation in metals, Rev. Mod. Phys. 94.4 (2022): 041002.
  * [36] X. Gui, Z. Sobczak, T. R. Chang, X. Xu, A. Huang, S. Jia, H.-T. Jeng, T. Klimczuk, and W. Xie, Superconducting SrSnP with Strong Sn–P Antibonding Interaction: Is the Sn Atom Single or Mixed Valent? Chem. Mater. 30(17), 6005-6013.
  * [37] For additional information, see the supplemental material to this manuscript.
  * [38] Z. X. Yin, X. Du, S. Zhang, C. Chen, D. Pei, J. S. Zhou, X. Gu, R. Z. Xu, Q. Q. Zhang, W. X. Zhao, Y. D. Li, Y. F. Xu, A. Bernevig, Z. K. Liu, E. K. Liu, Y. L. Chen, and L. X. Yang, Electronic structure of antiferromagnetic Dirac semimetal candidate GdIn3, Phys. Rev. Mater. 6, 084203 (2022).
  * [39] J. Voit, L. Perfetti, H. Berger, G. Margaritondo, G. Grüner, H. Höchst, and M. Grioni, Electronic Structure of Solids with Competing Periodic Potentials, Science 290.5491 (2000).