# Procedure Planning in Instructional Videos
via Contextual Modeling and Model-based Policy Learning

Jing Bi Jiebo Luo Chenliang Xu
University of Rochester
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Learning new skills by observing humans’ behaviors is an essential capability
of AI. In this work, we leverage instructional videos to study humans’
decision-making processes, focusing on learning a model to plan goal-directed
actions in real-life videos. In contrast to conventional action recognition,
goal-directed actions are based on expectations of their outcomes requiring
causal knowledge of potential consequences of actions. Thus, integrating the
environment structure with goals is critical for solving this task. Previous
works learn a single world model will fail to distinguish various tasks,
resulting in an ambiguous latent space; planning through it will gradually
neglect the desired outcomes since the global information of the future goal
degrades quickly as the procedure evolves. We address these limitations with a
new formulation of procedure planning and propose novel algorithms to model
human behaviors through Bayesian Inference and model-based Imitation Learning.
Experiments conducted on real-world instructional videos show that our method
can achieve state-of-the-art performance in reaching the indicated goals.
Furthermore, the learned contextual information presents interesting features
for planning in a latent space.

##  1 Introduction

Humans can learn new skills by watching demo videos. Although this seems
natural to human, it is challenging for AI. We have seen rich works on
modeling human behaviors from videos with the majority focusing on recognizing
actions [21, 18, 26]. However, solely perceiving what actions are performed
without modeling the underlying decision-making process is insufficient for AI
to learn new skills. The next-generation AI needs to figure out what actions
are necessary to achieve the desired goals [5] with the consideration of
actions’ potential consequences. In this paper, we focus on learning the goal-
directed actions from instructional videos. Recently, Chang  _et al_. [6]
proposed a new problem known as procedure planning in instructional videos. It
requires a model to 1) plan a sequence of verb-argument actions and 2)
retrieve the intermediate steps for achieving a given visual goal in real-life
tasks such as making a strawberry cake (see Fig. 1).

Figure 1: Overview of our proposed method. Given a starting observation (top-
left image) and a desired visual goal (bottom-left image), we extract the
contextual information of the planning trajectory upon which the Generation
Model outputs a sequence of actions. The model is responsible for learning
plannable latent representations with a focus on procedures and action
consequences. Thus, we can retrieve images of intermediate steps (top-right
images).

This task is different from the typical image-language translation problem in
the way that certain actions can be exchanged to achieve the same goal
(_e.g_., the order of adding salt and sugar usually does not matter), making
it difficult to predict the same action sequence as ground-truth using
sequence mapping.

Moreover, sequence-to-sequence based structure, suitable for modeling events
that tend to occur in sequence with high probability, is thought to involve no
consideration of the likely outcome [9]. Therefore, we formalize this task as
a planning problem with focus on two different sequential patterns that can be
easily observed in Fig. 4: In the context of making a cake, mixing ingredients
and washing cherries are interchangeable, _i.e_., short-term action
separation, but both should be ahead of the action putting cherries on the
top, _i.e_., long-term action association.

Figure 2: Procedure planning example. Given a starting observation (picture of
food ingredients) and a visual goal (picture of a made cake), the model needs
to learn how to complete real-world tasks such as making a cake by planning a
sequence of actions $a_{\scriptscriptstyle{1:T}}$ (blue circles) and
retrieving the intermediate observations $o_{\scriptscriptstyle{2:T-1}}$
(yellow circles).

Inspired by Raab _et al_. [9], we think that when performing goal-directed
tasks, it is beneficial to consider both the task contextual information and
the potential action consequences. Contextual information here refers to the
time-invariant knowledge (not changed during planning) that distinguishes a
particular task from the others. For example, if we know the goal is to make a
cake as shown in Fig. 1, it is less likely to plan an action like putting it
on the grill.

Therefore, we model the dependency between the actions and different goals as
the long-term action association in a Bayesian framework. As we show later in
the experiments, this serves a few purposes:  a) it provides a more structured
representation for the subsequent policy learning;  b) we can sample from the
posterior distribution for more diverse trajectories to facilitate the action
exploration; and  c) compared with the noisy pixel space, feature distances in
the learned latent space are more meaningful.  To achieve short-term action
separation, we model the action sequence as a Markov Decision Process (MDP) as
shown in Fig. 2, where the future action depends only upon the present state.
Besides, because goal-directed actions are often selected based on
expectations of their consequent outcomes [7], we propose to incorporate a
transition model into the Imitation Learning (IL) framework [16, 19] so that
we can explicitly model the environment jointly with policy learning.

This approach brings following advantages:  a) it helps policy to ﬂexibly
pursue a goal by leveraging causal knowledge of the actions’ potential
consequences;  b) when model-based simulations produce states with alternative
actions, the discrimination and selection between actions allow an agent to
find the currently most desired outcome [24, 34]; and  c) it bypasses the need
of an interactive environment that is required by classic planning algorithms
[31, 30], making it suitable for modelling the web videos.

We demonstrate the effectiveness of our approach by evaluating it on a real-
world instructional video dataset [43] (an example is shown in Fig. 2). The
results on the procedure planning task show that our learned model can uncover
the underlying human decision-making processes. Furthermore, the results on
the challenging walk-through planning task [22] confirm that our model learns
meaningful representations of the environment dynamics, which is crucial for
efficient plannings in the latent space. Finally, the visualization of
contextual information indicates that our proposed encoder structure can learn
a concise representation to capture distinct knowledge of different real-world
tasks. The main contributions of our work are summarized as follows:  a) we
propose a novel method to address the procedure planning problem, which
combines Bayesian Inference with Model-based Imitation Learning;  b) we
propose a neural network structure based on variational inference that learns
to embed sufficient information to convey the desired task, incorporating the
visual observations’ uncertainty; and  c) we propose two model-based IL
algorithms that explicitly learn the environment dynamics (in either a
stochastic or deterministic way) and integrate with the transition model to
simultaneously learn a plannable latent representation for accurate planning.

##  2 Related Work

Vision-based Human Behavior Understanding. Our planning tasks are highly
related to a popular AI research area: building a machine that can accurately
understand humans’ actions and intentions. The intention can be seen as the
sequence of actions needed to be taken to achieve an objective [5]. To
understand human attention, Zhang _et al_. [41] proposed the Deep Future Gaze
model to predict the gaze location in multiple future frames conditioned on
the current frame. Forthermore, Wei _et al_. [38] utilized a hierarchical
graph that jointly models attention of the gaze and intention of the
performing task from a RGB-D video. Rhinehart _et al_. [30] proposed an online
inverse reinforcement learning method to discover rewards for modeling and
forecasting first-person camera wearer’s long-term goals, together with
locations and transitions from streaming data. Merel _et al_. [27] extended
the Generative Adversarial Imitation Learning (GAIL) [16] framework to learn
human-like movement patterns from demonstrations consisting of only partial
observations. Unlike these previous works that predict the future, we try to
understand human behaviors by learning their goal-direction actions.

Deep Reinforcement Learning. Reinforcement learning (RL) is often employed to
learn and infer the MDP model simultaneously, which is a natural way to
understand how humans learn to optimize their behaviors in an environment
[35]. Recently, combined with deep learning, DRL is leveraged to solve several
vision problems such as Visual Tracking [33], Video Summarization [42],
Stroke-based rendering [18], and Vision-based Navigation [28]. For semantic-
level video understanding, DRL can also play an important role. For instance,
it is utilized for Activity Localization [36], Natural Language Grounding
[13], and Video Description [37]. However, these works often require expert
knowledge to design a useful reward function, whose goal is to learn a
behavior that maximizes the expected reward. In contrast, we work on the IL
problem without explicit usage of a hand-craft reward. Our work is most
closely related to inverse RL [30, 1] and contextual RL[39]. However, the
major difference is that we focus on learning from collected dataset which is
crucial for applications when online interaction is not permitted, _e.g_.,
safety-critical situation.

Planning in Latent Space. Planning is a natural and powerful approach to
decision-making with known dynamics, such as game playing and simulated robot
control. To plan in unknown environments, the agent needs to learn the
environment dynamics from previous experiences. Recent model-based RL schemes
have shown promise that deep networks can learn a transition model directly
from low-dimensional observations and plan with the learned model [40, 6, 11].
A closely related method is Universal Planning Networks (UPN) [32] that learns
a plannable latent space with gradient descent by minimizing an imitation
loss, i.e., learned from an expert planner. Plannable means the learned
representations are structured to perform a classic planning algorithm [22].
Our method further incorporate the contextual knowledge of assigned task to
the latent space and remove the assumption of differentiable action space.
Another line of work is causal InfoGAN [22], which tries to capture the
relations between two sequential images and models the causality of the
simulation environment in an unsupervised learning manner. Similarly, our Ext-
MGAIL model also focus on the stochastic transition model. However, making
predictions in raw sensory space is unnecessarily hard [10], we predict low
dimensional latent representations for future state and plan upon it.

##  3 Methods

We consider a set-up similar to Chang _et al_. [6]: we have access to $K$
trajectories
$\\{(o_{\scriptscriptstyle{1:T}}^{j},a_{\scriptscriptstyle{1:T}}^{j})\\}_{j=0}^{\scriptscriptstyle
K}{\sim}\pi_{E}$ collected by an expert trying to achieve different tasks.
Given a starting visual observation $o_{1}$ and a visual goal
$o_{\scriptscriptstyle T}$ that indicates for a particular task, we want to
learn a plannable representation upon which goal-directed actions are planned
to perform two complex planning tasks (Fig. 2):  a) procedure planning:
generate a valid sequence of actions $a_{\scriptscriptstyle{1:T}}$ to achieve
the indicated goal; and  b) walk-through planning: retrieve the intermediate
observations $o_{\scriptscriptstyle{2:T-1}}$ between the starting
$o_{\scriptscriptstyle{1}}$ and the goal.  Our key insight is that by
decomposing the procedure planning problem in Eq. 1 into two sub-problems, we
can decouple representation learning into two parts:  a) inferring the time-
invariant contextual information that conveys the task to achieve; and  b)
learning the time-varying plannable representations related to the decision-
making process and environment dynamics.  In this way, both representations
can be further used to retrieve $o_{\scriptscriptstyle{2:T-1}}$ for solving
the walk-through planning.

As shown in the overall architecture Fig. 3, we assume that the contextual
information contains all the details an agent need for achieving the desired
goal. Hence, we formulate the procedure planning problem
$p(a_{\scriptscriptstyle{1:T}}|o_{\scriptscriptstyle{1}},o_{\scriptscriptstyle{T}})$
as:

$p(a_{\scriptscriptstyle{1:T}}|o_{\scriptscriptstyle{1}},o_{\scriptscriptstyle{T}})=\iint
p(a_{\scriptscriptstyle{1:T}},s_{\scriptscriptstyle{1:T}}|z_{c})p(z_{c}|o_{\scriptscriptstyle
1},o_{\scriptscriptstyle
T})\mathrm{d}s_{\scriptscriptstyle{1:T}}\mathrm{d}z_{c},$ (1)

where we donate $z_{c}$ as the context variable that conveys the desired task,
$p(z_{c}|o_{1},o_{T})$ as the inference model for modeling posterior
distribution over the context variable and given observations and
$p(a_{\scriptscriptstyle{1:T}},s_{\scriptscriptstyle{1:T}}|z_{c})$ as the
generation model that plans a sequence of actions and hidden states that
transfer the initial state to the desired outcome. In the following sections,
we will first discuss how to infer the contextual information. We will then
solve the second sub-problem by imitating human behaviors with consideration
of offline policy evaluation [23], and utilize Hindsight Experience Replay
(HER) [3] to better leverage the expert demonstrations. Lastly, we will
discuss how to solve the walk-through planning problem with learned model.

Figure 3: The overall architecture: given the initial and the goal
observations, two parallel encoders will parameterize the mean and log-
variance of the Gaussian distribution. The context variable will then be
sampled from this distribution and fed into the generation model to roll out a
trajectory. We use a discriminator that tries to distinguish the state-action
pairs from the expert or the learned policy, which serves as the local reward
function.

###  3.1 Inference Model

As visualized in Fig. 3 the action $a_{t}$ at time-step $t$ is solely governed
by the current state $s_{t}$ that contains the information of current
observation and the information regarding the desired goal. We want $z_{c}$ to
represent the contextual information for achieving the goal, which should be
time-invariant, and the hidden state $s_{t}$ to contain the time-varying
information for the decision-making process. To achieve this separation, the
hidden states are only allowed to condition on $z_{c}$; thus, all the
information about the goal must pass through $z_{c}$ to avoid a shortcut from
the observations to actions. In this way, $s_{t}$ will be the only time-
dependent hidden variable used to recover the actions, and we reserve $z_{c}$
for compressing everything else. However, the true posterior distribution
$p_{\phi}(z_{c}|o_{1},o_{\scriptscriptstyle T})$ from video frames is
analytically intractable; thus, we use variational inference to approximate
posterior distribution from given observations. Note here we use the raw pixel
observations $o_{1}$ and $o_{\scriptscriptstyle T}$ for planning, which is
different from compared methods which use pre-computed visual features. The
model involves an encoder (shown in Fig. 3) that models the approximation
distribution $q_{\varphi}(z_{c}|o_{1},o_{t})$ and a decoder (omitted from Fig.
3 for simplicity) that models the prior $p_{\phi}(o_{1},o_{t}|z_{c})$. It can
be seen as a two-head Variational Auto-Encoder (VAE) [20] with one head
encoding $o_{1}$ and the other for $o_{T}$, and we call it a predictive VAE.
We jointly optimize $\phi$ and $\varphi$ by maximizing the evidence lower
bound:

$l(\phi,\varphi)=\mathbb{E}_{q_{\varphi}}[\log
p_{\phi}(o_{1},o_{\scriptscriptstyle T}|z_{c})]-{\rm
KL}(q_{\varphi}(z_{c}|o_{1},o_{\scriptscriptstyle T})||p(z_{c}))$ ,

(2)

where we assume $p(z_{c})$ is a Gaussian prior parameterized by the context
variable. By training in this way, the encoder
$q_{\varphi}(z_{c}|o_{1},o_{t})$ is enforced to learn a compact representation
from the given observations, $o_{1}$ and $o_{T}$, to convey the desired task,
which serves as the contextual information.

###  3.2 Generation Model

After inferring the context variable, the remaining question is: how to model
$p(a_{1:T},s_{1:T}|z_{c})$ to solve the planning problem? We assume the
underlying process in Fig. 2 is a fully observable Goal-conditioned Markov
Decision Process
$(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\mathcal{C})$, where
$\mathcal{S,A}$ is the state and action space. We denote $p(a_{t}|s_{t})$ as
policy $\pi_{\theta}$ and $p(s_{t}|z_{c},s_{t-1},a_{t-1})$ as transition model
$\mathcal{T}_{\mu}$. In this way, the generative model
$p(a_{\scriptscriptstyle{1:T}},s_{\scriptscriptstyle{1:T}}|z_{c})$ can be
factorized as:

$p(a_{\scriptscriptstyle{1:T}},s_{\scriptscriptstyle{1:T}}|z_{c})=\prod^{T}_{t=1}\pi_{\theta}(a_{t}|s_{t})\mathcal{T}_{\mu}(s_{t}|z_{c},s_{t-1},a_{t-1})\;,$
(3)

where we use the convention that $s_{0},a_{0}=0$.

A popular way to solve the MDP problem is using RL algorithms. However, we
only have access to expert trajectories
$\\{(o_{\scriptscriptstyle{1:T}}^{j},a_{\scriptscriptstyle{1:T}}^{j})\\}_{j=0}^{K}{\sim}\pi_{E}$
without a well-defined reward function, making it infeasible to directly apply
RL algorithms. Therefore, we adopt an IL approach and use the expert
trajectories as demonstrations. However, there are still several key
difficulties:  a) typical IL algorithm is model-free algorithm that is ideal
for learning habitual behavior without thought for actions’ consequences [24],
making it imperfect to learn goal-directed actions.  b) the static dataset
cannot provide feedback signals or the transferred states as the learning
agent interacts with it; and  c) each demonstration trajectory is performed by
the expert to reach a specific goal and thus might not be sufficiently
explored under different situations.  Below, we address these difficulties.

Effective Imitating with Transition Model. Instead of short-term environment
learning as in [6], we optimize model with the whole trajectory. Inspired by
GAIL [16], we formulate the IL problem as an occupancy measure matching
problem [17], where the goal is to minimize the Jenson-Shanon divergence of
trajectory distributions induced by the learned policy $\pi_{\theta}$ and the
expert policy $\pi_{E}$ respectively. In order to learn goal-directed actions,
and bypass the need of an interactive environment which is required the
original GAIL, we employ a transition model to roll out and jointly optimize
it with policy learning. There are two important reasons for the joint
optimization:  a) during the training, the action policy is not stationary,
which means a pre-trained transition model will not help the action policy
explore better decisions; and  b) the transition model can interact with
action policy so that the learned latent space is optimized on the entire
state-action pairs induced by the expert policy $\pi_{E}$, which helps it
incorporate information over multiple time steps.  The model can be either
deterministic or stochastic; thus, we introduce two versions of the transition
model.

Int-MGAIL: In Interior-Model GAIL, the transition model is built inside the
LSTM cell, which can be seen as a fully deterministic model. We modify the
LSTM cell and treat the long-term cell state as the state $s_{t}$ in Eq. 3 and
the short-term hidden state as our action $a_{t}$, so that we can enforce the
action  a) to interact with the hidden state to roll out the next state  b)
only depends on the current state.  At each time-step, the input to the cell
is the previous cell’s long-term and short-term state $s_{t-1}$ and $a_{t-1}$,
as shown in Eq. 4.

$\begin{split}f_{t}&=\sigma(W_{f}*a_{t}+U_{f}*s_{t}+b_{f})\;,\\\
i_{t}&=\sigma(W_{i}*a_{t}+U_{i}*s_{t}+b_{i})\;,\\\
a_{t}&=\textup{Tanh}(W_{a}*s_{t}+b_{a})\;,\\\
s_{t+1}&=f_{t}*[s_{t},z_{c}]+i_{t}*a_{t}\;.\end{split}$ (4)

Ext-MGAIL: Int-MGAIL provided a deterministic solution for modeling the
unknown environment, but it will underestimate the uncertainty of the
environment. Therefore, we further take the transition model as an external
module to explicitly model the environment transition in a stochastic way,
meaning different observations can follow the same state. To model the
uncertainty, we designed the action policy as a stochastic model with
Bernoulli probability vector of $a_{t}$ because of the discrete action space.
The stochastic modeling is crucial for successful planning when we have the
same start and goal states but different procedures.

Hindsight Relabeling. The problem with the static dataset is that each episode
only shows one possible way to reach the specified goal, which limits the
agent’s ability to explore what would happen had the circumstance been
different. Inspired by HER [3], we utilized the relabeling method that tries
to alleviate this problem by augmenting the demonstrations with “fake” goals
that were attained in the episode, allowing the agent to sufficiently explore
the state-action space and make better decisions for the future. Formally, we
have one valid trajectory
$\\{(o_{\scriptscriptstyle{1:T}}^{j},a_{\scriptscriptstyle{1:T}}^{j})\\}$ of
an expert attempting to reach the goal $o_{\scriptscriptstyle T}$ at $j$th
episode from the start. Then, the portion of this trajectory between any two
non-adjacent observations, $o_{m}$ and $o_{n}$, can also be seen as a valid
trajectory as the expert attempts to reach $o_{n}$ starting from $o_{m}$.
Therefore, for every trajectory in the original dataset, we select two non-
adjacent observations and augment the dataset with $D\leftarrow
D\cup{(o_{m:n},a_{m:n})}$. The intuition behind this process is that we can
replay each episode with a different goal than the one the expert was
initially trying to achieve.

###  3.3 Learning

We have three main components to be optimized:  a) the transition model
$\mathcal{T}_{\mu}(s_{t+1}|s_{t},a_{t},z_{c})$ that uses previous state-action
pair and context variable to predict the next state;  b) the policy model
$\pi_{\theta}(a_{t}|s_{t})$ that models the distribution over the set of
action under current state; and  c) the discriminator $\mathcal{D}_{\omega}$,
parameterized by $\omega$, tries to distinguish the $\\{(s_{t},a_{t})\\}$ from
the expert or the learned policy $\pi_{\theta}$.

We refer the expert trajectory as $\tau^{E}=\\{(s^{\scriptscriptstyle
E}_{t},a^{\scriptscriptstyle E}_{t})\\}$ and trajectory
$\tau=\\{(s_{t},a_{t})\\}$ as state-action pairs visited by the current
learned policy. We first randomly sample $\tau^{E}$ from the dataset and roll
out $\tau^{E}$ accordingly, then we optimize the discriminator by ascending
the gradient in Eq. 5:

$\mathbb{E}_{\pi_{\theta}}[\nabla_{\omega}\log(1-\mathcal{D}_{\omega}(s_{t},a_{t}))]+\mathbb{E}_{\pi_{\scriptscriptstyle
E}}[\nabla_{\omega}\log(\mathcal{D}_{\omega}(s^{\scriptscriptstyle
E}_{t},a^{\scriptscriptstyle E}_{t}))]\;.$ (5)

We further let the discriminator gradient back-propagate into the previous
time-step, helping the transition model to learn the further consequences
related to current action. However, we observe a high variance problem during
the training. Therefore, we employ an additional loss to help the generated
states $s_{t}$ quickly move to regions close to the expert-visited states.
Hence, the transition model is optimized by descending the gradient in Eq. 6:

$\begin{split}&\mathbb{E}_{a_{t}\sim\pi_{\theta}}[\nabla_{\mu}\log(1-\mathcal{D}_{\omega}(\mathcal{T}_{\mu}(s_{t-1},a_{t-1},z_{c}),a_{t}^{\scriptscriptstyle
E}))]\\\ &+\mathbb{E}_{s_{t}^{\scriptscriptstyle E}\sim\pi^{\scriptscriptstyle
E}}[\nabla_{\mu}\mathcal{L}(\mathcal{T}_{\mu}(s_{t-1},a_{t-1},z_{c}),s_{t}^{\scriptscriptstyle
E})]\;,\end{split}$ (6)

where $\mathcal{L}$ measures the distance between two latent vectors.

The last component is the action policy $\pi_{\theta}$. After optimizing the
discriminator, we can interpreted it as a local reward function and we
optimize policy to maximum the reward
$r(s_{t},a_{t})=log(D_{\omega}(s_{j},a_{j}))$. In order to imitate the expert
rather than mimicking, the action policy needs the ability to intentionally
explore actions that the expert did not perform. We adopted the offline policy
evaluation in offline-RL and follow the classical evaluation method [12, 8],
re-weighting the rewards by the importance sampling ratio (Eq. 7) to select a
better policy during training. Concretely, we first learn a classification
network as the behavior policy $\beta(a_{t}|s_{t})$ from demonstrations via
behavioral cloning. Then optimize $\pi_{\theta}$ with policy gradient which
tries to maximize the accumulated reward along the whole trajectory:

$\mathbb{E}_{\beta}[\frac{\pi_{\theta}(a_{t}|s_{t})}{\beta(a_{t}|s_{t})}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})Q(s_{t},a_{t})]-\lambda\mathcal{H}(\pi_{\theta})\;,$
(7)

where
$\mathcal{H}(\pi_{\theta})=\mathbb{E}_{\pi_{\theta}}[-\log\pi_{\theta}(a|s)]$
is the policy entropy.

Input: All observations $\\{o_{i}\\}^{N}_{i=1}$, set of action
$\\{a_{i}\\}_{i=1}^{M}$, models $\mathcal{T}_{\mu},\pi_{\theta}$ planning
length $T$

1:Initialize observation list $\beta\leftarrow\emptyset$

2:for $i=1,2,\cdots,N$ do

3: $s_{i}=\phi(o_{i})$

4:for $i=1,2,\cdots,N$ do

5: $s_{\rm next}=\mathcal{T}_{\mu}(s_{i},\pi_{\theta}(s_{i}))$

6: $\triangleright$ Find the index of the nearest state

7: $k=\underset{k}{\arg\min}\lVert s_{k}-s_{\rm next}\rVert_{2}^{2}$

8: $\triangleright$ Increment the transition probability over all action

9: $S_{i,k}\mathrel{+}=\sum\limits_{m=0}^{M}\pi_{\theta}(a_{m}|s_{i})$

10:$\beta\leftarrow\underset{\rho\in
Perm(T)}{\arg\max}\sum\limits_{i=1}^{T}S_{i,\rho(i)}$

Algorithm 1 Walk-through Planning

###  3.4 Walk-through Planning with Transition Model

Given the start and goal observations, we first infer the contextual
information by sampling from $q_{\varphi}(z_{c}|o_{1},o_{T})$. Based on the
sampled $z_{c}$, the generation model will roll out subsequent actions and
hidden states as the sampled trajectory. Given the pool of visual observations
$\\{o_{i}\\}$, we first construct the score matrix $S_{i,j}$ to capture the
transition probability between $o_{i}$ and $o_{j}$ with the sampled
trajectory, as shown in Alg. 1. After constructing the rank score table, we
can then perform walk-through planning to retrieve the intermediate
observations that lead to the goal. As suggested in [6], this problem can be
seen as ﬁnding a permutation function $b$ :
$\\{1,2,\cdots,T\\}\rightarrow\\{1,2,...,T\\}$ that maximizes the transition
probability along the permutation path, subject to the constraints that
$b(1)=1,b(T)=T$.

##  4 Experiments

We choose CrossTask [43] to conduct our experiments, which consists of 2,750
video (212 hours in total). Each video depicts one of the 18 primary long-
horizon tasks such as Grill Steak or Make French Strawberry Cake. For the
videos in each task, we randomly select 70% for training and 30% for testing.
Different tasks have various procedure steps: less complex tasks include jack
up a car (3 steps); more complex ones include pickle cucumbers or change tire
(11 steps), and the steps do not necessarily appear in the same order as the
task description as shown in Fig. 4.

Each video has densely annotated boundaries with caption labels that describe
the person’s actions in the video. We treat each video as a sequence of images
$I_{1:N}$ having annotated description $v_{1:M}$ with temporal boundaries
$(s_{1:M},e_{1:M})$. For $i$-th video clip, we choose frames around the
beginning of the captions $I_{s_{i}-\delta:s_{i}+\delta}$ as $o_{i}$, caption
description $v_{i}$ as the semantic meaning of action, and images nearby the
end $I_{e_{i}-\delta:e_{i}+\delta}$ as the next observation $o_{i+1}$. Here,
$\delta$ controls the duration of each observation, and we set $\delta=1$ for
all experiments. We further use the relabeling technique introduced in Section
3.2 to augment the data with randomly selected 30% of the expert trajectories.

To construct our state-space $\mathcal{S}$, we use pre-computed features
provided in CrossTask as our state estimation: One second of the video is
encoded into a 3,200-dimensional feature vector which is a concatenation of
the I3D, Resnet-152, and audio VGG features [15, 14, 4]. Note here we do not
use the state estimations for testing; we only use them for training the
Generation model. Lastly, we construct the action space $\mathcal{A}$ by
enumerating all combinations of the caption description’s predicates and
objects, which provides 105 action labels and are shared across all 18 tasks.

Implementation Details. For computing context variables, we use the DCGAN
architecture [29] as the image encoder and decoder in our model. The behavior
policy is a classification network that takes state estimation as input and
generates the probability over the action space. The policy network for both
Int-MGAIL and Ext-MGAIL share a similar structure as the off-policy Actor-
Critic network [8], which is two-headed: one for computing an action based on
a state and another one producing the expected return values of the action. In
the Ext-MGAIL, we assume our hidden state $s_{t}$ to be Gaussian; thus, the
transition model is Gaussian with mean and variance parameterized by a feed-
forward neural network. The discriminator networks for both models share the
same architecture, which is a similar network in the original GAIL [16].
Further implementation details can be found in the supplementary material.

Figure 4: Expert trajectories of Grill Steak task. Heavier color indicates
more frequently visited path

###  4.1 Evaluating Procedure Planning

We compared with the following methods:

\- Uniform Policy. At each step, the algorithm will uniformly sample one
action from all actions. This method serves as the empirical lower bound of
performance.

\- Universal Planning Networks (UPN) [32]. Like our method, UPN learned a
plannable latent representation where gradient descent can be used to compute
a plan that minimizes a supervised imitation loss. We expand the original UPN
to discrete action space by using a softmax layer to output probability over
discrete actions.

\- Dual Dynamics Networks (DDN) [6]. DDN is the first work proposing procedure
planning in instructional video problem. Similar to UPN, it learns the dual
dynamic of the state-action transition and perform sample-based planning upon
the learned latent representation.

When evaluating with the pre-collected dataset, a common way is the re-
weighted rewards [23]. But there is no defined rewards here. To keep
consistent with state-of-the-art methods, we use three different matrices to
evaluate the performance and limit the experiment to length 3-5 even our
method is applicable for longer trajectory modeling.

\- Success Rate is designed to evaluate the long-term action association,
_i.e_., the correctness of action sequence. Only if every action matches, this
plan is considered as a success. The result is the percentage of the successes
in test dataset.

\- Accuracy is used to evaluate the correctness of individual time step
action, which serves as a constraint relaxation of the success rate metric.
The individual action is considered as a success only if it matches the ground
truth at the same time step, which is written in terms of percentages.

\- mIoU is used to capture the cases where the model can output the right
actions but fail to preserve the actions’ order. We adopt this metric from [6]
that computes IoU
$\frac{|\\{a_{t}\\}\cap\\{\hat{a_{t}}\\}|}{|\\{a_{t}\\}\cup\\{\hat{a_{t}}\\}|}$
between the set of ground-truth $\\{a_{t}\\}$ and the planned actions
$\\{\hat{a_{t}}\\}$.

As illustrated in Table 1, UPN can learn representations that perform
reasonably well compared to the uniform baseline. However, as instructional
videos’ action space is not continuous, the gradient-based planner cannot work
well. The proposed Int-MGAIL outperforms baseline DDN at two different time-
scales. The reason is that we perform RL training, which maximizes the
accumulated reward alone the whole trajectory. By introducing the stochastic
process into the action policy, our Ext-MGAIL has a better performance. This
is because, given the same beginning and goal observation, there is more than
one valid sequence of actions. By designing a model with both stochastic and
deterministic components, we show that our agent successfully learns plannable
representations from real-world videos to outperform all the baseline
approaches on all metrics.

Table 1: Results of Procedure Planning. Our models signiﬁcantly outperform the baselines by $\sim 10\%$ improvement in terms of the success rate. Our Ext-MGAIL has a marginal improvement compared with Int-MGAIL; this shows that introducing a stochastic process in the policy can help the policy explore and thus improve the performance. |  | Uniform | UPN | DDN | Int | Ext
---|---|---|---|---|---|---
| Succ. | 0.01 | 2.89 | 12.18 | 17.03 | 21.27
${\scriptstyle{\rm T}=3}$ | Acc. | 0.94 | 24.39 | 31.29 | 44.66 | 49.46
| mIoU | 1.66 | 31.56 | 47.48 | 58.08 | 61.70
| Succ. | 0.01 | 1.19 | 5.97 | 9.47 | 16.41
${\scriptstyle{\rm T}=4}$ | Acc | 0.83 | 21.59 | 27.10 | 37.16 | 43.05
| mIoU | 1.66 | 28.85 | 48.46 | 57.24 | 60.93
Figure 5: Walk-through Planning qualitative results for Grill Steak. Given the
starting and goal observations, our model can output the correct order for
each step.

###  4.2 Evaluating Walk-through Planning

Different from procedure planning, given the the observations $o_{1}$
$o_{\scriptscriptstyle T}$, the model needs to generate the intermediate
observations $\\{o_{2},\cdots,o_{T}\\}$. Directly predicting the raw RGB image
is unnecessarily hard [10]. Similar to the setup in [6], instead of generating
frames, we retrieve the intermediate video clips from dataset in the correct
order. To find the path from $o_{1}$ to $o_{\scriptscriptstyle T}$, a rank
table is constructed to evaluate the transition probability between two clips.

In addition to the Uniform policy and DDN, we include the Causal InfoGAN
(CIGAN) into comparison. Like our approach, they plan the trajectory in latent
space but use the generative model to transform the trajectory to observations
directly. The advantage of CIGAN is that it can be trained to perform walk-
through planning without action supervision.

The evaluations are conducted on the following metrics. \- Hamming. As
described earlier, we are finding the best permutation of the observation
index. Then the distance is defined as
$d(y,\hat{y})=\sum_{i=1}^{T}I(1|y_{i}\neq\hat{y}_{i})$, which is good for
evaluating the single step observation order.

\- Pair Accuracy. To compare the distance between two permutation sequence, we
use pairwise accuracy to calculate the distance along the planned and ground
truth observation orders. This is defined as $\frac{2}{T(T-1)}\sum_{i<j,i\neq
j}^{T}I(1|y_{i}<\hat{y}_{j})$.

The results are shown in Table 2 and Fig. 5. CIGAN can learn reasonable models
beyond Uniform without using action supervision. However, the complexity of
the instructional videos requires explicit modeling of the forward dynamics
conditioned on the semantic actions. Our two methods outperform all baseline
models, which means both models are applicable to both planning and walk-
through planning. We also show that effective IL requires learning a
transition model and optimizing policy on multi-step transitions instead of
individual state-action pairs.

Table 2: Results of Walk-through Planning. Our model outperforms the baselines by explicitly modeling the transition dynamics between temporally adjacent observations. |  | Uniform | UPN | DDN | Int | Ext
---|---|---|---|---|---|---
${\scriptstyle{\rm T}=3}$ | Ham. | 1.06 | 0.57 | 0.26 | 0.19 | 0.13
Pacc. | 46.85 | 71.55 | 86.81 | 86.98 | 93.66
${\scriptstyle{\rm T}=4}$ | Ham. | 1.36 | 1.36 | 0.88 | 0.70 | 0.57
Pacc. | 52.23 | 68.41 | 81.21 | 86.42 | 89.74

###  4.3 Visualization of Contextual Information

In this section, we aim to answer the following two questions:  a) Can the
proposed Inference model learn the useful contextual information of different
tasks from the demonstrations?  b) Why does the contextual information help
the subsequent action learning?  To this end, we use t-SNE [25] to reduce the
dimension of the context variable $z_{c}$ to 2 and visualize $z_{c}$ all 18
tasks as shown in Fig. 6. For every task, we randomly sampled 100 pairs of
start and goal clips (1800 pairs in total) and extracted their contextual
information.

As we can see, all the samples are grouped by the tasks’ labels, which
suggests the Inference model has learned roughly distinct regions in the
hidden space to correspond to each task in the dataset. Further task
descriptions can be found in the supplementary material. Considering we never
used the task labels in the learning process, this result indicates an
underlying relationship between the different tasks’ observations. This
suggests that the Inference model can encode sufficient information to convey
the desired task. Furthermore, the Generation model can benefit from this
concise embedding on modeling decision-making because irrelevant deviations
from the raw pixel space will create exponentially diverging trajectories.

However, there still exist overlaps between different clusters, and some
samples drift from the majority. This reflects the fact that we cannot
entirely rely on the contextual information to recover the expert’s decision
process, which further validates the effectiveness of the Generation model.

###  4.4 Ablation Study

We conducted experiments with three variations: w/o r: The model is trained
with both sequence mapping loss and discriminator loss but w/o maximizing
accumulated reward. In this way, the model tries to match the short-term
actions w/o considering the trajectory as whole. w/o dis: We further drops the
discriminator loss in Eq. 6, making it a supervised seq2seq learning model. As
shown in Table 3, learning the model solely w/o dis will significantly hurt
the overall performance because the model simply optimized on expert
trajectory will be over-fitted to the regions that the expert traversed and
thus make it hard to generalize to other areas that might be helpful for
action policy learning. We think the main reason for the low success rates
with sequence mapping is that pure supervised learning will excessively focus
on recovering the same action sequence as ground-truth, ignoring the fact that
actions can be exchanged to achieve the same goal. w/o HER: We observed that
combining HER always brings performance boosts. Our stochastic
$\textrm{Ext}_{\textrm{w/o HER}}$ has a lower performance than Int-MGAIL; we
suspect that it is because, without HER, the original dataset is insufficient
for stochastic model optimizing the whole trajectory. This result is
consistent with our observation on the experiment with the additional dataset.

${\scriptstyle{\rm T}=3}$ |  Int w/o r |  Int w/o dis |  Ext w/o r |  Ext w/o dis |  Int w/o HER |  Ext w/o HER
---|---|---|---|---|---|---
Succ. | 7.18 | 5.89 | 15.18 | 11.42 | 14.39 | 18.01
Acc. | 18.74 | 11.66 | 27.29 | 23.46 | 37.43 | 43.86
mIoU | 27.51 | 20.66 | 37.48 | 30.97 | 54.18 | 57.16

Table 3: Results of Ablation Study. The performance decreases signiﬁcantly
when optimizing without sequence modeling or only with a supervised learning
loss, which shows the importance of learning over the whole trajectory. Figure
6: Visualization of the contextual information of the starting and goal
observations of all 18 tasks in the CrossTask dataset. The colorbar indicates
the ground-truth task labels, where different colors indicate different tasks.

###  4.5 Addtional Experiment

We further compare our model with previous best performing method DDN on a
second dataset [2]. Both of our models outperformed the DDN, but the increase
of the accuracy on procedure planning (Succ.${\sim}4$%, T=3) is smaller than
one obtained on the CrossTask, and the performance between Int-MGAIL and Ext-
MGAIL are very similar (Succ. 20.19/22.11%, T=3). We suspect that it is
because the new dataset does not provide sufficient samples for optimizing
long trajectories. More details can be found in the supplementary material.

##  5 Conclusion, Application, and Future Work

In this paper, we present a new method to address the procedure planning
problem focusing on learning goal-directed actions. Concretely, we propose a
predictive VAE structure that learns to embed the contextual information of
the desired task. Moreover, we propose two novel model-based imitation-
learning algorithms to solve the formulated decision-making problem in unknown
environments. Results on real-world instructional videos show that our
approach can learn a meaningful representation for planning and uncover the
human decision-making process.

Being able to learn goal-directed actions from the pixels, the proposed method
enables the AI system to extract useful information from expert
demonstrations. Moreover, learning policy from the offline dataset avoids
online interaction with the environment, making our method practical in real-
world applications, _e.g_., service robots.

A direction of future work is investigating different ways to combine
contextual information with environment dynamics, such as through self-
supervision. Another important future direction is to consider the collected
data as policy constraints such that robots can act safely in the real world
and continuously improve themselves by accumulating data of the environment
interactions, making robotic agents more capable of solving challenging real-
life tasks.

Acknowledgments. This work has been partially supported by the National
Science Foundation (NSF) under Grant 1813709 and the National Institute of
Standards and Technology (NIST) under Grant 60NANB17D191. The article solely
reflects the opinions and conclusions of its authors but not the funding
agents.

## References

  * [1] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
  * [2] Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien.  Unsupervised learning from narrated instruction videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4575–4583, 2016.
  * [3] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  In Advances in neural information processing systems, pages 5048–5058, 2017.
  * [4] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [5] Sergio Casas, Wenjie Luo, and Raquel Urtasun.  Intentnet: Learning to predict intention from raw sensor data.  In Conference on Robot Learning, pages 947–956, 2018.
  * [6] Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, and Juan Carlos Niebles.  Procedure planning in instructional videos.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [7] Nathaniel D Daw, Yael Niv, and Peter Dayan.  Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control.  Nature neuroscience, 8(12):1704–1711, 2005.
  * [8] Thomas Degris, Martha White, and Richard S Sutton.  Off-policy actor-critic.  arXiv preprint arXiv:1205.4839, 2012.
  * [9] Raab .et al.  Ch. 13. the development of goal-directed decision-making.  In Goal-Directed Decision Making. Academic Press, 2018.
  * [10] Chelsea Finn, Ian Goodfellow, and Sergey Levine.  Unsupervised learning for physical interaction through video prediction.  In Advances in neural information processing systems, pages 64–72, 2016.
  * [11] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In International Conference on Machine Learning, pages 2555–2565, 2019.
  * [12] Josiah Hanna, Scott Niekum, and Peter Stone.  Importance sampling policy evaluation with an estimated behavior policy.  In International Conference on Machine Learning, pages 2605–2613. PMLR, 2019.
  * [13] Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen.  Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8393–8400, 2019.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al.  Cnn architectures for large-scale audio classification.  In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131–135. IEEE, 2017.
  * [16] Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4565–4573. Curran Associates, Inc., 2016.
  * [17] Jonathan Ho, Jayesh Gupta, and Stefano Ermon.  Model-free imitation learning with policy optimization.  In International Conference on Machine Learning, pages 2760–2769, 2016.
  * [18] Zhewei Huang, Wen Heng, and Shuchang Zhou.  Learning to paint with model-based deep reinforcement learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8709–8718, 2019.
  * [19] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al.  Model-based reinforcement learning for atari.  arXiv preprint arXiv:1903.00374, 2019.
  * [20] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [21] Yu Kong and Yun Fu.  Human action recognition and prediction: A survey.  arXiv preprint arXiv:1806.11230, 2018.
  * [22] Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart J Russell, and Pieter Abbeel.  Learning plannable representations with causal infogan.  In Advances in Neural Information Processing Systems, pages 8733–8744, 2018.
  * [23] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  arXiv preprint arXiv:2005.01643, 2020.
  * [24] Mimi Liljeholm.  Instrumental divergence and goal-directed choice.  In Goal-Directed Decision Making, pages 27–48. Elsevier, 2018.
  * [25] Laurens van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(Nov):2579–2605, 2008.
  * [26] Brais Martinez, Davide Modolo, Yuanjun Xiong, and Joseph Tighe.  Action recognition with spatial-temporal discriminative filter banks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5482–5491, 2019.
  * [27] Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess.  Learning human behaviors from motion capture by adversarial imitation.  arXiv preprint arXiv:1707.02201, 2017.
  * [28] Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan.  Vision-based navigation with language-based assistance via imitation learning with indirect intervention.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12527–12537, 2019.
  * [29] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [30] Nicholas Rhinehart and Kris M Kitani.  First-person activity forecasting with online inverse reinforcement learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3696–3705, 2017.
  * [31] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  Nature, 550(7676):354–359, 2017.
  * [32] Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn.  Universal planning networks.  arXiv preprint arXiv:1804.00645, 2018.
  * [33] James Supancic III and Deva Ramanan.  Tracking as online decision-making: Learning a policy from streaming videos with reinforcement learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 322–331, 2017.
  * [34] Richard S Sutton.  Dyna, an integrated architecture for learning, planning, and reacting.  ACM Sigart Bulletin, 2(4):160–163, 1991.
  * [35] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [36] Weining Wang, Yan Huang, and Liang Wang.  Language-driven temporal activity localization: A semantic matching reinforcement learning model.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 334–343, 2019.
  * [37] Xin Wang, Wenhu Chen, Jiawei Wu, Yuan-Fang Wang, and William Yang Wang.  Video captioning via hierarchical reinforcement learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4213–4222, 2018.
  * [38] Ping Wei, Yang Liu, Tianmin Shu, Nanning Zheng, and Song-Chun Zhu.  Where and why are they looking? jointly inferring human attention and intentions in complex tasks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6801–6809, 2018.
  * [39] Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon.  Meta-inverse reinforcement learning with probabilistic context variables.  arXiv preprint arXiv:1909.09314, 2019.
  * [40] Kuo-Hao Zeng, William B Shen, De-An Huang, Min Sun, and Juan Carlos Niebles.  Visual forecasting by imitating dynamics in natural sequences.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2999–3008, 2017.
  * [41] Mengmi Zhang, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Jiashi Feng.  Deep future gaze: Gaze anticipation on egocentric videos using adversarial networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [42] Kaiyang Zhou, Yu Qiao, and Tao Xiang.  Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [43] Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic.  Cross-task weakly supervised learning from instructional videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3537–3545, 2019.


# EuCd2As2: a magnetic semiconductor

D. Santos-Cottin  Department of Physics, University of Fribourg, CH-1700
Fribourg, Switzerland  I. Mohelský  LNCMI, CNRS-UGA-UPS-INSA, 25, avenue des
Martyrs, F-38042 Grenoble, France  J. Wyzula  F. Le Mardelé  Department of
Physics, University of Fribourg, CH-1700 Fribourg, Switzerland  LNCMI, CNRS-
UGA-UPS-INSA, 25, avenue des Martyrs, F-38042 Grenoble, France  I. Kapon
Department of Physics, University of Geneva, CH-1204 Geneva, Switzerland  S.
Nasrallah  Department of Physics, University of Fribourg, CH-1700 Fribourg,
Switzerland  N. Barišić  Institute of Solid State Physics, TU Wien, A-1040
Vienna, Austria  Department of Physics, Faculty of Science, University of
Zagreb, Bijenička 32, HR-10000 Zagreb, Croatia  I. Živković  J. R. Soh
Institut de Physique, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015
Lausanne, Switzerland  F. Guo  M. Puppin  J. H. Dil  Lausanne Centre for
Ultrafast Science (LACUS), École Polytechnique Fédérale de Lausanne (EPFL),
CH-1015 Lausanne, Switzerland  Institut de Physique, École Polytechnique
Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland  B. Gudac  Z.
Rukelj  M. Novak  Department of Physics, Faculty of Science, University of
Zagreb, Bijenička 32, HR-10000 Zagreb, Croatia  A. B. Kuzmenko  Department of
Physics, University of Geneva, CH-1204 Geneva, Switzerland  C. C. Homes
National Synchrotron Light Source II, Brookhaven National Laboratory, Upton,
New York 11973, USA  Tomasz Dietl  International Research Centre MagTop,
Institute of Physics, Polish Academy of Sciences, Aleja Lotnikow 32/46,
PL-02668 Warsaw, Poland  WPI Advanced Institute for Materials Research, Tohoku
University, 2-1-1 Katahira, Aoba-ku, Sendai 980-8577, Japan  M. Orlita  LNCMI,
CNRS-UGA-UPS-INSA, 25, avenue des Martyrs, F-38042 Grenoble, France  Institute
of Physics, Charles University, CZ-12116 Prague, Czech Republic  Ana Akrap
<EMAIL_ADDRESS>Department of Physics, University of Fribourg, CH-1700
Fribourg, Switzerland

(August 28, 2024)

###### Abstract

EuCd2As2 is now widely accepted as a topological semimetal in which a Weyl
phase is induced by an external magnetic field. We challenge this view through
firm experimental evidence using a combination of electronic transport,
optical spectroscopy and excited-state photoemission spectroscopy. We show
that the EuCd2As2 is in fact a semiconductor with a gap of 0.77 eV. We show
that the externally applied magnetic field has a profound impact on the
electronic band structure of this system. This is manifested by a huge
decrease of the band gap, as large as 125 meV at 2 T, and consequently, by a
giant redshift of the interband absorption edge. However, the semiconductor
nature of the material remains preserved. EuCd2As2 is therefore a magnetic
semiconductor rather than a Dirac or Weyl semimetal, as suggested by ab initio
computations carried out within the local spin-density approximation.

Magnetic Weyl semimetals have harboured the great hope of bringing spintronics
and topology together. Materials where this might come to fruition are scarce,
and their solid experimental confirmations are even scarcer Liu  _et al._
(2019); Morali  _et al._ (2019); Nie  _et al._ (2022); Wang  _et al._ (2022);
Kanagaraj  _et al._ (2022). EuCd2As2 has been seen as one of a select few
magnetic Weyl semimetals—stable under ambient conditions, with large Eu spins
positioned on a frustrated triangular lattice. The interplay of frustrated
magnetism with topological bands made EuCd2As2 into a hopeful playground for a
broad range of exciting phenomena Ma  _et al._ (2019); Xu  _et al._ (2021).
Through an extreme sensitivity of valence and conduction bands to Eu
magnetism, external fields would then modify the band structure Soh  _et al._
(2019). This compound has up to this point been proposed and interpreted as a
Weyl semimetal, based upon ab initio band structure calculations, electronic
transport and photoemission measurements Ma  _et al._ (2019); Soh  _et al._
(2019); Jo  _et al._ (2020); Taddei  _et al._ (2022).

In this work, we study ultraclean EuCd2As2 crystals. Electronic transport
measurements indicate an extremely low hole concentration. Accordingly,
optical conductivity shows no detectable Drude component, and a strong
Reststrahlen phonon mode. Our pump-probe photoemission measurement points to a
clear band gap, and a carrier lifetime in the picosecond time scale. These
experiments, together with extensive optical measurements, provide decisive
proof that EuCd2As2 is a semiconductor with a gap of 770 meV, and not a
topological semimetal as previously thought Ma  _et al._ (2019, 2020); Jo  _et
al._ (2020); Soh  _et al._ (2018, 2020); Taddei  _et al._ (2022). Moreover,
our results demonstrate that the band structure of EuCd2As2 changes
dramatically in an external magnetic field, due to an intimate coupling to
localized Eu spins. All of these effects persist up to high temperatures, deep
within the paramagnetic phase. We understand this through a strong exchange
coupling of Eu 4$f$ localized spins to the band states originating from Eu
6$s$, 6$p$ and 5$d$ orbitals.

Figure 1: (a) Resistivity as a function of temperature for an insulating and a
metallic sample. Magnetic properties, (b) dc susceptibility, and (c)
magnetization at 4 K, are shown for the same two sample batches. (d) Static
ARPES band map. (e) Pump-probe ARPES experiment results at a delay of 200 fs,
where red means increased count rate after the pump-pulse excitation. (f) Time
traces of the intensity integrated at different energies $E_{b}$ above the
Fermi level. The integration is done for a 200 meV window around the indicated
energy. (g) Infrared reflectance at different temperatures, for the insulating
sample batch. (h) Transmission and reflectance at 5 K, and their multilayer
modelling, which results in (i) the real part of the optical conductivity,
$\sigma_{1}$.

Single crystals of EuCd2As2 were prepared by the Sn-flux method Ma  _et al._
(2019). Carrier density was changed through controlling the purity of the
starting materials. To increase the crystal size and quality, a two-step
process was employed, where the crystals from the first growth were used as a
seed material for the final growth. The layered trigonal lattice of EuCd2As2,
shown in the inset of Fig. 1(a), results in triangular and hexagonally shaped
crystals. The flux-growth of single crystals results in an optically isotropic
(001) surface. Blocks of Cd2As2 are sandwiched between the Eu planes, with
similar distances to binary semiconducting CdAs2 Schellenberg  _et al._
(2011). Infrared reflectance and transmission were measured using a Bruker
Vertex 80v spectrometer, at temperatures from 5 K to room temperature, with in
situ gold evaporation Homes  _et al._ (1993). High energy reflectance was
complemented by ellipsometry and X-ray data Tanner (2015). The magneto-
transmission and magneto-reflectivity were measured in the Faraday
configuration, in magnetic fields up to 16 T, and at temperatures from 4 K to
140 K. The light wave vector propagated parallel to the external magnetic
field, aligned with the crystallographic $c$ axis. A Bruker Vertex 80v
interferometer using an infrared source was coupled to the experiment using
light-pipe optics. The sample was located in a superconducting coil and
surrounded by the helium exchange gas. Time-resolved photoemission (tr-ARPES)
experiments were carried out at the EPFL LACUS, at the ASTRA end station
Crepaldi  _et al._ (2017) of the Harmonium beamline Ojeda  _et al._ (2015). A
single $s$-polarized harmonic at an energy of 37 eV was selected by a time-
preserving monochromator. The in situ cleaved sample was excited by an
$s$-polarized pump pulse at a fluence of 0.5 mJ/cm2. The measurements were
carried out at 80 K, with a temporal resolution better than 150 fs, an energy
resolution of 150 meV, and an angular resolution of $0.3^{\circ}$.

Figure 2:  (a) Near infrared transmission showing the interband absorption
edge at low fields, $B<1$ T. (b) Color plot of relative magneto-transmission,
$T_{B}/T_{\text{AVR}}$, in a broad energy range, and up to 2 T. (c) Magneto-
transmission $T_{B}/T_{0}$ and (d) its first field-derivative,
$d/dB[T_{B}/T_{0}]$, in a broad energy and magnetic field range.

Through a controlled crystal synthesis, we obtained insulating samples of
EuCd2As2, which have not been reported before. Figure 1(a) shows the
resistivity of an insulating sample, resulting from a higher-purity synthesis,
compared to a metallic sample, made through a standard purity synthesis. The
resistivity of a high-purity sample is thermally activated in the whole
temperature range, pointing to less intrinsic doping. The standard synthesis
results in metallic behavior above 50 K, consistent with the previous reports.
Interestingly, in both samples the resistivity peak at $T_{N}=9.5$ K coincides
with a sharp and symmetric peak in the magnetic susceptibility, $\chi_{dc}$ in
Fig. 1(b), and an antiferromagnetic (AFM) ordering Jo  _et al._ (2020). The
susceptibility in both metallic and insulating samples shows no difference
between the zero-cooled and field-cooled (measured in 10 mT) values, excluding
a possible ferromagnetic phase above $T_{N}$ Artmann  _et al._ (1996). The
magnetization $M(\mu_{0}H)$ is measured with the field applied along the $c$
axis, perpendicular to the Eu planes. In both samples, $M(\mu_{0}H)$ at first
steeply and linearly increases up to 0.8 T, followed by a kink at 1.8 T,
reaching a saturated value of 7 $\mu_{B}/$Eu atom. This corresponds to the
divalent Eu which has half filled 4$f$ orbitals, as was found in EuTe Oliveira
_et al._ (1972). The initial steep slope of $M(\mu_{0}H)$ is consistent with a
ferromagnetic structure within each layer, as well as with a relatively weak
magnetic anisotropy expected for atoms with a half-filled orbital. Despite a
strong difference in the resistivity, the susceptibility and magnetization are
very similar in both samples, implying a weak effect of residual doping on
magnetic properties.

The resistivity of the sample with lower residual doping strongly disagrees
with the widely accepted notion that EuCd2As2 is a semimetal, and instead
suggests it is a semiconductor. To confirm this, we turn to photoemission
measurements of the more resistive samples. Figure 1(d) shows a static ARPES
band map, indicating that the band structure agrees with the published spectra
Soh  _et al._ (2019); Ma  _et al._ (2019); Jo  _et al._ (2020). A strong
$4f^{7}$ band is centered around 1.3 eV below the Fermi level, and the valence
band is near the Fermi level. The results of a pump-probe experiment are shown
in Fig. 1(e), where we see the conduction band minumum at an energy around 800
meV above the Fermi level. No further states are seen in the band gap under
these conditions. The conduction band shows extremely high intensity and is
visible without any data treatment. Time traces of the intensity integrated at
different energies above the Fermi level are shown in Fig. 1(f). The valence
band maximum at $\sim 130$ meV, and the conduction band away from the minimum,
both show a time delay of around 1 ps. In contrast, the conduction band
minimum ($E_{b}=-0.86$ eV) shows hardly any decay within the full observed
time range of 10 ps. The long recombination time implies an energy barrier,
and confirms there is a band gap of $770\pm 70$ meV.

Figure 3:  (a–l) Colorplots of relative infrared magneto-reflectivity,
$R_{B}/R_{0}$. Each panel represents a different temperature, going from the
AFM phase below 9.5 K, deep into the paramagnetic phase. (m) Magnetization as
a function of magnetic field, at the same set of temperatures where magneto-
reflectivity is shown. (n) Onset of absorption extracted from the magneto-
reflectivity color plots, at the same set of temperatures as in (m).

The infrared properties of EuCd2As2 are shown in Fig. 1(g-i), determined for
the insulating sample. Far-infrared reflectance, shown at several different
temperatures in Fig. 1(g), is dominated by two strong in-plane, $E_{u}$ phonon
modes, at 80 and 180 cm-1 Homes, C. C. and Wang, Z. -C. and Fruhling, K. and
Tafti, F. (2022). The higher frequency mode is an unscreened Reststrahlen
mode, since there are no underlying free carriers to screen it. The rather
weak temperature dependence of the reflectance is again typical of
semiconductors. Moreover, the value of reflectance at low photon energies is
far below unity, which contrasts with behaviour of metals and semimetals. To
obtain the precise value of the semiconducting band gap, we determine
transmission through a thin EuCd2As2 sample. Transmission and reflectance are
modelled using a multilayer model of the dielectric response Kuzmenko (2005).
Kramers-Kronig analysis of reflectance is unreliable as the sample is highly
transparent in the mid-infrared range. The obtained optical conductivity,
$\sigma_{1}(\omega)$, is shown in Fig. 1(i). The onset of absorption coincides
with a sudden drop in transmission, and we place it at 770 meV (6200 cm-1). No
Drude component is present in the optical conductivity, in line with our
resistivity measurement. Metallic samples SM  give reflectance with more
screened phonons, in agreement with a previous optical study Wang  _et al._
(2016) and similar to EuCd2P2 Homes, C. C. and Wang, Z. -C. and Fruhling, K.
and Tafti, F. (2022). In metallic and insulating samples, the infrared phonons
appear at the same frequencies, and there is a strong increase of the optical
conductivity above 1 eV. In both cases, the Drude contribution, if at all
present, is minor. Based on the experimental evidence outlined thus far, we
assert that EuCd2As2 is a semiconductor—in our case with a light $p$-type
doping—in which the carrier density depends on the starting material purity
and the crystal synthesis.

The electronic band structure of EuCd2As2 is remarkably tunable even with a
small magnetic field. First, let us focus on the interband absorption edge,
which is evident from magneto-transmission. In Fig. 2a, transmission spectra
taken at 4 K are shown at photon energies around the onset, as the magnetic
field is increased in 50 mT steps up to 1 T. Relative magneto-transmission in
a broader energy and magnetic field range is shown in color plots in Fig.
2(b-d), where in (d) we show the field-derivative of the magneto-transmission.
The sharp zero-field step at 770 meV transforms into a double-step feature
already at the lowest fields. Below 0.5 T, the lower energy step shifts down
in energy, and the higher energy step moves up in energy. Their splitting is
given by 160 meV/T, which may be described in terms of an effective $g$ factor
reaching an extremely high value of $\sim 1500$ Kirchschlager  _et al._
(2004). This large splitting points to exchange interaction rather than Zeeman
splitting. Above 0.5 T, the higher energy step also begins to redshift. The
splitting between the two steps saturates at about 50 meV, a value remaining
constant up to 16 T. We may understand this as a signature of spin polarized
bands, which differently absorb light with opposite circular polarization.
Remarkably, the band gap decreases by $\Delta E_{\text{g}}=125$ meV under 2 T
of applied field, reaching a plateau above 5 T.

While the gap is strongly reduced, it does not close in field, and no band
inversion is seen. Instead, the redshift of the band gap $\Delta
E_{\text{g}}(B)$, seen prominently in Fig 2(b-d), is proportional to Eu spin
magnetization $M(\mu_{0}H)$, similarly to EuTe Schmutz  _et al._ (1978). This
means that the molecular field approximation is valid, and $\Delta
E_{\text{g}}$ can be written in the form:

$\Delta E_{\text{g}}=-\frac{1}{2}{\cal{J}}_{\text{eff}}SM(T,H)/M_{S},$ (1)

where ${\cal{J}}_{\text{eff}}$ is an effective exchange energy between band
carriers and Eu spins, $S=7/2$, and $M_{S}$ is Eu spin magnetization at
saturation. Since for highly localized $4f$ electrons we expect a weak
hybridization with band states, ${\cal{J}}_{\text{eff}}$ originates from the
intraatomic potential exchange interaction. This interaction is ferromagnetic,
and according to optical spectra of free Eu1+ ions in the orbital momentum
$L=0$ state (no spin-orbit coupling), ${\cal{J}}_{6s{\text{-}}4f}=52$,
${\cal{J}}_{6p{\text{-}}4f}=33$ and ${\cal{J}}_{5d{\text{-}}4f}=215$ meV
Russell  _et al._ (1941); Dietl  _et al._ (1994), the values in accord with
chemical trends of $spd$–$f$ exchange energies determined for rare earths in
solids Li  _et al._ (1991). Applying the above expression to our data, we
obtain ${\cal{J}}_{\text{eff}}\approx 80$ meV, which indicates that 5$d$ Eu
states contribute to the wave function of the bands around the gap. Therefore,
we propose that the bands undergo a large splitting via $spd-f$ exchange
coupling to Eu spins.

Additionally, there are weak oscillations of the signal below the gap, see Fig
2(d). We assign them to giant Faraday rotation which is induced in our gapped
system, with spin-split electronic bands, due to interband absorption of light
that differs for opposite circular polarizations SM ; Ohnoutek  _et al._
(2016); Bartholomew  _et al._ (1986).

Finally, in the data we see markedly weak lines that blueshift in energy as
the magnetic field increases. The most prominent such line sets in at 300 meV
in zero field. Transmission is tremendously sensitive, which means this lines
comes from a weakly allowed transition, possibly a transition involving spin-
split levels of bound magnetic polarons at residual acceptors. Overall, our
results provide a clear indication that the Eu magnetic sublattice controls
the electronic band structure of the compound via a strong Eu onsite exchange
interaction.

The natural question that follows is whether the observed effects are limited
to the low-temperature AFM phase. Figure 3 shows a series of magneto-
reflectivity color plots taken up to 16 T, at temperatures ranging from 4 to
140 K. In the magneto-reflectivity, we see the same kind of features as in the
magneto-transmission: the band edge redshifting as magnetic field is applied,
and a splitting between the upper and lower band gap edge. The zero-field band
gap increases first, then decreases with increasing temperature. The splitting
can be described by an effective $g$ factor, which drops from 1500 at 4 K to
80 at 140 K SM . At all the temperatures up to 100 K, we see that the bands
become polarized in high magnetic field, even deep within the paramagnetic
phase. This shows that EuCd2As2 is a strong paramagnet, and that qualitatively
there is no difference in its response whether the zero-field state is an AFM
or a paramagnet. Interestingly, at all temperatures, the extracted band edge
as a function of external magnetic field behaves qualitatively very similar to
magnetization, see. Fig. 3(m-n). This means that the Eq. (1) remains valid for
all temperatures and all magnetic fields.

In conclusion, with the full weight of the experimental evidence presented in
this Letter, we show decisively that EuCd2As2 is not a topological semimetal,
but rather a semiconductor with a band gap of 770 meV. The absorption onset
dependence on the magnetic field mimics the shape of magnetization. The band
gap strongly decreases in magnetic fields, but it never becomes inverted, and
the semiconductor nature of the material remains preserved. Nonetheless, the
local Eu magnetic moments are responsible for band structure changes, through
strong 4$f$ exchange coupling to valence and conduction states.

We thank B. Fauqué, M. Müller, D. van der Marel, N. Schröter, A. Grushin and
F. de san Juan for enriching conversations. We are grateful to N. Miller for
the helpful comments. A.A. acknowledges funding from the Swiss National
Science Foundation through project PP00P2_202661. This research was supported
by the NCCR MARVEL, a National Centre of Competence in Research, funded by the
Swiss National Science Foundation (grant number 205602). F.G. and J.H.D.
acknowledge the funding through SNSF grant 200021_200362. M.N. and N.B.
acknowledge the support of CeNIKS project co-financed by the Croatian
Government and the EU through the European Regional Development Fund
Competitiveness and Cohesion Operational Program (Grant No.
KK.01.1.1.02.0013). This work has been supported by the ANR DIRAC3D. We
acknowledge the support of LNCMI-CNRS, a member of the European Magnetic Field
Laboratory (EMFL). This work was supported by the Foundation for Polish
Science through the International Research Agendas program co-financed by the
European Union within the Smart Growth Operational Programme.

## References

  * Liu  _et al._ (2019) D. F. Liu, A. J. Liang, E. K. Liu, Q. N. Xu, Y. W. Li, C. Chen, D. Pei, W. J. Shi, S. K. Mo, P. Dudin, T. Kim, C. Cacho, G. Li, Y. Sun, L. X. Yang, Z. K. Liu, S. S. P. Parkin, C. Felser, and Y. L. Chen, “Magnetic Weyl semimetal phase in a Kagomé crystal,” Science 365, 1282–1285 (2019).
  * Morali  _et al._ (2019) Noam Morali, Rajib Batabyal, Pranab Kumar Nag, Enke Liu, Qiunan Xu, Yan Sun, Binghai Yan, Claudia Felser, Nurit Avraham, and Haim Beidenkopf, “Fermi-arc diversity on surface terminations of the magnetic Weyl semimetal Co3Sn2S2,” Science 365, 1286–1291 (2019).
  * Nie  _et al._ (2022) Simin Nie, Tatsuki Hashimoto, and Fritz B. Prinz, “Magnetic Weyl semimetal in K3Mn3(AsO4)3 with the minimum number of Weyl points,” Physical Review Letters 128, 176401– (2022).
  * Wang  _et al._ (2022) Jin-Feng Wang, Qing-Xin Dong, Zhao-Peng Guo, Meng Lv, Yi-Fei Huang, Jun-Sen Xiang, Zhi-An Ren, Zhi-Jun Wang, Pei-Jie Sun, Gang Li, and Gen-Fu Chen, “NdAlSi: A magnetic Weyl semimetal candidate with rich magnetic phases and atypical transport properties,” Physical Review B 105, 144435– (2022).
  * Kanagaraj  _et al._ (2022) M. Kanagaraj, Jiai Ning, and Liang He, “Topological Co3Sn2S2 magnetic Weyl semimetal: From fundamental understanding to diverse fields of study,” Reviews in Physics 8, 100072 (2022).
  * Ma  _et al._ (2019) J. Z. Ma, S. M. Nie, C. J. Yi, J. Jandke, T. Shang, M. Y. Yao, M. Naamneh, L. Q. Yan, Y. Sun, A. Chikina, V. N. Strocov, M. Medarde, M. Song, Y. M. Xiong, G. Xu, W. Wulfhekel, J. Mesot, M. Reticcioli, C. Franchini, C. Mudry, M. Müller, Y. G. Shi, T. Qian, H. Ding, and M. Shi, “Spin fluctuation induced Weyl semimetal state in the paramagnetic phase of EuCd2As2,” Science Advances 5, eaaw4718 (2019).
  * Xu  _et al._ (2021) Y. Xu, L. Das, J. Z. Ma, C. J. Yi, S. M. Nie, Y. G. Shi, A. Tiwari, S. S. Tsirkin, T. Neupert, M. Medarde, M. Shi, J. Chang, and T. Shang, “Unconventional transverse transport above and below the magnetic transition temperature in Weyl semimetal EuCd2As2,” Physical Review Letters 126, 076602– (2021).
  * Soh  _et al._ (2019) J. R. Soh, F. de Juan, M. G. Vergniory, N. B. M. Schröter, M. C. Rahn, D. Y. Yan, J. Jiang, M. Bristow, P. Reiss, J. N. Blandy, Y. F. Guo, Y. G. Shi, T. K. Kim, A. McCollam, S. H. Simon, Y. Chen, A. I. Coldea, and A. T. Boothroyd, “Ideal Weyl semimetal induced by magnetic exchange,” Physical Review B 100, 201102(R) (2019).
  * Jo  _et al._ (2020) Na Hyun Jo, Brinda Kuthanazhi, Yun Wu, Erik Timmons, Tae-Hoon Kim, Lin Zhou, Lin-Lin Wang, Benjamin G. Ueland, Andriy Palasyuk, Dominic H. Ryan, Robert J. McQueeney, Kyungchan Lee, Benjamin Schrunk, Anton A. Burkov, Ruslan Prozorov, Sergey L. Bud’ko, Adam Kaminski, and Paul C. Canfield, “Manipulating magnetism in the topological semimetal EuCd2As2,” Physical Review B 101, 140402(R) (2020).
  * Taddei  _et al._ (2022) K. M. Taddei, L. Yin, L. D. Sanjeewa, Y. Li, J. Xing, C. dela Cruz, D. Phelan, A. S. Sefat, and D. S. Parker, “Single pair of Weyl nodes in the spin-canted structure of EuCd2As2,” Physical Review B 105, L140401– (2022).
  * Ma  _et al._ (2020) Junzhang Ma, Han Wang, Simin Nie, Changjiang Yi, Yuanfeng Xu, Hang Li, Jasmin Jandke, Wulf Wulfhekel, Yaobo Huang, Damien West, Pierre Richard, Alla Chikina, Vladimir N. Strocov, Joël Mesot, Hongming Weng, Shengbai Zhang, Youguo Shi, Tian Qian, Ming Shi, and Hong Ding, “Emergence of Nontrivial Low-Energy Dirac Fermions in Antiferromagnetic EuCd2As2,” Advanced Materials 32, 1907565 (2020).
  * Soh  _et al._ (2018) J. R. Soh, C. Donnerer, K. M. Hughes, E. Schierle, E. Weschke, D. Prabhakaran, and A. T. Boothroyd, “Magnetic and electronic structure of the layered rare-earth pnictide EuCd2As2 and EuCd2Sb2,” Physical Review B 98, 064419– (2018).
  * Soh  _et al._ (2020) J. R. Soh, E. Schierle, D. Y. Yan, H. Su, D. Prabhakaran, E. Weschke, Y. F. Guo, Y. G. Shi, and A. T. Boothroyd, “Resonant x-ray scattering study of diffuse magnetic scattering from the topological semimetals EuCd2As2 and EuCd2Sb2,” Physical Review B 102, 014408– (2020).
  * Schellenberg  _et al._ (2011) Inga Schellenberg, Ulrike Pfannenschmidt, Matthias Eul, Christian Schwickert, and Rainer Pöttgen, “A 121Sb and 151Eu Mössbauer spectroscopic investigation of EuCd2X2 (X = P, As, Sb) and YbCd2Sb2,” Zeitschrift für anorganische und allgemeine Chemie 637, 1863–1870 (2011).
  * Homes  _et al._ (1993) Christopher C. Homes, M. Reedyk, D. A. Cradles, and T. Timusk, “Technique for measuring the reflectance of irregular, submillimeter-sized samples,” Applied Optics 32, 2976–2983 (1993).
  * Tanner (2015) D. B. Tanner, “Use of x-ray scattering functions in Kramers-Kronig analysis of reflectance,” Physical Review B 91, 035123– (2015).
  * Crepaldi  _et al._ (2017) Alberto Crepaldi, Majed Chergui, Helmuth Berger, Arnaud Magrez, Philippe Bugnon, Frank van Mourik, José Ojeda, Christopher A. Arrell, Gianmarco Gatti, Silvan Roth, and Marco Grioni, “Time-resolved ARPES at LACUS: Band Structure and Ultrafast Electron Dynamics of Solids,” CHIMIA 71, 273 (2017).
  * Ojeda  _et al._ (2015) J. Ojeda, C. A. Arrell, J. Grilj, F. Frassetto, L. Mewes, H. Zhang, F. van Mourik, L. Poletto, and M. Chergui, “Harmonium: A pulse preserving source of monochromatic extreme ultraviolet (30–110 eV) radiation for ultrafast photoelectron spectroscopy of liquids,” Structural Dynamics 3, 023602 (2015).
  * Artmann  _et al._ (1996) A. Artmann, A. Mewis, M. Roepke, and G. Michels, “AM2X2-Verbindungen mit CaAl2Si2-Struktur. XI. Struktur und Eigenschaften der Verbindungen ACd2X2 (A: Eu, Yb; X: P, As, Sb),” Zeitschrift für anorganische und allgemeine Chemie 622, 679–682 (1996).
  * Oliveira  _et al._ (1972) N. F. Oliveira, S. Foner, Y. Shapira, and T. B. Reed, “EuTe. I. Magnetic behavior of insulating and conducting single crystals,” Phys. Rev. B 5, 2634–2646 (1972).
  * Homes, C. C. and Wang, Z. -C. and Fruhling, K. and Tafti, F. (2022) Homes, C. C. and Wang, Z. -C. and Fruhling, K. and Tafti, F., “Optical properties and carrier localization in the layered phosphide EuCd2P2,” https://arxiv.org/abs/2209.10606 (2022).
  * Kuzmenko (2005) A. B. Kuzmenko, “Kramers–Kronig constrained variational analysis of optical spectra,” Review of Scientific Instruments 76, 083108 (2005).
  * (23) In the Supplementary Materials we include additional data to support our work.
  * Wang  _et al._ (2016) H. P. Wang, D. S. Wu, Y. G. Shi, and N. L. Wang, “Anisotropic transport and optical spectroscopy study on antiferromagnetic triangular lattice EuCd2As2: An interplay between magnetism and charge transport properties,” Physical Review B 94, 045112– (2016).
  * Kirchschlager  _et al._ (2004) R. Kirchschlager, W. Heiss, R. T. Lechner, G. Bauer, and G. Springholz, “Hysteresis loops of the energy band gap and effective g factor up to 18’000 for metamagnetic EuSe epilayers,” Applied Physics Letters 85, 67–69 (2004).
  * Schmutz  _et al._ (1978) L. E. Schmutz, G. Dresselhaus, and M. S. Dresselhaus, “Optical absorption of EuTe in high magnetic fields,” Solid State Communications 28, 597–600 (1978).
  * Russell  _et al._ (1941) Henry Norris Russell, Walter Albertson, and Dorothy N. Davis, “The Spark spectrum of Europium, Eu II,” Physical Review 60, 641–656 (1941).
  * Dietl  _et al._ (1994) T. Dietl, C. Śliwa, G. Bauer, and H. Pascher, “Mechanisms of exchange interactions between carriers and Mn or Eu spins in lead chalcogenides,” Physical Review B 49, 2230–2233 (1994).
  * Li  _et al._ (1991) Hong-Shuo Li, Y P Li, and J M D Coey, “R-T and R-R exchange interactions in the rare-earth (R)-transition-metal (T) intermetallics: an evaluation from relativistic atomic calculations,” Journal of Physics: Condensed Matter 3, 7277 (1991).
  * Ohnoutek  _et al._ (2016) L. Ohnoutek, M. Hakl, M. Veis, B. A. Piot, C. Faugeras, G. Martinez, M. V. Yakushev, R. W. Martin, Č. Drašar, A. Materna, G. Strzelecka, A. Hruban, M. Potemski, and M. Orlita, “Strong interband Faraday rotation in 3D topological insulator Bi2Se3,” Scientific Reports 6, 19087 (2016).
  * Bartholomew  _et al._ (1986) D. U. Bartholomew, J. K. Furdyna, and A. K. Ramdas, “Interband Faraday rotation in diluted magnetic semiconductors: Zn1-xMnxTe and Cd1-xMnxTe,” Physical Review B 34, 6943–6950 (1986).