# Finding Islands of Predictability in Action Forecasting

Dan Scarafoni
<EMAIL_ADDRESS>Irfan Essa
<EMAIL_ADDRESS>Thomas Plötz
<EMAIL_ADDRESS>
###### Abstract

We address dense action forecasting: the problem of predicting future action
sequence over long durations based on partial observation. Our key insight is
that future action sequences are more accurately modeled with variable, rather
than one, levels of abstraction, and that the optimal level of abstraction can
be dynamically selected during the prediction process. Our experiments show
that most parts of future action sequences can be predicted confidently in
fine detail only in small segments of future frames, which are effectively
“islands” of high model prediction confidence in a “sea” of uncertainty. We
propose a combination Bayesian neural network and hierarchical convolutional
segmentation model to both accurately predict future actions and optimally
select abstraction levels. We evaluate this approach on standard datasets
against existing state-of-the-art systems and demonstrate that our “islands of
predictability” approach maintains fine-grained action predictions while also
making accurate abstract predictions where systems were previously unable to
do so, and thus results in substantial, monotonic increases in accuracy.

##  1 Introduction

Dense action forecasting aims to understand and anticipate future action
sequences before they occur given an initial observation period. This is vital
for a broad spectrum of problems. Robots collaborating with humans, for
example, must be able to anticipate actions in order to safely work with their
partners [1, 2]. Long term action sequences, in particular, are often very
unpredictable, and predictors must adapt to the fundamental uncertainties of
the video [3, 4, 5, 6]. One solution is to use coarser-grained labels to
“hedge a bet” and predict an action abstraction when a concrete, fine-grained
prediction is impossible. [7].

(a) The activity diagram from which action sequences were sampled for 50
Salads demonstrates that many action sequences are, to a degree, random by
design [8]

(b) We can use uncertainty estimation (1) to find the predictable areas (2)
and predict correct coarse-grained actions elsewhere to improve forecasting
accuracy (3).

We propose a system to find and replace inaccurate, fine-grained future action
sequence predictions with accurate coarse-grained ones. This system extends
uncertainty estimation to dense action forecasting and leverages temporal
convolutional networks to select the correct level of abstraction at each time
step to maintain high prediction accuracy. Our experiments show that most
parts of future action sequences can be predicted confidently in fine detail
only in small segments of future frames, which are effective “islands” of high
model prediction confidence in a “sea” of uncertainty. This we contrast with
existing methods which try to make fine-grained predictions at every point in
the future [9, 3]. In the standard 50 Salads dataset, for example, ingredients
can be chopped in many different orders; however, dressing is almost always
poured on the salad at the end. Thus, predicting what ingredients will be
chopped next is often impossible (see Fig. 1(a)), but it is almost certain
that the dressing will be poured onto the salad near the end of the video.

Our system gives “partial credit” for guesses where correct coarse actions are
substituted for fine. Since the relative value of coarse or fine predictions
varies depending on the specific use case, and so, there is a need to bias the
system towards coarse or fine-grained guesses depending on which preference
will ultimately maximize performance. To this end, we introduce Granularity
Loss, which further tunes predictions to the data and evaluation scheme. We
perform extensive testing across all possible evaluation schemes with standard
metrics on benchmark datasets and demonstrate that our method substantially
improves performance on state-of-the-art systems. Fig. 1(b) demonstrates that
substantial gains in accuracy can be made using our system.
Our primary contributions are (1) a novel system that leverages uncertainty
estimation and hierarchical temporal segmentation to find the predictable
sections (“islands”) of future action sequences and substitute accurate,
coarse-grained predictions where none such exist. (2) a novel loss function,
Granularity Loss, for tuning the system to specific evaluation schemes. (3)
The introduction of uncertainty estimation to the dense action forecasting
problem domain. (4) Extensive experimentation on standard datasets
demonstrates that our method substantially improves dense action forecasting
for state-of-the-art systems.

##  2 Related work

Future action prediction has seen explosive growth in recent years. Our work
deals with dense action forecasting: the prediction of long-term (on the order
of several minutes), action sequences and their durations [10, 3, 11, 12, 13].

Despite significant gains, dense action forecasting remains very difficult,
rarely exceeding $30\%$ accuracy for the longest term predictions on standard
data sets [4, 14, 11]. Some datasets are known to be highly unpredictable and
were designed as such (see Fig. 1(a)) [8, 6]. Recent work has shown that often
coarser-grained actions can be reliably predicted even when fine-grained
actions cannot [7]. This motivates accurate prediction of coarser-grained
activity labels outside of the “islands.” In this work, we posit a method to
estimate such predictable “islands” and make multiple-granularity predictions
to accurately estimate the future.

###  2.1 Uncertainty estimation in video action prediction

Predicting the correctness of forecast actions requires uncertainty
estimation. More generally, uncertainty estimation is an important area of
research in computer vision. Equipping a model with confidence about its
prediction allows a system to understand and cope with sensor and model error.
Classification logits do not necessarily reflect model confidence, and thus
more sophisticated methods are necessary [15]. Generalized methods for
uncertainty estimation in computer vision have made great strides in recent
years, and there have been many attempts to model dense action forecasting
systems with probability distributions [3, 11, 16, 17]; However, no such
techniques exist for dense action forecasting [15, 18]. There is a need to
develop uncertainty estimation for this problem, as prediction confidence is
essential to “hedging bets” as to how precise a prediction should be [7].

##  3 Finding islands of predictability

###  3.1 Problem statement

Given per-frame action predictions for multiple action granularities, our
system selects the ones that both maximize accuracy and granularity. Formally,
given a video of $T$ frames with $t_{0}$ observed frames, $g\in G$ levels of
granularity, and a prediction for each granularity level in data point $n$ at
frame $t$ $\hat{a}^{t,n}_{g}$, the goal is to design a Granularity Selector
system $H$ with parameters $\theta$ which maximizes

$\displaystyle\theta^{*}=arg\,max_{\theta}\sum_{n=1}^{N}\sum_{t=t_{0}^{n}+1}^{T}\mathcal{A}(H(\hat{a}^{t,n}_{1}\dots\hat{a}^{t,n}_{G};\theta),a_{g^{*}}^{t,n},\beta)$
(1)

$\mathcal{A}$ is a weighted accuracy metric [7]. It is parameterized by
$\beta$, which determines the accuracy weight or “partial credit” given to
correct coarser levels of granularity if the fine-grained actions are not
predicted. $H$ is the Granularity Selector which selects a final granularity
for each frame from options at different levels

$\displaystyle H:\hat{a}_{1}\dots\hat{a}_{G}\rightarrow\hat{a}_{g^{*}}$ (2)

###  3.2 Discovering islands of predictability

Figure 2: An overview of our approach. Video features serve as input to the
Prediction Module, which makes future action predictions across multiple
levels of granularity. The probabilities for each action serve as input to the
Granularity Selector, which ultimately decides at what level of abstraction
each frame will be labeled.

We present a two-part system for finding islands of predictability, where we
first a Prediction Module predicts action labels and durations across
different granularity levels, and then a Granularity Selector chooses final
granularities from these. We show an overview of our system in Fig. 2.

The Prediction Module is a dense action forecaster which predicts future
actions over long periods (on the order of several minutes). Formally, given a
series of $t_{0}$ observed frames $x_{1}\dots x_{t_{0}}$, which correspond to
a list of activities $a_{1}\dots a_{n}$, we want to predict the remaining
actions $a_{n+1}\dots a_{N}$ as well as the time lengths for these actions
$\ell_{n+1}\dots\ell_{N}$. The predictions then serve as input to the
Granularity Selector.

Any classifier can have its outputs converted into confidence measurements
with ensembling [15, 18]. We improve upon this notion by designing a novel
Bayesian neural network Prediction Module that outputs confidence scores for
each prediction. This represents the output layer $f(x)$ as a probability
distribution $f(\textbf{x};\theta)\equiv p(\cdot,x)$. The action output for
each class in each step is a Gaussian distribution representing a
probabilistic distribution for each class.

$\displaystyle\hat{a}^{t,n}_{c}\sim N(\mu_{t,c},\sigma_{t,c})~{}t\in T,c\in
C,n\in N$ (3)

where $\hat{a}^{t,n}_{c}$ is the logit value of class $c$ at time step $t$ for
data point $n$. This allows the logits to be treated as true measures of
prediction certainty. Our RNN also must take into account the fact that the
probability distribution at prediction $m$ becomes not only a function of the
input data, but previous predictions as well

$\displaystyle f(\textbf{x},\textbf{m};\theta)\equiv
p(\cdot,x,\hat{a}^{1,n},\dots\hat{a}^{m-1,n})$ (4)

After replacing the output classifier with the Bayesian modification, it can
be trained as a BNN with a simple probabilistic modification to the loss
function [18].

This network is trained not on the outputs of each action step, altered to be
a Dirichlet distribution $Dir(y|\alpha)$ with concentration parameters $\alpha
j>0$, a categorical distribution [19]. Our network is the first of its kind to
allow for confidence measurements of long-term action predictions. Not only is
this essential for selecting what level of granularity to use, but, more
fundamentally, it allows for informed estimations of the predictability of the
future.

The Granularity Selector then selects the final granularity. It is most
important to select fine-grained predictions when the system has high
confidence in its guesses and coarse otherwise. The input of this network is
the predicted mean probabilities for each fine-grained class $c_{f}\in C_{f}$
and each coarse-grained class $c_{c}\in C_{c}$ at each prediction step $o_{t}$
for $t\in[t_{0}+1\dots T]$:

$\displaystyle o_{t}=[\mu_{1_{f}}\dots\mu_{C_{f}}\mu_{1_{c}}\dots\mu_{C_{c}}]$
(5)

Picking hierarchy labels should be as follows: A system should select the
fine-grained label whenever correct, and the coarse label otherwise. Formally,
we choose labels $y$ as such:

$\displaystyle y^{t,n}=\begin{cases}0&\text{if
}\hat{a}_{1}^{t,n}=a^{t,n}_{1}\\\ 1&\text{otherwise}\end{cases}$ (6)

Where $a^{tn}_{1}$ is the ground truth fine-grained label. This allows labels
to be derived quickly and intuitively from any existing dataset and Prediction
Module. This corresponds to action segmentation in that it takes, as input,
video frame features and labels each with an action. Any feed-forward
classifier can be used for the Granularity Selector, but the temporal context
has shown itself to be invaluable in video segmentation [20, 21]. We,
therefore, use a temporal convolutional segmentation network to gather
features over both temporal and hierarchical dimensions to make a more refined
Granularity Selector. To train the network we use standard cross entropy loss:

$\displaystyle\mathcal{L}_{A}(\hat{y},y)=\frac{1}{N}\sum_{n=1}^{N}\frac{1}{T_{n}}\sum_{t=1}^{T_{n}}-log(\hat{y}_{g^{*}}^{n,t})$
(7)

Where $\hat{y}_{g^{*}}^{n,t}$ is the probability for the true granularity.
Ideally, all correct guesses should have high confidence, but, in practice,
the correlation between confidence and accuracy is not perfect. There is,
therefore, a need to bias the system towards coarse guesses or fine guesses
depending on their relative value when such decisions are unclear. To this
end, we introduce the novel Granularity Loss, $\mathcal{L}_{G}$, which
penalizes coarse action selection.

$\displaystyle\mathcal{L}_{G}(\hat{y},y)=\frac{1}{N}\sum_{n=1}^{N}\frac{1}{T_{n}}\sum_{t=1}^{T_{n}}-log(\hat{y}_{g^{*}}^{t,n})I(\hat{y}^{t,n}=0)$
(8)

Where $I$ is the identity function mapping to $1$ when the statement within is
true and 0 otherwise, and $y^{n,t}$ is the true label. $\hat{H}$ is a
variation of $H$ that takes the probability features $O$ and outputs the
correct granularity level

This brings the total optimization function to

$\displaystyle\theta^{*}=arg\,min_{\theta}\mathcal{L}_{A}(\hat{H}(O;\theta),y)+\gamma\mathcal{L}_{G}(\hat{H}(O;\theta),y)$
(9)

where $\gamma$ is the granularity penalty i.e. the additional weight placed on
the fine-grained actions versus coarse.

##  4 Experiments and results

In this section, we will demonstrate the performance gains of this methodology
across multiple action Prediction Modules and Granularity Selector versus
state-of-the-art dense action forecasting.

###  4.1 Datasets and metrics

We evaluate our model on Breakfast and 50 Salads, two leading benchmarks for
action prediction [22, 8]. The Breakfast dataset contains 712 videos, each
several minutes in length and shot with multiple camera angles. Each video
belongs to one out of ten activities, such as making coffee or scrambling
eggs. The frames are annotated with fine-grained labels like “pour cream” and
“grab cup”. There are 48 actions total. For evaluation, we use the standard 4
splits as proposed.

The 50 Salads dataset contains 50 videos showing people preparing salads. The
average duration is 6.4 minutes, and 20 actions per video. In addition, to
start and end action labels, there are 17 fine-grained action labels such as
“cut cucumber” and “add oil.” For evaluation, we use five-fold cross-
validation and report the average where ten videos are held out for a test in
each split. We design coarse label sets for 50 Salads and Breakfast of five
and three labels each. These action sets are designed to be highly predictable
and non-trivial. By finding and substituting inaccurate fine-grained labels
for these accurate, coarse-grained labels, we demonstrate that our methodology
allows for monotonic, often substantial, gains in accuracy, regardless of the
value of substituted coarse labels versus fine.

The standard metric for dense action forecasting is mean over class accuracy
[9]. When coarse granularity is allowed for substitute predictions, it is
generally interpreted as “hierarchical” action recognition or prediction,
where coarser granularity measures are allowed to count as “partial credit”
instead of finer-grained guesses [7, 23]. For our experiments, we parameterize
the weighted accuracy with $\beta$ which represents the weight of coarser-
grained predictions. As with previous work, we experiment with valuing coarse
less than fine grained ($\beta<1$), equal to fine grained ($\beta=1$), and
more than fine grained ($\beta>1$) [7, 24].

###  4.2 Finding islands of predictability for dense action forecasting

Tab 1 shows state of the art dense forecasting performance. Without ground
truth observation labels at test time, few exceed $20\%$ accuracy, and none
can exceed $40\%$ accuracy. This underscores the difficulty of this task. We
note that with access to ground truth observations at test time, prediction
accuracy does increase [14, 9]. However, in real scenarios, these labels will
not be present, and so we argue that it is of paramount importance to focus on
non-ground-truth based systems. Accordingly, we emphasize non-ground-truth-
based experiments in this work.

We demonstrate that our system can identify these inaccurate segments ahead of
time and substitute far more accurate coarse-grained labels in their place to
improve performance. At the same time, we show that our system does not reduce
performance. Thus, we show that finding the islands of predictability leads to
substantial monotonic gains in performance. We show our results in Fig. 3. For
this experiment, we evaluate Granularity Selectors across all values $\beta$
and demonstrate performance over all possible preferences. We demonstrate the
efficacy of three action prediction systems. Bayes is our BNN predictor. Cycle
Consistency is the action predictor from [25]. This system uses segmentation
to extract features and a GRU with cycle consistency to make future action
predictions. Temporal Aggregation implements a multi-granular encoder to make
predictions [10]. Unlike the other baselines, we implement the version that
takes ground-truth labels as input to demonstrate the effectiveness of our
framework on ground-truth input as well as non-ground-truth.

For both baselines, prediction uncertainty is estimated via the ensembling of
three separately trained classifiers. In the graphs, the blue bar shows the
baseline system accuracy, while the green bar represents additional accuracy
gained from the Granularity Selector. Note that as $\beta$ approaches zero,
accuracy approaches the fine-grained accuracy. Similarly, as $\beta$
approaches infinity, it approaches the coarse-grained accuracy. Results
demonstrate that finding islands of predictability adds increased accuracy,
often substantially. This is especially true for benchmark values of $\beta$
(0.5, 1, and 2) [7]. Across four of six systems, our proposed system doubles
at least one accuracy value. The two exceptions, Temporal Aggregation-
Breakfast and Cycle consistency- Breakfast, still have double-digit improved
accuracy scores. The proposed Bayesian network achieves the largest increase
in accuracy, the highest overall accuracy for both datasets, as well as top
accuracy for $\beta=0.5$ on 50 Salads.

In only one experiment, Temporal Aggregation- 50 Salads, does the Granularity
Selector reduce performance slightly. We notice that 50 Salads has a higher
relative improvement from the Granularity Selector, which is to be expected
since its fine-grained labels are more unpredictable and can thus benefit more
from coarse-grained labels [6].

| 50 Salads | Breakfast
---|---|---
System | 20%/50% | 30%/50% | 20%/50% | 30%/50%
RNN [9] | 13.49 | 9.77 | 15.82 | 19.21
CNN [9] | 9.87 | 10.86 | 14.54 | 18.76
Uncertainty aware [5] | 12.82 | 12.31 | 14.20 | 16.86
Time conditioned [26] | 15.99 | 15.59 | 15.84 | 19.75
Cycle consistency [25] | 15.25 | 15.89 | 21.54 | 25.20
Adversarial grammar [11] | 21.2 | 19.8 | N/A | N/A
Temporal Aggregation [10] | 15.57 | 15.5 | 20.85 | 23.61
Bayesian predictor (ours) | 19.71 | 18.69 | 19.21 | 25.1
Attn-GRU [4] | 23.88 | 26.39 | 20.85 | 23.61
MAVAP [27] | 16.1 | 14.8 | 32.9 | 35.5
Table 1: MoC accuracy in previous works on the 50 Salads and Breakfast
datasets are evaluated with standard protocols of observing $20-30\%$ of a
video and predicting $50\%$ of the remaining video. Results never exceed 30%,
a fact which highlights the difficulty of this problem and the inherent
unpredictability of the future. Figure 3: Accuracy improvements via islands of
predictability (green) over state-of-the-art action forecasting (blue).
Results were evaluated on 50 Salads (left) and Breakfast (right). Results show
that hierarchical selection provides additional gains in accuracy across
multiple baselines, with gains and overall performance being greatest for the
proposed (Bayes) system and on the 50 Salads dataset.

###  4.3 Optimizing hierarchy selection

Figure 4: Hierarchy selection results on 50 Salads and Breakfast. Percent
frames guessed coarse are displayed above each bar.

We run experiments to optimize the Granularity Selector given a pretrained
action Prediction Module. As a baseline here, we consider a simple multilayer
perceptron model and contrast it with the temporal convolutional segmentation
model. The results are shown in Fig. 4. Results indicate that the temporal
convolutional network achieves superior performance, which indicates, by
extension, that temporal context is important for hierarchy selection. The
temporal convolutional network tends to heavily favor either coarse or fine-
grained classes. The MLP tends to guess granularity more proportionally with
$\gamma$, however, it is still less accurate.

Figure 5: Heat maps demonstrating the relationship between granularity penalty
($\gamma$), $\beta$, and accuracy score on 50 Salads validation data.
Increased $\gamma$ results in a decreased preference for coarse actions, and
increased fine-grained accuracy, and shows that tuned Granularity Loss is
necessary to optimize hierarchy selection.

###  4.4 The impact of Granularity Loss

To train an effective system, we must understand the relationship between the
performance of that system and Granularity Loss. Fig. 5 shows how accuracy
measurements (coarse, fine, and hierarchical) change with $\beta$ and
$\gamma$. Values for fine-grained accuracy peak when $\gamma$ is maximized
with little influence from $\beta$, and coarse-grained accuracy similarly
peaks when $\gamma$ is minimized. Despite only being an evaluation metric,
$\beta$ has a non-zero impact on weighted accuracy as validation weighted
accuracy is used as a stopping criterion during training. Results indicate
intuitively that accuracy is maximized when the network is trained to have
$\gamma$ proportional to the inverse of the weighted accuracy’s $\beta$ value.
In all experiments, this is the value of $\gamma$ used unless otherwise
stated.

###  4.5 Qualitative results

We analyze qualitative results in Fig. 6. Each shows the ground truth actions
(top), the best possible selection of coarse and fine labels as per Alg. 6
(middle), and hierarchies selected per frame over each value $\gamma$
(bottom). Most of the optimal actions predicted for each frame are coarse-
grained, with “islands” of predictability where fine-grained actions are the
best choice. This trend is not absolute, however, as the second Breakfast
example shows an instance where most of the frames are classified as fine-
grained. Considering that this example is dominated by one long fine-grained
action, it is the result of this action being correctly guessed by the dense
action forecaster.

The bottom bar for each example shows which frames the Granularity Selector
labels as “fine-grained” as $\gamma$ increases; the shade of color determines
at what value $\gamma$ the selector begins selecting the fine-grained label
for the frame. We see that as $\gamma$ increases, the selector classifies more
frames as “fine-grained.” In other words, the “islands” expand until they
occupy the entirety of the predicted sequence. This behavior is similar to
islands in a shallow sea becoming “larger and larger” as water levels recede.
This is important behavior, as the Granularity Loss is designed to encourage
the system to favor fine-grained predictions over coarse at higher values of
$\gamma$.

(a) 50 Salads

(b) Breakfast

Figure 6: Qualitative Results demonstrating ground truth fine-grained actions
(top), best possible distribution of coarse and fine labels (middle), and
hierarchies selected per frame as $\gamma$ increases (bottom). Results
demonstrate not only evidence for the existence of “Islands of Predictability”
in future action sequences, but also that increasing granularity penalty
increases the bias towards classifying frames as fine-grained.

###  4.6 Prediction confidence for dense action forecasting

For uncertainty estimation, we evaluate three novel uncertainty estimation
systems: ensemble, dropout ensemble, and Bayesian. The ensemble method trains
an ensemble of classifiers, dropout repeatedly sampling from logits in a
network with test-time dropout [15].

Method | NLL | NLL MSE
---|---|---
Ensemble | 0.040 | 0.0009
Dropout | 0.033 | 0.0023
Bayesian Network | 0.004 | 0.0046
Table 2: Prediction NLL for each forecasting network, as well as the
difference in NLL to ground truth. The Bayesian network performs best at
minimizing uncertainty in prediction, but the Ensemble provides variance most
similar to the ground-truth data.

We utilize the long-term action prediction protocols from previous works [25]
but use NLL (negative log-likelihood) instead of MoC accuracy as the metric.
NLL is the standard metric for predictor uncertainty, and the smallest
predicted value is best [15, 18, 19]. Measurements were taken from the 20%
observed and 50% predicted experiment to best gauge effectiveness over long
durations. However, because of the inherent unpredictability of the future,
there is arguably a need to also make diverse predictions, and thus not
minimize NLL [3]. We also, therefore, minimize the difference between the
predicted NLL and the per-frame NLL of the entire dataset. This first involves
using linear interpolation to increase the length of all predicted and ground-
truth actions to the length of the longest video. This ensures that all videos
can be compared despite differing lengths. Then the NLL is calculated for each
frame for both the ground truth and predicted sequences. After this, we take
the mean squared error between probability distributions of predictions
($\hat{p_{a}}$) and ground truth ($p_{a}$) over time:

$\displaystyle
MSE_{NLL}=\frac{1}{N}\sum_{n=1}^{N}\frac{1}{T_{n}}\sum_{t=1}^{T_{n}}(\hat{p_{a}}^{n,t}-p_{a}^{n,t})^{2}$
(10)

The results for 50 Salads are shown in Tab. 2. Results indicate that our novel
Bayes network performs the best in terms of minimizing NLL, but that the
ensemble network performs best in terms of minimizing $MSE_{NLL}$. This means
that while the Bayes network optimizes certainty, the ensemble network best
reflects the uncertainty of the ground truth data. Fig. 7 shows the
qualitative results of NLL over time, and shows evidence for the existence of
islands of predictability with drastic dips in predicted NLL.

Figure 7: The NLL over time of select fine-grained 50 Salads (left) and
Breakfast (right) sequences. ”Islands” of predictability are highlighted.

##  5 Conclusion

By estimating the uncertainty of future predictions, we show that future
actions are largely difficult to foresee in fine detail, with “islands of
predictability” punctuating this sea. By finding these islands and
substituting coarse labels in other places, we substantially improve accuracy
versus state-of-the-art systems.

## References

  * [1] Stefano V. Albrecht and Peter Stone.  Autonomous agents modelling other agents: A comprehensive survey and open problems.  Artificial Intelligence, 258:66–95, May 2018.
  * [2] Jamison Heard, Rachel Heald, Caroline E. Harriott, and Julie A. Adams.  A Diagnostic Human Workload Assessment Algorithm for Human-Robot Teams.  In Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pages 123–124, Chicago IL USA, March 2018. ACM.
  * [3] He Zhao and Richard P. Wildes.  On Diverse Asynchronous Activity Anticipation.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, Lecture Notes in Computer Science, pages 781–799, Cham, 2020. Springer International Publishing.
  * [4] Y. B. Ng and B. Fernando.  Forecasting Future Action Sequences With Attention: A New Approach to Weakly Supervised Action Forecasting.  IEEE Transactions on Image Processing, 29:8880–8891, 2020.  Conference Name: IEEE Transactions on Image Processing.
  * [5] Yazan Abu Farha and Juergen Gall.  Uncertainty-Aware Anticipation of Activities.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 1197–1204, Seoul, Korea (South), October 2019. IEEE.
  * [6] Romero Morais, Vuong Le, Truyen Tran, and Svetha Venkatesh.  Learning to Abstract and Predict Human Actions.  arXiv:2008.09234 [cs], August 2020.  arXiv: 2008.09234.
  * [7] Didac Suris, Ruoshi Liu, and Carl Vondrick.  Learning the Predictability of the Future.  pages 12607–12617, 2021.
  * [8] Sebastian Stein and Stephen J. McKenna.  Combining embedded accelerometers with computer vision for recognizing food preparation activities.  In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, UbiComp ’13, pages 729–738, New York, NY, USA, September 2013. Association for Computing Machinery.
  * [9] Yazan Abu Farha, Alexander Richard, and Juergen Gall.  When will you do what? - Anticipating Temporal Occurrences of Activities.  In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 5343–5352, 2018.  ISSN: 10636919.
  * [10] Fadime Sener, Dipika Singhania, and Angela Yao.  Temporal Aggregate Representations for Long-Range Video Understanding.  arXiv:2006.00830 [cs], July 2020.  arXiv: 2006.00830.
  * [11] A. J. Piergiovanni, Anelia Angelova, Alexander Toshev, and Michael S. Ryoo.  Adversarial Generative Grammars for Human Activity Prediction.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, Lecture Notes in Computer Science, pages 507–523, Cham, 2020. Springer International Publishing.
  * [12] Fadime Sener and Angela Yao.  Zero-Shot Anticipation for Instructional Activities.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 862–871, Seoul, Korea (South), October 2019. IEEE.
  * [13] Jiaqi Guan, Ye Yuan, Kris M. Kitani, and Nicholas Rhinehart.  Generative Hybrid Representations for Activity Forecasting With No-Regret Learning.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 170–179, Seattle, WA, USA, June 2020\. IEEE.
  * [14] Harshala Gammulle, Simon Denman, Sridha Sridharan, and Clinton Fookes.  Forecasting Future Action Sequences with Neural Memory Networks.  arXiv:1909.09278 [cs], September 2019.  arXiv: 1909.09278.
  * [15] Yarin Gal and Zoubin Ghahramani.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.  In Proceedings of The 33rd International Conference on Machine Learning, pages 1050–1059. PMLR, June 2016.  ISSN: 1938-7228.
  * [16] Nazanin Mehrasa, Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori.  A Variational Auto-Encoder Model for Stochastic Point Processes.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3160–3169, Long Beach, CA, USA, June 2019. IEEE.
  * [17] Yazan Abu Farha and Juergen Gall.  MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation.  2019\.  arXiv: 1903.01945.
  * [18] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles.  In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [19] Jochen Gast and Stefan Roth.  Lightweight Probabilistic Deep Networks.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3369–3378, Salt Lake City, UT, June 2018. IEEE.
  * [20] Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall.  MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.  Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * [21] Chien Yi Chang, De An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles.  D3TW: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2019-June:3541–3550, 2019.  arXiv: 1901.02598 ISBN: 9781728132938.
  * [22] Hilde Kuehne, Ali Arslan, and Thomas Serre.  The language of actions: Recovering the syntax and semantics of goal-directed human activities.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 780–787, 2014.  ISBN: 9781479951178.
  * [23] Teng Long, Pascal Mettes, Heng Tao Shen, and Cees G. M. Snoek.  Searching for Actions on the Hyperbole.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1138–1147, Seattle, WA, USA, June 2020\. IEEE.
  * [24] Cinna Wu, Mark Tygert, and Yann LeCun.  A hierarchical loss and its problems when classifying non-hierarchically.  PloS One, 14(12):e0226222, 2019.
  * [25] Yazan Abu Farha, Qiuhong Ke, Bernt Schiele, and Juergen Gall.  Long-Term Anticipation of Activities with Cycle Consistency.  arXiv:2009.01142 [cs], September 2020.  arXiv: 2009.01142.
  * [26] Qiuhong Ke, Mario Fritz, and Bernt Schiele.  Time-Conditioned Action Anticipation in One Shot.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9917–9926, Long Beach, CA, USA, June 2019. IEEE.
  * [27] Siyuan Brandon Loh, Debaditya Roy, and Basura Fernando.  Long-Term Action Forecasting Using Multi-Headed Attention-Based Variational Recurrent Neural Networks.  pages 2419–2427, 2022.


#  Anisotropy of the magnetic-field-induced phase pocket in the non-Kramers
doublet system PrIr2Zn20

Shunichiro Kittaka  Department of Physics, Faculty of Science and Engineering,
Chuo University, Bunkyo, Tokyo 112-8551, Japan  Institute for Solid State
Physics, University of Tokyo, Kashiwa, Chiba 277-8581, Japan  Takahiro Onimaru
Department of Quantum Matter, Graduate School of Advanced Science and
Engineering, Hiroshima University, Higashi-Hiroshima, Hiroshima 739-8530,
Japan  Keisuke T. Matsumoto  Graduate School of Science and Engineering, Ehime
University, Matsuyama, Ehime 790-8577, Japan  Toshiro Sakakibara  Institute
for Solid State Physics, University of Tokyo, Kashiwa, Chiba 277-8581, Japan

###### Abstract

We provide thermodynamic evidence for the presence of a magnetic-field-induced
small phase pocket near the antiferro-quadrupole (AFQ) phase boundary in the
non-Kramers $\Gamma_{3}$ doublet system ${\rm PrIr_{2}Zn_{20}}$. In
particular, we measured the specific heat as functions of temperature $T$,
magnetic field $B$, and field angle $\phi_{B}$, and found a second specific-
heat anomaly in a relatively wide field-angle range near $B\parallel[001]$,
although fine tuning of the field strength is required. We also investigated
the rotational magnetocaloric effect and evaluated an entropy change in this
phase pocket. The present findings demonstrate that multipole degrees of
freedom give rise to a magnetic-field-induced exotic order in ${\rm
PrIr_{2}Zn_{20}}$, suggesting the possibility of switching between the order
parameters or emergence of a multiple-$\bm{q}$ order of quadrupoles.

Unveiling novel quantum phases driven by higher-order multipole moments is one
of the central issues in condensed matter physics.Onimaru and Kusunose (2016)
Among various systems, Pr-based compounds having the non-Kramers doublet
$\Gamma_{3}$ ground state in the cubic crystalline electric field (CEF) have
attracted much interest because magnetic dipole moments are absent; only
electric quadrupole $O_{20}$, $O_{22}$, and octupole $T_{xyz}$ moments are
active. For instance, the $\Gamma_{3}$ system PrPb3 exhibits an antiferro-
quadrupole (AFQ) order at $T_{\rm Q}=0.4$ K,Onimaru $et$ $al$. (2005) and
shows a magnetic-field-induced phase transition, likely from the $O_{20}$ to
$O_{22}$ phase, under a magnetic field $B$ along the $[110]$ axis.Onimaru $et$
$al$. (2007a); Sato $et$ $al$. (2010); Onimaru $et$ $al$. (2007b) Thus, the
non-Kramers $\Gamma_{3}$ systems provide great opportunities to study
multipole physics.

Recently, numerous experimental efforts have been made on the Pr$T_{2}X_{20}$
system ($T$: transition metals, $X$: Al, Zn and Cd) which has a cubic
CeCr2Al20-type structure. So far, it has been well established that the CEF
ground state of some Pr$T_{2}X_{20}$ compounds is a non-magnetic $\Gamma_{3}$
doublet, and a wide variety of multipole phenomena have been found in this
system. For instance, PrTi2Al20 exhibits a ferro-quadrupole (FQ) order Sakai
and Nakatsuji (2011) at $T_{\rm FQ}=2$ K and shows a magnetic-field-induced
first-order phase transition driven by change in the FQ order parameter due to
the competition between anisotropic quadrupole interaction and Zeeman
effect.Taniguchi $et$ $al$. (2019); Kittaka $et$ $al$. (2020) In addition,
PrV2Al20 exhibits an AFQ order below 0.6 K,Sakai and Nakatsuji (2011) possibly
accompanied by a double transition related to quadrupole and octupole
orderings;Tsujimoto $et$ $al$. (2014) it has a high-field phase associated
with a rearrangement of quadrupole moments under $B\parallel[100]$.Shimura
$et$ $al$. (2013, 2019) Moreover, ${\rm PrIr_{2}Zn_{20}}$ Onimaru $et$ $al$.
(2011) and PrRh2Zn20 Onimaru $et$ $al$. (2012) show an AFQ order at $T_{\rm
Q}=0.11$ and 0.06 K, respectively. Since these Pr-based materials exhibit
superconductivity within the ordered phase,Sakai $et$ $al$. (2012);
Matsubayashi $et$ $al$. (2012); Tsujimoto $et$ $al$. (2014); Onimaru $et$
$al$. (2010, 2012) multipole fluctuations are expected to play a key role in
mediating Cooper pairs.

In this paper, we focus on the non-Kramers doublet system ${\rm
PrIr_{2}Zn_{20}}$. Low-temperature properties of this material are mainly
dominated by the non-magnetic $\Gamma_{3}$ doublet because the entropy release
estimated from specific-heat measurements reaches $R\ln 2$ [$\sim 5.76$ J/(mol
K)] at 2 K in zero field. Indeed, the energy gap between the ground state of a
$\Gamma_{3}$ doublet and the first-excited state of a $\Gamma_{4}$ triplet is
estimated to be 30 K.Onimaru $et$ $al$. (2011) However, at $T_{\rm Q}=0.11$ K,
the entropy release is only 20% of $R\ln 2$. Since the non-Fermi-liquid (NFL)
nature has been observed above $T_{\rm Q}$, the rest of the entropy is likely
released by the formation of a quadrupole Kondo lattice due to the
hybridization between quadrupoles and conduction electrons.Onimaru $et$ $al$.
(2016) The AFQ phase is destroyed by a magnetic field of 5 T along the $[001]$
axis, whereas it survives at higher magnetic fields along the $[111]$ and
$[110]$ axes.Ishii $et$ $al$. (2011) The propagation vector was revealed to be
$\bm{k}=(1/2,1/2,1/2)$ from neutron diffraction measurements in magnetic
fields of $B\parallel[110]$,Iwasa $et$ $al$. (2017) indicating that the
$O_{22}$-type order parameter is dominant. Although a possible second
transition was observed in the field range $1.5\ {\rm T}\leq B\leq 3$ T for
$B\parallel[001]$ in the earlier specific-heat study,Onimaru $et$ $al$. (2011)
it was not detected in the recent one.Onimaru $et$ $al$. (2016) Therefore, it
was interpreted to be due to an extrinsic effect.Onimaru $et$ $al$. (2016)
However, in the present work, we have performed field-angle-resolved
measurements of the specific heat and revealed that the double transition is
intrinsic and strongly depends on the field strength and its orientation.

High-quality single crystals of ${\rm PrIr_{2}Zn_{20}}$ were grown by the
melt-growth method.Onimaru $et$ $al$. (2010) A single-crystalline sample with
its mass of 1.24 mg was used in this study. The specific heat $C$ was measured
by the standard quasi-adiabatic heat-pulse method using a homemade calorimeter
in a dilution refrigerator (Oxford, Kelvinox AST Minisorb). The sample was
attached on the stage of our calorimeter so that the $[1\bar{1}0]$ axis is
roughly along the vertical direction. The magnetic field $B$ was applied by
using a vector magnet which generates up to 7 T (3 T) along a horizontal
(vertical $z$) direction. The orientation of the magnetic field can be
controlled in three dimensions by rotating the refrigerator around the $z$
axis using a stepping motor. In this study, we precisely applied the magnetic
field along the $(1\bar{1}0)$ plane, including the $[001]$, $[112]$, $[111]$,
and $[110]$ axes. From the field-angle dependence of its specific heat, it was
found that the $[1\bar{1}0]$ axis of the present sample is tilted away from
the $z$ direction by roughly $10^{\circ}$ due to the non-rectangular sample
shape. Therefore, we tuned the horizontal and vertical magnetic fields at each
field angle so that the magnetic field orients to the target direction with
high accuracy of better than $0.1^{\circ}$. The relative change in the entropy
with rotating the magnetic field was investigated using the rotational
magnetocaloric technique.Kittaka $et$ $al$. (2018)

Figure 1:  (a) Temperature dependence of the $4f$ contribution to the specific
heat of ${\rm PrIr_{2}Zn_{20}}$, $C_{4f}/T$, in various magnetic fields
applied along the $[001]$ direction. Each set of the data is shifted
vertically by 100 J/(mol K2) for clarity. (b), (c) Field dependence of
$C_{4f}/T$ at 0.08 and 0.09 K along the (b) $[001]$ and (c) $[110]$ axes.

Figure 1(a) shows the temperature dependence of the $4f$ contribution to the
specific-heat data, $(C-C_{\rm N})/T$, hereafter referred to as $C_{4f}/T$,
measured under a magnetic field applied along the cubic $[001]$ axis. Here,
the nuclear contribution to the specific heat, $C_{\rm N}$, is calculated by
using the Hamiltonian of a nuclear spin of a Pr nucleus ($I=5/2$) [see details
in the Supplemental Material (SM) of Ref. Kittaka $et$ $al$., 2020]. For
simplicity, in this calculation, the site-averaged magnitude of the Pr
magnetic moment is assumed to be $m_{\rm Pr}(B)=0.202B$ $\mu_{\rm B}/$Pr below
6 T;Onimaru $et$ $al$. (2016) the field-angle dependence of $m_{\rm Pr}(B)$ is
neglected although the actual magnetization is weakly anisotropic.Onimaru $et$
$al$. (2011) In zero field, the present sample exhibits a sharp specific-heat
peak at $T_{\rm Q}=0.125$ K. This peak is much sharper and $T_{\rm Q}$ is
slightly higher compared with the previous reports Onimaru $et$ $al$. (2011,
2016) [see SM (I) SM ].

With increasing the magnetic field along the $[001]$ axis, $T_{\rm Q}$
decreases monotonically up to 2 T, consistent with the previous report.Onimaru
$et$ $al$. (2016) In the magnetic-field range between 2 and 3 T, $T_{\rm Q}$
remains approximately 0.11 K, independent of the magnetic-field strength.
However, the specific-heat peak at $T_{\rm Q}$ is suppressed with increasing
$B$, and it is smeared out around 3 T. Instead, a second specific-heat anomaly
develops at a slightly higher temperature of $T_{\rm Q}^{\rm h}\sim 0.12$ K
above 2 T. These two specific-heat anomalies clearly coexist in the field
range 2.2 T $\lesssim B\lesssim$ 2.7 T, as shown in Fig. 1(a). This second
peak at $T_{\rm Q}^{\rm h}$ becomes most remarkable at 3.1 T and becomes
broader at higher $B$.

In the earlier specific-heat study,Onimaru $et$ $al$. (2011) a similar double-
peak structure was observed in the field range $1\lesssim B\lesssim 3$ T using
a sample whose specific heat shows a relatively broad transition at $T_{\rm
Q}$ of 0.11 K in zero field. However, it was not detected in a more recent
study,Onimaru $et$ $al$. (2016) which reports the temperature dependence of
the specific heat at 1 T intervals, using a different sample with a slightly
higher $T_{\rm Q}$ [see SM (I) SM ]. These facts imply that the double-peak
structure depends on the sample quality and that fine tuning of the magnetic
field is essential to observe it.

Figure 1(b) shows the field dependence of $C_{4f}/T$ for $B\parallel[001]$ at
0.08 and 0.09 K. Above 2 T, the low-temperature $C_{4f}(B)$ increases
remarkably with $B$ up to 4 T. A clear peak at $B\sim 4$ T indicates the
development of an exotic Fermi-liquid (FL) state, as reported
previously.Onimaru $et$ $al$. (2016) A kink anomaly at 3 T might be related to
the disappearance of the specific-heat peak at $T_{\rm Q}$; a corresponding
anomaly was likely to have been observed in the elastic moduli $C_{11}$,Ishii
$et$ $al$. (2011) the resistivity Ikeura $et$ $al$. (2014) and its coefficient
$A$.Onimaru $et$ $al$. (2016) No prominent anomaly was found in $C_{4f}(B)$
for $B\parallel[110]$, as shown in Fig. 1(c).

Figure 2:  Contour plots of $C_{4f}/T$ in the $B$–$T$ plane for magnetic-field
orientations along the (a) $[001]$, (b) $[112]$, (c) $[111]$, and (d) $[110]$
axes. Crosses (squares) represent the peak or shoulder anomalies observed in
$C_{4f}(T)/T$ [$C_{4f}(B)$].

The $B$–$T$ phase diagram for $B\parallel[001]$, along with a contour plot of
$C_{4f}(B,T)/T$, is shown in Fig. 2(a). Here, the specific-heat anomalies
found in the temperature scan are represented by crosses. The presence of a
magnetic-field-induced phase pocket, hereafter referred to as “the A phase”,
is suggested by the double-peak structure in the $C(T)$ data. This A phase
occurs at higher temperatures than $T_{\rm Q}(B)$. It is unclear whether or
not $T_{\rm Q}(B)$ and $T_{\rm Q}^{\rm h}(B)$ lines merge because either
specific-heat anomaly is smeared out near possible merging points.

Temperature dependences of $C_{4f}/T$ in different field orientations for
$B\parallel[112]$, $[111]$, and $[110]$ have also been investigated [see SM
(II) SM ]. Figures 2(b)-(d) display contour plots of $C_{4f}/T$ in the $B$–$T$
plane for $B\parallel[112]$, $[111]$, and $[110]$, respectively. Whereas the
magnetic-field-induced double transition can be seen for $B\parallel[112]$ and
$[111]$, it is absent in $B\parallel[110]$ at least below 6 T.

Figure 3:  (a) Field-angle dependence of $C_{4f}/T$ at 0.12 K under a magnetic
field rotated within the $(1\bar{1}0)$ plane. Numbers labeling the curves
represent the rotating magnetic field in Tesla. Each set of the data is
shifted vertically by 150 J/(mol K2) for clarity. (b) Contour plot of
$C_{4f}/T$ at 0.12 K in the $B$–$\phi_{B}$ plane. Circles in (b) show the
location of the peak anomalies represented by triangles in (a).

To clarify the field-orientation dependence of the A phase, we measured
$C_{4f}$ at several magnetic fields rotated within the $(1\bar{1}0)$ plane.
The results of $C_{4f}(\phi_{B})$ at 0.12 K are shown in Fig. 3(a), where
$\phi_{B}$ is the field angle measured from the $[001]$ axis within the
$(1\bar{1}0)$ plane. At 3 and 3.5 T, two peaks can be clearly seen in
$C_{4f}(\phi_{B})$. Although the peak intensity is significantly suppressed
above 4 T, the double-peak feature can be confirmed even at 5 T. These peaks
are represented by circles in a contour plot of $C_{4f}(B,\phi_{B})/T$ at 0.12
K [Fig. 3(b)]. The A phase exists in a relatively wide $\phi_{B}$ range and
shifts toward higher magnetic fields with increasing $\phi_{B}$ from
$0^{\circ}$. This symmetric field-angle dependence eliminates the possibility
that the second transition is caused by a domain with a tilted crystalline
axis in the sample.

Figure 4:  The relative change in the entropy measured from the value at 0.15
K, $\Delta S_{4f}(T)$, at 0.15, 0.14, 0.13, 0.12, 0.11, 0.10, and 0.09 K (from
top to bottom), plotted as a function of $B$ along the (a) $[001]$, (b)
$[112]$, (c) $[111]$, and (d) $[110]$ axes.

In order to obtain the entropy information, the temperature dependence of the
entropy relative to the value at 0.15 K was estimated as

$\Delta S_{4f}(T)=S_{4f}(T)-S_{4f}(0.15\ {\rm K})=-\int_{T}^{0.15\ {\rm
K}}\frac{C_{4f}(T)}{T}dT$ (1)

for each magnetic field [see SM (II) SM ]. Then, the data points of $\Delta
S_{4f}$ at several selected temperatures were extracted and plotted as a
function of $B$, $\Delta S_{4f}(B)$, in Figs. 4(a)-4(d) for each field
orientation. From $\Delta S_{4f}(B)=S_{4f}(T^{\ast},B)-S_{4f}(0.15\ {\rm
K},B)$ at a selected temperature $T^{\ast}$, we can detect the field variation
of $S_{4f}(T^{\ast},B)$ itself when $S_{4f}(0.15\ {\rm K},B)$ is independent
of $B$. In the previous study,Onimaru $et$ $al$. (2016) it was revealed that,
for $B\parallel[001]$, $S_{4f}(0.15\ {\rm K},B)$ is nearly unchanged up to 3
T, but it is suppressed above 5 T due to the increase in the characteristic
temperature $T_{0}$, which is related to the formation of the quadrupole Kondo
lattice. Because $T_{0}$ estimated from resistivity measurements is nearly
constant up to 4 T for $B\parallel[001]$,Onimaru $et$ $al$. (2016) one might
assume that the present base entropy $S_{4f}(0.15\ {\rm K},B)$ is independent
of $B$ up to 4 T for $B\parallel[001]$.

At 0.11-0.12 K, an apparent $\Delta S_{4f}(B)$ plateau was observed in the
field range $2\ {\rm T}\lesssim B\lesssim 3$ T, bounded by the two arrows in
Fig. 4(a); $\Delta S_{4f}(B)$ for $B\parallel[001]$ at 0.11 K exhibits rapid
enhancements at the boundaries. In addition, the magnetic-field-induced
transition becomes sharpest at 3.1 T for $B\parallel[001]$ [Fig. 1(a)], and
the specific heat anomalies disappear where the phase transition terminates.
Further investigations, such as high-resolution magnetostriction Wörl $et$
$al$. (2019); Kittaka $et$ $al$. (2023) and magnetization measurements with
fine tuning of temperature, are required to clarify whether the phase
transition to the A phase is of first order.

In the previous reports,Onimaru $et$ $al$. (2011, 2016) the entropy release
below $T_{\rm Q}$ was estimated to be 2 J/(mol K) in zero field. As presented
in Figs. 4(a)-4(d), $|\Delta S_{4f}(B)|$ at 0.09 K (well below $T_{\rm Q}$)
remains roughly 2 J/(mol K) up to 3 T in any field orientation; no prominent
entropy change was observed around 2 T for $B\parallel[001]$, at which the
elastic modulus $C_{11}$ (Ref. Ishii $et$ $al$., 2011) and $C_{4f}(B)$ [Fig.
1(b)] show an anomaly. This fact supports the absence of the phase boundary at
2 T or tiny entropy change at this possible phase boundary.

Figure 5:  Field-angle dependences of (a), (b) the rotational magnetocaloric
effect $(\partial T/\partial\phi_{B})_{S}$, (c), (d) the relative change in
the entropy $\Delta S_{\phi}$, and (e), (f) the specific heat $C$ under a
magnetic field of 3 and 6 T, respectively, rotated within the $(1\bar{1}0)$
plane at 0.12 K. Here, $C$ and $\Delta S_{\phi}$ contain nuclear contribution.

To clarify the field-angle dependence of the entropy, the rotational
magnetocaloric effect, $(\partial T/\partial\phi_{B})_{S}$, was investigated
at 0.12 K under a magnetic field of 3 and 6 T rotated within the $(1\bar{1}0)$
plane; the results are shown in Figs. 5(a) and 5(b), respectively. Combined
with the $C(\phi_{B})$ data at 0.12 K in Figs. 5(e) and 5(f), the $\phi_{B}$
dependence of the relative change in the entropy has been investigated as

$\Delta
S_{\phi}=S(\phi_{B})-S(90^{\circ})=-\int_{90^{\circ}}^{\phi_{B}}\frac{C}{T}\biggl{(}\frac{\partial
T}{\partial\phi_{B}}\biggl{)}_{S}d\phi_{B}.$ (2)

The results at 3 and 6 T are shown in Figs. 5(c) and 5(d), respectively. The
$\Delta S_{4f}(B)$ plateau in the A phase can be again confirmed from $\Delta
S_{\phi}$ at 0.12 K and 3 T in the angle range
$10^{\circ}\lesssim\phi_{B}\lesssim 30^{\circ}$ [Fig 5(c)]. The phase
transition from the NFL state to the A phase at 0.12 K reduces the entropy by
only 0.07 J/(mol K), much less than the entropy decrease of roughly 0.5 J/(mol
K) in the phase transition from the A phase to the AFQ phase. These results
along with the $\Delta S_{\phi}$ plateau suggest that the ground state of the
A phase is well separated from the first-excited state and still has a
substantial entropy.

At 0.12 K and 6 T, $\Delta S_{\phi}$ reaches its largest value of 0.48 J/(mol
K) at $\phi_{B}\sim 40^{\circ}$, which is comparable to the $\Delta S_{\phi}$
value in the A phase at 3 T. Therefore, this peak may also be related to the
occurrence of the A phase. However, at high magnetic fields, it is difficult
to identify the boundary of the A phase from $C(\phi_{B})$ and $S(\phi_{B})$,
because $T_{0}$ drastically depends on the field angle.

Finally, let us discuss possible origins for the A phase in ${\rm
PrIr_{2}Zn_{20}}$. A first possible scenario is switching of the AFQ order
parameters from $O_{22}$ to $O_{20}$. Indeed, the competition between
quadrupole interaction and the Zeeman effect can lead to switching of order
parameters, as predicted from theoretical studies.Hattori and Tsunetsugu
(2014, 2016) However, it is still uncertain whether this switching can occur
only in the high-temperature region; usual phase transitions understood in
terms of a change in the order parameter, as in the case of PrPb3, are
accompanied by a change of the ground state at 0 K. Such a high-temperature
phase pocket without changing the ground state at 0 K is unusual for the
$\Gamma_{3}$ doublet system. A second possible scenario is the occurrence of
an exotic multipole ordering such as multiple-$\bm{q}$ orders of quadrupoles.
The magnetic-field-induced phase pocket in ${\rm PrIr_{2}Zn_{20}}$ is
reminiscent of the skyrmion-lattice phase (so-called A phase) in MnSi, which
is induced under a magnetic field in a narrow temperature and magnetic-field
range just below the helimagnetic transition temperature. In the A phase of
MnSi, a superposition of three helical states, the so-called triple-$\bm{q}$
state, was observed via small-angle neutron scattering experiments.Mühlbauer
$et$ $al$. (2009) Similar two specific-heat anomalies were reported in MnSi as
well.Bauer $et$ $al$. (2013) Not only in the non-centrosymmetric systems,
skyrmion-lattice phases have also been found in the centrosymmetric lattice
systems.Spachmann $et$ $al$. (2021); Hirschberger $et$ $al$. (2019); Yasui
$et$ $al$. (2020); Ishiwata $et$ $al$. (2020) On theoretical grounds, rich
phase diagrams in the $\Gamma_{3}$ doublet system, such as triple-$\bm{q}$
orders of multipoles Tsunetsugu $et$ $al$. (2021); Ishitobi and Hattori (2021)
and the composite state,Inui and Motome (2020) have been proposed recently.
Further investigations, such as neutron-scattering experiments and nuclear-
magnetic-resonance measurements, are essential to elucidate the nature of the
A phase in ${\rm PrIr_{2}Zn_{20}}$.

In summary, we have provided thermodynamic evidence for the presence of a
magnetic-field-induced multipole ordered phase (so-called the A phase) in
${\rm PrIr_{2}Zn_{20}}$. The A phase occurs only in the high-temperature
region and depends strongly on the magnetic-field orientation. Such a high-
temperature phase pocket is novel in the $\Gamma_{3}$ doublet system and paves
the path to further understanding of the wide variety of multipole orders.

###### Acknowledgements.

We thank K. Hattori for useful discussion. This work was supported by KAKENHI
(23H04868, 23H04870, 23H01128, 18H04306) from JSPS.

## References

  * Onimaru and Kusunose (2016) T. Onimaru and H. Kusunose, J. Phys. Soc. Jpn. 85, 082002 (2016).
  * Onimaru $et$ $al$. (2005) T. Onimaru, T. Sakakibara, N. Aso, H. Yoshizawa, H. S. Suzuki and T. Takeuchi, Phys. Rev. Lett. 94, 197201 (2005).
  * Onimaru $et$ $al$. (2007a) T. Onimaru, T. Sakakibara, A. Harita, T. Tayama, D. Aoiki and Y. $\bar{\mathrm{O}}$nuki, J. Phys. Soc. Jpn. 73, 2377 (2007a).
  * Sato $et$ $al$. (2010) Y. Sato, H. Morodomi, K. Ienaga, Y. Inagaki, T. Kawae, H. S. Suzuki and T. Onimaru, J. Phys. Soc. Jpn. 79, 093708 (2010).
  * Onimaru $et$ $al$. (2007b) T. Onimaru, N. Aso, K. Prokes, H. S. Suzuki, T. J. Sato and T. Sakakibara, J. Phys. Chem. Solids 68, 2091 (2007b).
  * Sakai and Nakatsuji (2011) A. Sakai and S. Nakatsuji, J. Phys. Soc. Jpn. 80, 063701 (2011).
  * Taniguchi $et$ $al$. (2019) T. Taniguchi, K. Hattori, M. Yoshida, H. Takeda, S. Nakamura, T. Sakakibara, M. Tsujimoto, A. Sakai, Y. Matsumoto, S. Nakatsuji and M. Takigawa, J. Phys. Soc. Jpn. 88, 084707 (2019).
  * Kittaka $et$ $al$. (2020) S. Kittaka, T. Taniguchi, K. Hattori, S. Nakamura, T. Sakakibara, M. Takigawa, M. Tsujimoto, A. Sakai, Y. Matsumoto and S. Nakatsuji, J. Phys. Soc. Jpn. 89, 043701 (2020).
  * Tsujimoto $et$ $al$. (2014) M. Tsujimoto, Y. Matsumoto, T. Tomita, A. Sakai and S. Nakatsuji, Phys. Rev. Lett. 113, 267001 (2014).
  * Shimura $et$ $al$. (2013) Y. Shimura, Y. Ohta, T. Sakakibara, A. Sakai and S. Nakatsuji, J. Phys. Soc. Jpn. 82, 043705 (2013).
  * Shimura $et$ $al$. (2019) Y. Shimura, Q. Zhang, B. Zeng, D. Rhodes, R. Schonemann, M. Tsujimoto, Y. Matsumoto, A. Sakai, T. Sakakibara, K. Araki, W. Zheng, Q. Zhou, L. Balicas and S. Nakatsuji, Phys. Rev. Lett. 122, 256601 (2019).
  * Onimaru $et$ $al$. (2011) T. Onimaru, K. T. Matsumoto, Y. F. Inoue, K. Umeo, T. Sakakibara, Y. Karaki, M. Kubota and T. Takabatake, Phys. Rev. Lett. 106, 177001 (2011).
  * Onimaru $et$ $al$. (2012) T. Onimaru, N. Nagasawa, K. T. Matsumoto, K. Wakiya, K. Umeo, S. Kittaka, T. Sakakibara, Y. Matsushita and T. Takabatake, Phys. Rev. B 86, 184426 (2012).
  * Sakai $et$ $al$. (2012) A. Sakai, K. Kuga and S. Nakatsuji, J. Phys. Soc. Jpn. 81, 083702 (2012).
  * Matsubayashi $et$ $al$. (2012) K. Matsubayashi, T. Tanaka, A. Sakai, S. Nakatsuji, Y. Kubo and Y. Uwatoko, Phys. Rev. Lett. 109, 187004 (2012).
  * Onimaru $et$ $al$. (2010) T. Onimaru, K. T. Matsumoto, Y. F. Inoue, K. Umeo, Y. Saiga, Y. Matsushita, R. Tamura, K. Nishimoto, I. Ishii, T. Suzuki and T. Takabatake, J. Phys. Soc. Jpn. 79, 033704 (2010).
  * Onimaru $et$ $al$. (2016) T. Onimaru, K. Izawa, K. T. Matsumoto, T. Yoshida, Y. Machida, T. Ikeura, K. Wakiya, K. Umeo, S. Kittaka, K. Araki, T. Sakakibara and T. Takabatake, Phys. Rev. B 94, 075134 (2016).
  * Ishii $et$ $al$. (2011) I. Ishii, H. Muneshige, Y. Suetomi, T. K. Fujita, T. Onimaru, K. T. Matsumoto, T. Takabatake, K. Araki, M. Akatsu, Y. Nemoto, T. Goto and T. Suzuki, J. Phys. Soc. Jpn. 80, 093601 (2011).
  * Iwasa $et$ $al$. (2017) K. Iwasa, K. T. Matsumoto, T. Onimaru, T. Takabatake, J.-M. Mignot and A. Gukasov, Phys. Rev. B 95, 155106 (2017).
  * Kittaka $et$ $al$. (2018) S. Kittaka, S. Nakamura, H. Kadowaki, H. Takatsu and T. Sakakibara, J. Phys. Soc. Jpn. 87, 073601 (2018).
  * (21) (Supplemental Material) (I) Sample dependence of the specific heat, and (II) entropy analysis are provided online.
  * Ikeura $et$ $al$. (2014) T. Ikeura, T. Matsubara, Y. Machida, K. Izawa, N. Nagasawa, K. T. Matsumoto and T. Onimaru, JPS Conf. Proc. 3, 011091 (2014).
  * Wörl $et$ $al$. (2019) A. Wörl, T. Onimaru, Y. Tokiwa, Y. Yamane, K. T. Matsumoto, T. Takabatake and P. Gegenwart, Phys. Rev. B 99, 081117(R) (2019).
  * Kittaka $et$ $al$. (2023) S. Kittaka, Y. Kono, K. Tsunashima, D. Kimoto, M. Yokoyama, Y. Shimizu, T. Sakakibara, M. Yamashita and K. Machida, Phys. Rev. B 107, L220505 (2023).
  * Hattori and Tsunetsugu (2014) K. Hattori and H. Tsunetsugu, J. Phys. Soc. Jpn. 83, 034709 (2014).
  * Hattori and Tsunetsugu (2016) K. Hattori and H. Tsunetsugu, J. Phys. Soc. Jpn. 85, 094001 (2016).
  * Mühlbauer $et$ $al$. (2009) S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii and P. Böni, Science 323, 915 (2009).
  * Bauer $et$ $al$. (2013) A. Bauer, M. Garst and C. Pfleiderer, Phys. Rev. Lett. 110, 177207 (2013).
  * Spachmann $et$ $al$. (2021) S. Spachmann, A. Elghandour, M. Frontzek, W. Löser and R. Klingeler, Phys. Rev. B 103, 184424 (2021).
  * Hirschberger $et$ $al$. (2019) M. Hirschberger, T. Nakajima, S. Gao, L. Peng, A. Kikkawa, T. Kurumaji, M. Kriener, Y. Yamasaki, H. Sagayama, H. Nakao, K. Ohishi, K. Kakurai, Y. Taguchi, X. Yu, T. hisa Arima and Y. Tokura, Nat. Commun. 10, 5831 (2019).
  * Yasui $et$ $al$. (2020) Y. Yasui, C. J. Butler, N. D. Khanh, S. Hayami, T. Nomoto, T. Hanaguri, Y. Motome, R. Arita, T. hisa Arima, Y. Tokura and S. Seki, Nat. Commun. 11, 5925 (2020).
  * Ishiwata $et$ $al$. (2020) S. Ishiwata, T. Nakajima, J.-H. Kim, D. S. Inosov, N. Kanazawa, J. S. White, J. L. Gavilano, R. Georgii, K. M. Seemann, G. Brandl, P. Manuel, D. D. Khalyavin, S. Seki, Y. Tokunaga, M. Kinoshita, Y. W. Long, Y. Kaneko, Y. Taguchi, T. Arima, B. Keimer and Y. Tokura, Phys. Rev. B 101, 134406 (2020).
  * Tsunetsugu $et$ $al$. (2021) H. Tsunetsugu, T. Ishitobi and K. Hattori, J. Phys. Soc. Jpn. 90, 043701 (2021).
  * Ishitobi and Hattori (2021) T. Ishitobi and K. Hattori, Phys. Rev. B 104, L241110 (2021).
  * Inui and Motome (2020) K. Inui and Y. Motome, Phys. Rev. B 102, 155126 (2020).

Supplemental Material for
Anisotropy of the magnetic-field-induced phase pocket in the non-Kramers
doublet system ${\rm PrIr_{2}Zn_{20}}$
Shunichiro Kittaka,1,2 Takahiro Onimaru3, Keisuke T. Matsumoto4, and Toshiro
Sakakibara2

1Department of Physics, Faculty of Science and Engineering, Chuo University,
Bunkyo, Tokyo 112-8551, Japan
2Institute for Solid State Physics, University of Tokyo, Kashiwa, Chiba
277-8581
3Department of Quantum Matter, Graduate School of Advanced Science and
Engineering,
Hiroshima University, Higashi-Hiroshima, Hiroshima 739-8530, Japan
4Graduate School of Science and Engineering, Ehime University, Matsuyama,
Ehime 790-8577, Japan
(Dated: )

## I. Sample dependence of the specific heat

Figure S1(a) compares the specific-heat data of three samples of ${\rm
PrIr_{2}Zn_{20}}$ in zero field. The data of the sample 1 were obtained in the
present study whereas the data of the samples 2 and 3 were taken from Refs.
Onimaru et al., 2016 and Onimaru et al., 2011, respectively. It has been
demonstrated that the antiferro-quadrupole (AFQ) transition temperature
$T_{\rm Q}$ of ${\rm PrIr_{2}Zn_{20}}$ depends on the sample quality. Clearly,
the present sample has the highest $T_{\rm Q}$ of 0.125 K. Moreover, the peak
height of the sample 1 is two times larger than the other two. The sample 2
has $T_{\rm Q}$, almost the same as $T_{\rm Q}$ of the sample 3, but exhibits
a specific-heat peak slightly larger than that of the sample 3. Therefore, on
the basis of $T_{\rm Q}$ and the specific-heat peak, it is possible to
consider that the sample 1 has the highest quality among the three, and the
sample 2 has slightly better quality than the sample 3, although the residual
resistivity ratio (RRR) of the sample 1 is comparable to that of the sample 2
(RRR $\sim 100$).

Figure S1(b) compares the specific-heat data of the three samples at 2 T for
$B\parallel[001]$. Here, the nuclear specific heat $C_{\rm N}$ was not
subtracted in the data represented by closed symbols. The low-temperature
upturn in $C(T)/T$ at 2 T disappears by subtracting the nuclear contribution,
as exemplified in Fig. S1(b) by open circles for the sample 1. Thus, the
nuclear specific heat is not significantly large compared with the specific-
heat anomaly associated with the AFQ transition. At 2 T, only the sample 3
shows a double transition, which remained observed in the relatively wide
field range $1.5\ {\rm T}\lesssim B\lesssim 3$ T.Onimaru et al. (2011) This
study revealed that a double transition can also be seen for the sample 1, but
in the narrower field range $2.2\ {\rm T}\lesssim B\lesssim 2.7$ T. Therefore,
at least for the samples 1 and 3, the small phase pocket exists, although the
field range is slightly different. The presence of the small phase pocket in
the sample 2 is unclear because specific-heat measurements were not performed
in the key field range $2\ {\rm T}<B<3$ T. In any case, the magnetic-field-
induced small phase pocket depends on the sample quality.

## II. Entropy analysis

Figures S2(a) shows the temperature dependence of the $4f$ contribution to the
specific heat $C_{4f}$ ($=C-C_{\rm N}$) of the sample 1 divided by
temperature, $C_{4f}/T$, at several magnetic fields for $B\parallel[001]$. The
relative change of the $4f$ contribution to the entropy can be estimated by
integrating $C_{4f}/T$ as

$\Delta S_{4f}(T)=S_{4f}(T)-S_{4f}(0.15\ {\rm K})=-\int_{T}^{0.15\ {\rm
K}}\frac{C_{4f}(T)}{T}dT.$ (3)

Here, $\Delta S_{4f}$ is measured from the value at $T=0.15$ K. Temperature
dependence of $\Delta S_{4f}$ for $B\parallel[001]$ is presented in Fig.
S2(b). In the same manner, temperature dependences of $C_{4f}/T$ and $\Delta
S_{4f}$ for $B\parallel[112]$, $[111]$, and $[110]$ are shown in Figs. S3, S4,
and S5, respectively. In these figures, the entropy change occurs mainly due
to the AFQ transition, except for the data above 4 T in $B\parallel[001]$, at
which the AFQ transition is significantly suppressed and a characteristic
temperature $T_{0}$ is enhanced remarkably.

## References

  * Onimaru et al. (2016) T. Onimaru, K. Izawa, K. T. Matsumoto, T. Yoshida, Y. Machida, T. Ikeura, K. Wakiya, K. Umeo, S. Kittaka, K. Araki, T. Sakakibara, and T. Takabatake, Phys. Rev. B 94, 075134 (2016).
  * Onimaru et al. (2011) T. Onimaru, K. T. Matsumoto, Y. F. Inoue, K. Umeo, T. Sakakibara, Y. Karaki, M. Kubota, and T. Takabatake, Phys. Rev. Lett. 106, 177001 (2011).

Figure S1:  Temperature dependences of the specific heat divided by
temperature, $C/T$, for the samples 1, 2, and 3 at (a) 0 and (b) 2 T for
$B\parallel[001]$. The data of the samples 2 and 3 were taken from Refs.
Onimaru et al., 2016 and Onimaru et al., 2011, respectively. Open symbols in
(b) represent the data after subtracting the nuclear contribution for the
sample 1.

Figure S2:  Temperature dependences of (a) $C_{4f}/T$ and (b) $\Delta S_{4f}$
for the sample 1 at several magnetic fields for $B\parallel[001]$.

Figure S3:  Temperature dependences of (a) $C_{4f}/T$ and (b) $\Delta S_{4f}$
for the sample 1 at several magnetic fields for $B\parallel[112]$.

Figure S4:  Temperature dependences of (a) $C_{4f}/T$ and (b) $\Delta S_{4f}$
for the sample 1 at several magnetic fields for $B\parallel[111]$.

Figure S5:  Temperature dependences of (a) $C_{4f}/T$ and (b) $\Delta S_{4f}$
for the sample 1 at several magnetic fields for $B\parallel[110]$.