# Hopping Too Late: Exploring the Limitations of
Large Language Models on Multi-Hop Queries

Eden Biran1 Daniela Gottesman1 Sohee Yang2 Mor Geva1 Amir Globerson1,3
1Tel Aviv University 2UCL 3Google Research
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs) can solve complex multi-step problems, but little
is known about how these computations are implemented internally. Motivated by
this, we study how LLMs answer multi-hop queries such as _“The spouse of the
performer of Imagine is”_. These queries require two information extraction
steps: a latent one for resolving the first hop (_“the performer of Imagine”_)
into the bridge entity (John Lennon), and one for resolving the second hop
(_“the spouse of John Lennon”_) into the target entity (Yoko Ono).
Understanding how the latent step is computed internally is key to
understanding the overall computation. By carefully analyzing the internal
computations of transformer-based LLMs, we discover that the bridge entity is
resolved in the early layers of the model. Then, _only_ after this resolution,
the two-hop query is solved in the later layers. Because the second hop
commences in later layers, there could be cases where these layers no longer
encode the necessary knowledge for correctly predicting the answer. Motivated
by this, we propose a novel “back-patching” analysis method whereby a hidden
representation from a later layer is patched back to an earlier layer. We find
that in up to 57% of previously incorrect cases there exists a back-patch that
results in the correct generation of the answer, showing that the later layers
indeed sometimes lack the needed functionality. Overall our methods and
findings open further opportunities for understanding and improving latent
reasoning in transformer-based LLMs.

Hopping Too Late: Exploring the Limitations of
Large Language Models on Multi-Hop Queries

Eden Biran1 Daniela Gottesman1 Sohee Yang2 Mor Geva1 Amir Globerson1,3 1Tel
Aviv University 2UCL 3Google Research<EMAIL_ADDRESS>

##  1 Introduction

Figure 1: An illustration of our findings: we observe evidence of latent
reasoning in two-hop queries where 1) During the early layers the first hop is
resolved and the source entity Imagine now encodes the bridge entity John
Lennon 2) During the middle layers critical information propagates to the last
position 3) During the later layers the second hop is resolved and the last
token now encodes the target entity Yoko Ono. We additionally illustrate back-
patching: patching a hidden representation from a later layer back into an
earlier layer in order to fix cases where the pathway fails.

Despite groundbreaking performance in a multitude of tasks, large language
models (LLMs) still struggle with complex knowledge queries (Press et al.,
2023; Dziri et al., 2023). For example, LLMs often err when asked to complete
multi-hop queries such as _“The spouse of the performer of Imagine is”_.
Answering such queries requires composition and reasoning skills, which have
been the focus of many recent works Hou et al. (2023); Brinkmann et al.
(2024); Yang et al. (2024); Li et al. (2024a); Wang et al. (2024); Petty et
al. (2024). Besides the general interest in compositional skills, the
investigation of latent multi-hop abilities of LLMs would also have
significant implications for areas such as generalization (Lake and Baroni,
2018; Onoe et al., 2023) and model editing (Zhong et al., 2023; Cohen et al.,
2024).

To strengthen the latent reasoning ability of LLMs, it is essential to first
understand the internal mechanism of how LLMs successfully complete two-hop
queries using latent multi-hop reasoning. While there have been several works
that investigate such a mechanism, a precise latent reasoning pathway has only
been found in relatively small language models trained on synthetic datasets
(Brinkmann et al., 2024; Li et al., 2024a; Wang et al., 2024) and has not been
thoroughly investigated in large pretrained models.

Throughout this work we consider and compare two complementing settings, one
where the model correctly answers the two-hop query and the other when it does
not. To this end, in §2, we start by creating and publishing a dataset
containing 82,020 two-hop queries based on data from Wikidata (Vrandečić and
Krötzsch, 2014).

Our approach to analyzing multi-hop reasoning begins with the hypothesis that
it involves activating the same “knowledge-extraction module” twice: once for
the first hop (_“the performer of Imagine is John Lennon”_) and once for the
second (_“the spouse of John Lennon is Yoko Ono”_). The key question we ask in
this work is where are these two procedures implemented in the LLM. A natural
way to answer this question is to seek the first point in the network where
the outputs of these two steps appear. The typical approach to this would
employ vocabulary projections (nostalgebraist, 2020; Geva et al., 2021;
Sakarvadia et al., 2023; Yang et al., 2024; Li et al., 2024a). However, we
instead opt for a more recent method named Patchscopes (Ghandeharioun et al.,
2024) which has clear advantages over the commonly used vocabulary projection.

Using this approach, in §3 we find that the outcome of the first hop is
clearly evident in the hidden representation of the end-token of the first
hop. We further find that this resolving happens during the early layers, as
depicted in Figure 1 (stage 1). Given the localization of the first hop, in §4
we then ask where the second hop is resolved. We find that it typically
appears in the last token of the prompt _only_ in layers after the resolution
of the first hop, as further illustrated in Figure 1 (stage 3).

These observations suggest a mechanism where the first hop is resolved with
the answer, that then propagates to the last token to resolve the second hop
(this is depicted by stage 2 in Figure 1). Moreover, they suggest why this
process may fail — If the two resolutions are done by different groups of
layers, it must be that these two groups are both able to perform knowledge
extraction, be it via their attention or MLP sublayers. However, since
transformers have a limited amount of layers, and their functionality is
different (Meng et al., 2022; Geva et al., 2023), it is quite likely that the
layer at which the second hop is performed will not have the desired
functionality. To verify this hypothesis, in §5, we propose an analysis method
named back-patching. The crux of the method involves taking a hidden
representation from a later layer, patching it back into the same position of
the same prompt in an earlier layer, and then continuing the forward pass. In
a way, this allows the model more layers to finish the computation without the
need for additional parameters or training (Petty et al., 2024). Unlike
previous methods (Sakarvadia et al., 2023; Li et al., 2024a), back-patching
makes no assumption on the structure of the prompt and does not require any
knowledge of the participating entities. Testing back-patching on queries
initially predicted incorrectly, we find that up to 57% of incorrect cases
have the potential to be correctly predicted using back-patching when choosing
the optimal source and target layers.

Overall, our contributions can be summarized as follows:

  * •

We provide a novel dataset of two-hop queries, which can serve to
systematically study this important setting.

  * •

We employ Patchscopes to inspect entities encoded in hidden representations
while processing two-hop queries.

  * •

We identify a sequential latent reasoning pathway in LLMs, where the first hop
query is initially resolved into the bridge entity which is then used to
answer the second hop.

  * •

We propose an analysis method named back-patching, that verifies our results
and could additionally be used to improve the performance of LLMs on multi-hop
question answering.

We release our code and dataset at
https://github.com/edenbiran/HoppingTooLate.

##  2 Experimental Setup

###  2.1 Two-Hop Queries

We consider facts denoted by a triplet $(e,r,e^{\prime})$ where $e$ is a
source entity (e.g., John Lennon), $r$ is a relation (e.g., Spouse) and
$e^{\prime}$ is a target entity (e.g., Yoko Ono). These facts can then be
converted to factual statements (e.g., _“The spouse of John Lennon is Yoko
Ono”_) and also into queries by omitting $e^{\prime}$ (e.g., _“The spouse of
John Lennon is ”_). We then prompt models with these queries in order to test
their knowledge and reasoning.

We create two-hop queries by composing two facts where one’s target entity is
the other’s source entity. Namely, $((e_{1},r_{1},e_{2})$ and
$(e_{2},r_{2},e_{3}))$. We refer to $e_{1}$ as the source entity, $e_{2}$ as
the bridge entity, and $e_{3}$ as the target entity. For example, the two-hop
query _“The spouse of the performer of Imagine is”_ can be decomposed into
(Imagine, Performer, John Lennon) and (John Lennon, Spouse, Yoko Ono).

We further define two tokens of specific interest: the last token of
$e_{1}$111This token is also the last token of the first hop. (e.g., _“The
spouse of the performer of Imagine”_) and the last token of the whole two-hop
prompt (e.g., _“The spouse of the performer of Imagine is”_), which we denote
as $t_{1}$ and $t_{2}$, respectively.

###  2.2 Dataset

We create a dataset containing 82,020 two-hop queries based on data from
Wikidata (Vrandečić and Krötzsch, 2014). First, we sample entities with a
maximal amount of Wikidata statements from a set of predefined Wikidata entity
types. These entities will act as $e_{2}$. Second, we use a predefined set of
relations in order to construct the two triplets $(e_{1},r_{1},e_{2})$ and
$(e_{2},r_{2},e_{3})$. Finally we convert both triplets into a single natural
language phrase using manually crafted templates per relation. We refer to our
codebase for further details including entity types and relations222All data
was retrieved during April 2024..

We next attempt to filter out cases where no latent reasoning is performed.
Given a two hop query $((e_{1},r_{1},e_{2}),(e_{2},r_{2},e_{3}))$ we test two
prompts constructed to detect and filter out cases where the model performs
reasoning shortcuts (Xu et al., 2022; Wang et al., 2023a; Ju et al., 2024).
The first prompt is the query $((\text{""},r_{1},e_{2}),(e_{2},r_{2},e_{3}))$
(i.e., the query without $e_{1}$), aimed at filtering out cases where the
model predicts generally popular entities. The second prompt is
$((e_{1},\text{""},e_{2}),(e_{2},r_{2},e_{3}))$ (i.e., the query without
$r_{1}$), aimed at filtering out cases where the model predicts correctly due
to a high correlation between $e_{1}$ and $e_{3}$. For example, given the two-
hop query _“The spouse of the performer of Imagine is”_ we filter out cases
where the model predicts Yoko Ono for either _“The spouse of the performer
is”_ or _“The spouse of Imagine is”_. We perform this filtering for each model
using greedy decoding creating a per model subset of the dataset. Table 1
displays the amount of examples left after this filtering process.

Model | Post Filtering | Cases Tested
---|---|---
Correct | Incorrect
LLaMA 2 7B | $70,625$ | $388$ | $351$
LLaMA 2 13B | $70,972$ | $554$ | $413$
LLaMA 3 8B | $71,569$ | $379$ | $371$
LLaMA 3 70B | $70,334$ | $656$ | $595$
Table 1: The amount of examples post shortcut filtering and the final amount
used throughout all experiments.

We are specifically interested in understanding the differences between when
the model completes the two-hop query correctly and incorrectly. Therefore we
generate two data subsets for these cases accordingly.

The first subset is made up of cases where the model correctly answers both
the two-hop query and the first hop. For example, given the two-hop query
_“The spouse of the performer of Imagine is”_ , we verify that the model
correctly predicts the answer Yoko Ono when prompted with the two-hop query
and additionally correctly predicts John Lennon when prompted with the first
hop _“The performer of Imagine is”_. We use the latter filter to make sure
that the model indeed “knows” the answer to the first hop. We then randomly
sample 100 examples of each bridge entity type, in order to create a more
balanced subset (if less than 100 examples of a specific type exist we take
them all).

The second subset includes cases where the model correctly answers both the
first and second hop in isolation, but fails to answer the full two-hop query.
As an example for the filtering procedure, given the same query above, we
verify that the model correctly predicts John Lennon for the first hop _“The
performer of Imagine is”_ and Yoko Ono for the second hop _“The spouse of John
Lennon is”_ , but fails to predict Yoko Ono for the two-hop query _“The spouse
of the performer of Imagine is”_. We focus on these cases, because we are
interested to understand why answering the two-hop query fails despite the
model “knowing” the two separate facts. As we show later, this failure is
likely due to the first fact being resolved too late. In a similar fashion to
the first setting, we then randomly sample 50 examples of each bridge entity
type (in order to keep the correct and incorrect subsets of approximately the
same size).

Table 1 reports the amount of examples used in experiments by model and
settings.

###  2.3 Models

We analyze the following LLaMA models: LLaMA 2 7B and 13B (Touvron et al.,
2023) and LLaMA 3 8B and 70B (Meta AI, 2024). The 7B and 8B models have 32
layers, while the 13B model has 40 layers and the 70B model has 80 layers.

##  3 Localizing First Hop Resolution

One could imagine multiple possible ways for a model to solve multi-hop
queries. One such way would be to effectively treat the two-hop query as a
single relation, enabling the extraction of the final answer without the need
to recall $e_{2}$. Despite this, if the model truly is performing latent
reasoning, the most intuitive way to do so would start by resolving the first
hop at some point within the computation. In order to localize where this
happens, we use the Patchscopes analysis method. Our results show that the
model indeed resolves the first hop during the early layers, at the position
of $t_{1}$.

Additionally, as the model’s prediction is emitted from the last position of
the prompt, critical information regarding the now-resolved bridge entity must
propagate to this token. We further explore and validate this process in
Appendix B.

###  3.1 Interpreting Hidden Representations Using Patchscopes

We use the Patchscopes framework (Ghandeharioun et al., 2024) to create a task
that describes the entity encoded in a specific hidden representation.
Patchscopes maps a given representation to a sentence in natural language,
thus considerably extending vocabulary projections which map to a single token
(nostalgebraist, 2020).

The procedure is as follows. First, given a source prompt, a source token and
a source layer, the source prompt is passed through the model’s forward
computation and the hidden representation $v$ of the source token at the
source layer is recorded. This representation $v$ is what we aim to probe in
search for an encoded entity. Second, we pass the same prompt used by
Ghandeharioun et al. (2024): ‘‘Syria: Syria is a country in the Middle East,
Leonardo DiCaprio: Leonardo DiCaprio is an American actor, Samsung: Samsung is
a South Korean multinational corporation, x’’ through the model, replacing the
hidden representation of ‘‘x’’ with $v$ at a specific target layer. Finally,
the forward pass is continued and text is generated.

The chosen target prompt encourages the model to generate a continuation that
states the semantic content (i.e., the entity name) encoded in the source
hidden representation, along with a short description of it. We find this task
is better fitting than other proposed probes such as vocabulary projections
(nostalgebraist, 2020; Geva et al., 2022) or training linear classifiers on
hidden representations (Belinkov and Glass, 2019; Belinkov, 2022). This is
because exhibiting the ability to extract information from a hidden
representation gives reason to believe that the same information can be
extracted when answering the two-hop query. In addition Patchscopes has the
clear advantage of decoding a representation into a natural language
description.

###  3.2 Experiment

We run Patchscopes on $t_{1}$, recording the hidden representation at each
source layer and then patching it into all target layers. Following each patch
we sample three generations. For each source layer we check whether one of the
generations is of $e_{2}$, and if so, we consider the representation to encode
the bridge entity at the source layer.

###  3.3 Results

Model | Setting |  $t_{1}$ encodes $e_{2}$ |  $t_{2}$ encodes $e_{3}$
---|---|---|---
Cases | Layer | Cases | Layer
LLaMA 2 7B | Correct | $56.1\%$ | $8.9$ | $73.4\%$ | $16.2$
Incorrect | $41.8\%$ | $9.1$ | $47.0\%$ | $17.5$
LLaMA 2 13B | Correct | $49.2\%$ | $7.7$ | $71.1\%$ | $16.9$
Incorrect | $48.6\%$ | $7.4$ | $33.4\%$ | $16.9$
LLaMA 3 8B | Correct | $46.4\%$ | $8.9$ | $74.4\%$ | $13.5$
Incorrect | $46.9\%$ | $11.9$ | $40.4\%$ | $17.2$
LLaMA 3 70B | Correct | $63.5\%$ | $27.1$ | $86.7\%$ | $30.7$
Incorrect | $57.8\%$ | $29.2$ | $50.0\%$ | $35.9$
Table 2: The results of Patchscopes executions. The table reports the
percentage of cases where a target entity was successfully decoded from a
source position. The table additionally reports the mean of the first layer
where the target entity was successfully decoded. Figure 2: Percentage of
cases per layer where target entities were first successfully decoded using
Patchscopes. The percentages are out of all correctly answered cases for LLaMA
2 13B. Figure 3: Heat-map of the layers where the entity description task
successfully decodes $e_{2}$ from the position of $t_{1}$. The percentages are
out of all successful decodings for LLaMA 2 13B run on correctly answered
cases.

Table 2 present the percentage of cases $e_{2}$ was observed in the hidden
representation of $t_{1}$. We refer to these as cases where $e_{2}$ was
successfully resolved. We note the non-trivial percentages of 41%-63% across
all models and settings. Additionally, we note the general drop in resolved
cases when the model fails to correctly complete the two-hop query, but that
this setting still displays a surprisingly high success rate. Next, we examine
the mean layer at which $e_{2}$ first appears in both correct and incorrect
cases. We find that in the incorrect cases the resolving generally occurs at
higher layers compared to those of the correct cases. This difference alludes
to how early the resolving happens potentially playing a part in successfully
answering two-hop queries. We further verify this using back-patching in §5.

Figure 2 displays the proportion of cases per layer where $e_{2}$ is first
successfully decoded from $t_{1}$ using Patchscopes. We observe that the
resolving occurs almost exclusively during the early layers. Interestingly,
this implies that the knowledge required for this resolution must reside in
these earlier layers. Similar figures for additional models are provided in
Appendix C. The plot additionally shows the curve regarding the resolving of
$e_{3}$ from $t_{2}$, which is discussed in §4.

This finding can further be seen in Figure 3, which displays a heat-map of the
layers where Patchscopes successfully decodes $e_{2}$ from $t_{1}$. Two hot-
spots are apparent. The first corresponds to success cases with an early
source layer, showing that $e_{2}$ generally appears at this position during
the early layers. The second corresponds to later source layers that display
success almost exclusively when patched after the final layer, which is
roughly equivalent to vocabulary projection (nostalgebraist, 2020). This is a
clear advantage of Patchscopes, as despite the entity not being among the top
tokens in the projection during early layers, it is in fact encoded in the
hidden representation. See Appendix D for heat-maps of all models.

##  4 Second Hop is Resolved at Last Position

In §3, we observe that the first hop is often resolved in position $t_{1}$.
Intuitively, we’d expect the computation to use this result towards resolving
the second hop (i.e., the final answer). In this section, we study this by
finding where $e_{3}$ first emerges and how it relates to the first emergence
of $e_{2}$. We find that $e_{3}$ indeed emerges predominantly after $e_{2}$ at
the position of $t_{2}$. Furthermore, we search for the final piece of the
pathway, the predicted token. By projecting the residual updates made by the
attention and MLP sublayers on the hidden representation of $t_{2}$, we find
that the MLP sublayers play the larger part in conjuring the predicted token.

###  4.1 Method

Previous works (Geva et al., 2022, 2023) have shown that the residual updates
made by the MLP and attention sublayers can be interpreted by projecting them
to the vocabulary using the layer norm and output embedding matrix. We use
this method in order to locate specific entities in the flow of information by
observing the rank of the entity in the vocabulary projection. We additionally
use Patchscopes as presented in §3.

###  4.2 Experiment

We first employ Patchscopes on $t_{2}$, this time checking if the target
entity $e_{3}$ was encoded at this position. Additionally, we project the
sublayer updates performed on $t_{2}$, and check if the token with the highest
probability in the projection is the first token of the generated text. If so,
we conclude that the update of a particular sublayer promotes the final
prediction.

###  4.3 Results

Model | Setting | Attention | MLP
---|---|---|---
Cases | Layer | Cases | Layer
LLaMA 2 7B | Correct | $25.5\%$ | $24.7$ | $33.2\%$ | $25.2$
Incorrect | $14.8\%$ | $23.4$ | $28.4\%$ | $26.0$
LLaMA 2 13B | Correct | $28.8\%$ | $32.0$ | $68.5\%$ | $33.1$
Incorrect | $14.2\%$ | $27.0$ | $51.5\%$ | $31.7$
LLaMA 3 8B | Correct | $9.7\%$ | $26.7$ | $17.6\%$ | $27.8$
Incorrect | $25.3\%$ | $27.4$ | $11.8\%$ | $24.6$
LLaMA 3 70B | Correct | $21.9\%$ | $56.3$ | $48.1\%$ | $68.0$
Incorrect | $21.8\%$ | $65.7$ | $36.1\%$ | $67.0$
Table 3: Percentages and mean layers where for the first time the predicted
token is the most probable token in the vocabulary projection of the sublayer
update.

Table 2 contains the percentage of cases $e_{3}$ was decoded from $t_{2}$,
effectively resolving the second hop. As expected, these percentages are high
for cases the model predicted correctly (71%-86%)333We attribute the gap from
100% to the approximations involved in decoding a hidden representation. and
low for cases the model predicted incorrectly (33%-50%). Additionally, it is
apparent from both Table 2 and Figure 2 that the target entity is resolved by
the mid-upper layers at the position of $t_{2}$. This suggests that the
information required to perform the second hop exists in the parameters of
these layers.

Inspecting the sublayer projections in Table 3, we find that the MLP sublayers
play a larger role in promoting the predicted token compared to that of the
attention sublayers. Despite this, the non negligible percentages achieved by
the attention sublayers could suggest multiple pathways, in agreement with
previous work (Geva et al., 2023; Merullo et al., 2023) that considered both
possibilities. Looking at the mean layers at which the prediction is first
promoted, one can see that these promotions occur in the upper layers, after
the last token has been resolved to $e_{3}$.

Figure 4: A comparison of the first layers of each stage in the pathway
between correct and incorrect cases for LLaMA 3 8B.

After describing all major pathway stages, we turn to a comparison between
correct and incorrect cases and observe the following pattern. Generally, in
incorrect cases, the entities seem to be resolved later while the information
seems to propagate and be extracted earlier. Figure 4 displays this pattern in
LLaMA 3 8B, plotting the first observed layers of each stage (we refer to
Appendix E for plots of additional models). This gives evidence of the
importance of how early the first hop is resolved.

##  5 Back-patching Improves Two-Hop Performance

(a) Back-patching $t_{1}$

(b) Back-patching $t_{2}$

Figure 5: Heat-map of the layers where back-patching succeeds. The percentages
are out of all successful back-patching instances for LLaMA 2 13B.

Our results consistently display a sequential nature of computation, where the
first hop is resolved into the bridge entity by the lower layers, which is
then used to answer the second hop in the upper layers. These results suggest
that when answering two-hop queries fail, it could be because the first-hop is
resolved at layers that no longer contain the information needed to resolve
the second hop. In order to verify this hypothesis, we test the following
intriguing conjecture. We know that the earlier layers certainly contain
knowledge, since they are regularly used to resolve the first hop. Then, we
ask what will happen if we take a hidden representation from a later layer and
“feed” it into these earlier layers. Will the correct information now be
extracted? As our results next show, this is often the case. This “back-
patching” analysis thus provides strong support that the success in two hops
requires the later layers to have the same functionality as earlier ones.

Overall, this could point to an inherent limitation of the transformer
architecture, where the model “runs out” of layers, propagates and extracts
information before it is fully resolved, or is forced to use only the upper
layers for knowledge extraction.

###  5.1 Method

Following our intuition above, we perform the following back-patching
experiment. Given a prompt, we first record a hidden representation $v$ from a
source layer $\ell_{s}$. Then, while rerunning the same prompt, we patch $v$
into the same position at a target layer $\ell_{t}$ such that
$\ell_{t}<\ell_{s}$. The forward pass then continues and text is generated
using greedy decoding. This effectively feeds $\ell_{t}$ with information
encoded in $v$ that originated in $\ell_{s}$.

###  5.2 Experiment

We perform the back-patching experiment twice, once on the token $t_{1}$ and
once on the token $t_{2}$. In order to quantify the potential of back-
patching, we check whether one of source-target pairs resulted in the
generation of the correct answer. We then perform an analysis of the
successful source and target layers.

###  5.3 Results

Model | Setting | $t_{1}$ | $t_{2}$
---|---|---|---
LLaMA 2 7B | Correct | $100\%$ | $100\%$
Incorrect | $41.02\%$ | $42.45\%$
LLaMA 2 13B | Correct | $100\%$ | $100\%$
Incorrect | $32.44\%$ | $36.07\%$
LLaMA 3 8B | Correct | $100\%$ | $100\%$
Incorrect | $38.81\%$ | $47.16\%$
LLaMA 3 70B | Correct | $100\%$ | $100\%$
Incorrect | $57.31\%$ | $57.81\%$
Table 4: Back-patching success rates by setting and back-patched token.

Table 4 presents the back-patching results. First we note the high success
rates, correctly answering between 32%-57% of incorrect cases while keeping
the correct cases at 100% across models and token positions. This means that
in many cases rerunning specific parts of the computation is enough in order
to successfully extract the target information. On a more practical note, this
also shows that significant gains can be made by correctly choosing the source
and target layers, which we leave for future work.

One can additionally observe that back-patching $t_{2}$ achieves a higher
success rate than back-patching $t_{1}$, which may point at the stages related
to the last position of the prompt as being the more dominant failure point in
answering two-hop queries. Moreover, the fact that back-patching $t_{2}$ does
not assume any information about the structure of the prompt shows an
additional strength of the method.

Figure 5(a) displays the heat-map of the layers where back-patching $t_{1}$
succeeds for LLaMA 2 13B (see Appendix F for additional models). We first note
the higher success rate when the source layers are the model’s lower layers.
These lower layers correlate to the resolving of the first hop, therefore
back-patching them allows the model a better chance at solving the first hop.
Second, we note the non existent success rate of the higher target layers.
Unsurprisingly, in this case the model does not gain enough time to correctly
complete the pathway, resulting in a low success rate.

Figure 5(b) displays the heat-map of the layers where back-patching $t_{2}$
succeeds for LLaMA 2 13B. Unlike when back-patching $t_{1}$, in this case the
higher success rate can be found in the the upper source layers. These layers
correlate to the resolving of the second hop at the last token position, hence
back-patching the hidden representation taken from them allows the model a
better chance at solving the second hop. Furthermore, we note the higher
success rate in lower target layers. This allows the model to gain time to
resolve the second hop. This also gives the model access to parametric
knowledge that might only be encoded in the lower layers, allowing the model
to fully utilize its whole knowledge-base.

##  6 Related Work

Tracing and locating factual knowledge has been the focus of many recent works
aiming to undercover and control the way transformers make predictions.
Several of these works are motivated by knowledge editing (De Cao et al.,
2021; Mitchell et al., 2022; Meng et al., 2022; Zhang et al., 2024) while
others focus on understanding the inner workings of transformers (Dai et al.,
2022; Geva et al., 2022, 2023). Our work follows and expands on this second
group by extending them to a more complex case, two-hop queries.

The question whether models can implicitly reason over stored parametric
knowledge has also gained recent popularity. One avenue of related research
includes circuit discovery (Nanda et al., 2023; Wang et al., 2023b; Conmy et
al., 2023; Brinkmann et al., 2024), but these works focus mainly on synthetic
tasks. More closely related to this paper, recent works (Sakarvadia et al.,
2023; Yang et al., 2024; Li et al., 2024a) have found existential evidence of
latent reasoning in multi-hop queries, albeit without accounting for the flow
of information throughout layers and positions. In a concurrent work by Wang
et al. (2024) a multi-hop reasoning circuit similar to ours was discovered in
a small transformer language model trained on synthetic data until grokking
occurs. Our work finds further proof of such latent reasoning in a more
practical setting that includes large language models, real world knowledge
and a regular pretraining setup. We additionally use different mechanistic
interpretability methods combined with an understanding of the information
flow during the forward pass.

Even without fully identifying the mechanism behind latent reasoning, several
works have tried to address models shortcomings. One such work by Sakarvadia
et al. (2023) assumes knowledge of the bridge entity, which is then injected
directly into the computation. Likewise, recent work by Li et al. (2024a) also
assumes knowledge of the bridge entity and uses knowledge editing techniques
to improve performance, which is known to have potentially disruptive effects
on models (Zhang et al., 2024; Hsueh et al., 2024; Li et al., 2024b). Both
these previously proposed methods have drawbacks, which a potential method
based on back-patching would not share.

##  7 Conclusion

Our work uses mechanistic tools to study the latent reasoning abilities of
LLMs on two hop queries. We observe strong evidence of a sequential latent
reasoning pathway in which the second hop is answered in the mid-upper layers
only after the first hop is answered by the lower layers. We find that this
sequential nature could point to a possible limitation of transformers, where
the second hop must be answered using only the knowledge encoded in the upper
layers. To further validate this, we introduce and evaluate an analysis method
named back-patching, based on extracting hidden representations from later
layers and patching them back into earlier layers. Finally, we show that back-
patching significantly improves performance on previously incorrect queries.
Overall, our methods and findings open opportunities for understanding and
improving latent reasoning in LLMs.

## Limitations

Several of our experiments rely on mechanistic methods that decode hidden
representations and residual updates in various ways. While these methods have
been widely used in many recent works (Geva et al., 2022, 2023; Dar et al.,
2023; Yang et al., 2024; Li et al., 2024a), they can only be seen as an
approximation. However, using multiple different techniques (such as both
Patchscopes and sublayer projections) helps alleviate this concern.
Additionally, our experiments are generally designed to give positive signals
that result in lower bound results, hence we believe that these approximations
do not undermine our findings.

In this work we study one prominent latent reasoning pathway, although others
most likely exist McGrath et al. (2023); Yang et al. (2024). Additionally, we
do not account for all possible parts of the discovered pathway (e.g., how the
relations come in to play). Despite this, by focusing on the most relevant
components we find sufficient proof for our main finding regarding the
sequential nature of the computation.

Our work examines only two-hop queries. While we expect our methods and
conclusions to hold for queries with three or more hops, and even for
unrelated cases of latent reasoning, we leave these areas for future work.

Despite the promising results of back-patching, our current proposed method is
not a practical inference method, as only a subset of back-patches generate
the correct answer. However, if one could efficiently select a subset of
source and target layers a priori, this could make for a viable method.
Although we believe that if achieving the best performance in multi-hop
question answering is the sole goal, methods such as chain of thought
prompting (Wei et al., 2022; Press et al., 2023) would be far more effective.

## Acknowledgements

This work was supported by the Tel Aviv University Center for AI and Data
Science (TAD) and the Israeli Science Foundation.

## References

  * Bau (2024) David Bau. 2024.  Baukit.
  * Belinkov (2022) Yonatan Belinkov. 2022.  Probing classifiers: Promises, shortcomings, and advances.  _Computational Linguistics_ , 48(1):207–219.
  * Belinkov and Glass (2019) Yonatan Belinkov and James Glass. 2019.  Analysis methods in neural language processing: A survey.  _Transactions of the Association for Computational Linguistics_ , 7:49–72.
  * Brinkmann et al. (2024) Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. 2024.  A mechanistic analysis of a transformer trained on a symbolic multi-step reasoning task.  _arXiv preprint arXiv:2402.11917_.
  * Cohen et al. (2024) Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024.  Evaluating the ripple effects of knowledge editing in language models.  _Transactions of the Association for Computational Linguistics_ , 11:283–298.
  * Conmy et al. (2023) Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023.  Towards automated circuit discovery for mechanistic interpretability.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 16318–16352. Curran Associates, Inc.
  * Dai et al. (2022) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022.  Knowledge neurons in pretrained transformers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8493–8502, Dublin, Ireland. Association for Computational Linguistics.
  * Dar et al. (2023) Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023.  Analyzing transformers in embedding space.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 16124–16170, Toronto, Canada. Association for Computational Linguistics.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6491–6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang (Lorraine) Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena Hwang, Soumya Sanyal, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. 2023.  Faith and fate: Limits of transformers on compositionality.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 70293–70332. Curran Associates, Inc.
  * Geva et al. (2023) Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023.  Dissecting recall of factual associations in auto-regressive language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12216–12235, Singapore. Association for Computational Linguistics.
  * Geva et al. (2022) Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. 2022.  Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 30–45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Geva et al. (2021) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ghandeharioun et al. (2024) Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024.  Patchscopes: A unifying framework for inspecting hidden representations of language models.  _arXiv preprint arXiv:2401.06102_.
  * Hou et al. (2023) Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. 2023.  Towards a mechanistic interpretation of multi-step reasoning capabilities of language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4902–4919, Singapore. Association for Computational Linguistics.
  * Hsueh et al. (2024) Cheng-Hsun Hsueh, Paul Kuo-Ming Huang, Tzu-Han Lin, Che-Wei Liao, Hung-Chieh Fang, Chao-Wei Huang, and Yun-Nung Chen. 2024.  Editing the mind of giants: An in-depth exploration of pitfalls of knowledge editing in large language models.  _arXiv preprint arXiv:2406.01436_.
  * Ju et al. (2024) Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. 2024.  Investigating multi-hop factual shortcuts in knowledge editing of large language models.  _arXiv preprint arXiv:2402.11900_.
  * Lake and Baroni (2018) Brenden Lake and Marco Baroni. 2018.  Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 2873–2882. PMLR.
  * Li et al. (2024a) Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, and Ying Wei. 2024a.  Understanding and patching compositional reasoning in llms.  _arXiv preprint arXiv:2402.14328_.
  * Li et al. (2024b) Zhoubo Li, Ningyu Zhang, Yunzhi Yao, Mengru Wang, Xi Chen, and Huajun Chen. 2024b.  Unveiling the pitfalls of knowledge editing for large language models.  In _The Twelfth International Conference on Learning Representations_.
  * McGrath et al. (2023) Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. 2023.  The hydra effect: Emergent self-repair in language model computations.  _arXiv preprint arXiv:2307.15771_.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 17359–17372. Curran Associates, Inc.
  * Merullo et al. (2023) Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023.  Language models implement simple word2vec-style vector arithmetic.  _arXiv preprint arXiv:2305.16130_.
  * Meta AI (2024) Meta AI. 2024.  Introducing meta llama 3: The most capable openly available llm to date.
  * Mitchell et al. (2022) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022.  Fast model editing at scale.  In _International Conference on Learning Representations_.
  * Nanda et al. (2023) Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023.  Progress measures for grokking via mechanistic interpretability.  In _The Eleventh International Conference on Learning Representations_.
  * nostalgebraist (2020) nostalgebraist. 2020.  interpreting gpt: the logit lens.
  * Onoe et al. (2023) Yasumasa Onoe, Michael Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. 2023.  Can LMs learn new entities from descriptions? challenges in propagating injected knowledge.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5469–5485, Toronto, Canada. Association for Computational Linguistics.
  * Petty et al. (2024) Jackson Petty, Sjoerd van Steenkiste, Ishita Dasgupta, Fei Sha, Dan Garrette, and Tal Linzen. 2024.  The impact of depth on compositional generalization in transformer language models.  _arXiv preprint arXiv:2310.19956_.
  * Press et al. (2023) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023.  Measuring and narrowing the compositionality gap in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5687–5711, Singapore. Association for Computational Linguistics.
  * Sakarvadia et al. (2023) Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. 2023.  Memory injections: Correcting multi-hop reasoning failures during inference in transformer-based language models.  In _Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP_ , pages 342–356, Singapore. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Wang et al. (2024) Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. 2024.  Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization.  _arXiv preprint arXiv:2405.15071_.
  * Wang et al. (2023a) Fei Wang, Wenjie Mo, Yiwei Wang, Wenxuan Zhou, and Muhao Chen. 2023a.  A causal view of entity bias in (large) language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15173–15184, Singapore. Association for Computational Linguistics.
  * Wang et al. (2023b) Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023b.  Interpretability in the wild: a circuit for indirect object identification in GPT-2 small.  In _The Eleventh International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2022) Nan Xu, Fei Wang, Bangzheng Li, Mingtao Dong, and Muhao Chen. 2022.  Does your model classify entities reasonably? diagnosing and mitigating spurious correlations in entity typing.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8642–8658, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Yang et al. (2024) Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024.  Do large language models latently perform multi-hop reasoning?  _arXiv preprint arXiv:2402.16837_.
  * Zhang et al. (2024) Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. 2024.  A comprehensive study of knowledge editing for large language models.  _arXiv preprint arXiv:2401.01286_.
  * Zhong et al. (2023) Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. 2023.  MQuAKE: Assessing knowledge editing in language models via multi-hop questions.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 15686–15702, Singapore. Association for Computational Linguistics.

##  Appendix A Technical Details

We use the Baukit library Bau (2024) for several of the experiments. Each
experiment was run on 1-4 A100 or H100 GPUs and lasted at most 24 hours. The
models were retrieved and run using the HuggingFace Transformers library Wolf
et al. (2020). When generating with sampling we use the default parameters
from the Transformers library. We use full precision for the 7B, 8B and 13B
models, and half precision for the 70B model.

##  Appendix B Information Propagates to Last Token

As the model’s prediction is emitted from the last position of the prompt,
critical information regarding the bridge entity must propagate to this token.
By using attention knockout, projection, and Patchscopes, we verify this
claim. We find that in a large majority of cases relevant information can
indeed be detected passing from $t_{1}$ to $t_{2}$.

###  B.1 Method

In similar fashion to previous works (Wang et al., 2023b; Geva et al., 2023),
we block hidden representations from attending to other representations at
specific layers and test if the model’s final generation changes. We perform
the knockout by setting the weight of the attention from a source to a target
position at a specific layer to $-\infty$, causing this attention edge to
contribute nothing to the residual update performed on the source
representation. We experiment with blocking a window of layers as previous
work (Geva et al., 2023) shows that information propagation is spread among
multiple layers.

We additionally employ Patchscopes and sublayer vocabulary projections, as
described in §3 and §4 respectively.

###  B.2 Experiment

We run attention knockout blocking information propagating from $t_{1}$ to
$t_{2}$. We block a window of 7 layers in order to capture information flowing
through multiple layers. If blocking a set of layers causes the model to no
longer correctly predict the answer (in the incorrect setting we check for a
change in the generated text), we consider these layers to contain information
critical to the prediction.

We additionally project the updates made by the attention and MLP sublayers to
the last token, as described in §4. In the vocabulary projection we check if
$e_{2}$ is the token with the highest probability, and if so consider the
update to “contain” the bridge entity.

Finally, we run Patchscopes on the last token of the prompt, as done in §3,
checking if $e_{2}$ is encoded in $t_{2}$. Each one of these experiments gives
a unique signal that information must have propagated from $t_{1}$ to $t_{2}$.

###  B.3 Results

Model | Setting | Detected | Mean Layer
---|---|---|---
LLaMA 2 7B | Correct | $85.82\%$ | $12.83$
Incorrect | $95.44\%$ | $8.10$
LLaMA 2 13B | Correct | $81.94\%$ | $15.45$
Incorrect | $71.11\%$ | $16.99$
LLaMA 3 8B | Correct | $78.36\%$ | $10.15$
Incorrect | $82.74\%$ | $7.45$
LLaMA 3 70B | Correct | $90.39\%$ | $29.14$
Incorrect | $94.28\%$ | $20.06$
Table 5: Percentages and mean layer of cases where critical information was first detected propagated from $t_{1}$ to $t_{2}$ using attention knockout, projection, or Patchscopes. Model | Setting | Resolved Cases | Mean Layer
---|---|---|---
LLaMA 2 7B | Correct | $41.4\%$ | $15.3$
Incorrect | $36.7\%$ | $16.4$
LLaMA 2 13B | Correct | $34.2\%$ | $18.2$
Incorrect | $37.5\%$ | $17.9$
LLaMA 3 8B | Correct | $37.4\%$ | $14.3$
Incorrect | $25.6\%$ | $15.3$
LLaMA 3 70B | Correct | $46.9\%$ | $35.2$
Incorrect | $46.2\%$ | $34.5$
Table 6: The results of Patchscopes when decoding $e_{2}$ from $t_{2}$. The
table reports the percentage of cases where $e_{2}$ was successfully decoded
from $t_{2}$. The table additionally reports the mean of the first layer
$e_{2}$ was successfully decoded.

Table 5 shows the percentage of cases where we detect critical information
propagating from $t_{1}$ to $t_{2}$ using at least one of the methods stated
above. We note the relatively high percentages, indicating the information
indeed flows as expected. Additionally, the table shows the that the layers
relevant to the propagation are mostly the middle layers.

Table 6 contains the percentage of cases $e_{2}$ was decoded from $t_{2}$,
showing that this position indeed represents the bridge entity in 25%-46% of
cases. Comparing between the results of Patchscopes on $t_{2}$ to those on
$t_{1}$ (Table 2) we observe a stark difference in the successful source
layers. The results show that the $t_{2}$ first encodes $e_{2}$ during the
mid-upper layers, while $t_{1}$ does so in the lower layers. This can further
be seen in the Patchscopes heat-maps in Figure 7. This gives additional reason
to believe that a sequential computation is taking place.

##  Appendix C Patchscopes First Decoded Layers

(a) LLaMA 2 7B

(b) LLaMA 3 8B

(c) LLaMA 3 70B

Figure 6: Percentage of cases per layer where target entities were first
successfully decoded using Patchscopes.

Figure 6 depicts the percentage of cases per layer where target entities were
first successfully decoded Patchscopes for additional models (as described in
§3).

##  Appendix D Patchscopes Heat-maps

(a) $e_{2}$ decoded from $t_{1}$, LLaMA 2 7B

(b) $e_{2}$ decoded from $t_{2}$, LLaMA 2 7B

(c) $e_{3}$ decoded from $t_{2}$, LLaMA 2 7B

(d) $e_{2}$ decoded from $t_{1}$, LLaMA 2 13B

(e) $e_{2}$ decoded from $t_{2}$, LLaMA 2 13B

(f) $e_{3}$ decoded from $t_{2}$, LLaMA 2 13B

(g) $e_{2}$ decoded from $t_{1}$, LLaMA 3 8B

(h) $e_{2}$ decoded from $t_{2}$, LLaMA 3 8B

(i) $e_{3}$ decoded from $t_{2}$, LLaMA 3 8B

(j) $e_{2}$ decoded from $t_{1}$, LLaMA 3 70B

(k) $e_{2}$ decoded from $t_{2}$, LLaMA 3 70B

(l) $e_{3}$ decoded from $t_{2}$, LLaMA 3 70B

Figure 7: Patchscopes heat-maps of successful source and target layers for all
models.

Figure 7 depicts the heat-maps of Patchscope success cases as introduced in
§3. We report Patchscope experiments for decoding $e_{2}$ from $t_{1}$,
$e_{2}$ from $t_{2}$, and $e_{3}$ from $t_{2}$ for all models.

##  Appendix E First Observed Layers of Pathway Stages

(a) LLaMA 2 7B

(b) LLaMA 2 13B

(c) LLaMA 3 70B

Figure 8: A comparison of the first layers of each stage in the pathway
between correct and incorrect cases for additional models.

Figure 8 depicts a comparison of the first layers of each stage in the pathway
between correct and incorrect cases (as shown in §4) for additional models.

##  Appendix F Back-patching Heat-maps

(a) Source token is $t_{1}$, LLaMA 2 7B

(b) Source token is $t_{2}$, LLaMA 2 7B

(c) Source token is $t_{1}$, LLaMA 3 8B

(d) Source token is $t_{2}$, LLaMA 3 8B

(e) Source token is $t_{1}$, LLaMA 3 70B

(f) Source token is $t_{2}$, LLaMA 3 70B

Figure 9: Back-patching heat-maps of successful source and target layers case
for additional models.

Figure 9 depicts the back-patching heap-maps of successful source and target
layers (as shown in §5) for addition models.


# MammalNet: A Large-scale Video Benchmark
for Mammal Recognition and Behavior Understanding

Jun Chen1∗ Ming Hu1∗ Darren J. Coker1 Michael L. Berumen1
Blair Costelloe2,3 Sara Beery4 Anna Rohrbach5 Mohamed Elhoseiny1
1King Abdullah University of Science and Technology (KAUST)
2 Max Planck Institute of Animal Behavior , 3 University of Konstanz
4Massachusetts Institute of Technology, 5University of California, Berkeley

###### Abstract

Monitoring animal behavior can facilitate conservation efforts by providing
key insights into wildlife health, population status, and ecosystem function.
Automatic recognition of animals and their behaviors is critical for
capitalizing on the large unlabeled datasets generated by modern video devices
and for accelerating monitoring efforts at scale. However, the development of
automated recognition systems is currently hindered by a lack of appropriately
labeled datasets. Existing video datasets 1) do not classify animals according
to established biological taxonomies; 2) are too small to facilitate large-
scale behavioral studies and are often limited to a single species; and 3) do
not feature temporally localized annotations and therefore do not facilitate
localization of targeted behaviors within longer video sequences. Thus, we
propose _MammalNet_ , a new large-scale animal behavior dataset with taxonomy-
guided annotations of mammals and their common behaviors. MammalNet contains
over 18K videos totaling 539 hours, which is $\sim$10 times larger than the
largest existing animal behavior dataset [36]. It covers 17 orders, 69
families, and 173 mammal categories for animal categorization and captures 12
high-level animal behaviors that received focus in previous animal behavior
studies. We establish three benchmarks on MammalNet: standard animal and
behavior recognition, compositional low-shot animal and behavior recognition,
and behavior detection. Our dataset and code have been made available at:
https://mammal-net.github.io.

††footnotetext:  ∗Equal contribution.

##  1 Introduction

Datasets | Dataset Properties | Tasks
---|---|---
Publicly Available? |  Taxonomy-guided Animal Annotation? |  No. of Videos |  No. of Actions |  No. of Behaviors |  No. of Animal Categories |  No. of Mammal Categories |  Total Duration |  Animal Classification |  Action/Behavior Recognition |  Action/Behavior Detection
Wild Felines [20] |  ✕ |  ✕ |  2,700 |  3 |  - |  3 |  3 |  - |  $\checkmark$ |  $\checkmark$ |  ✕
Wildlife Actions [29] |  ✕ |  ✕ |  10,600 |  7 |  - |  32 |  11 |  - |  $\checkmark$ |  $\checkmark$ |  ✕
Animal Kingdom [36] |  $\checkmark$ |  ✕ |  4,301 |  140 |  - |  850 |  - |  50 (h) |  ✕ |  $\checkmark$ |  ✕
MammalNet (ours) |  $\checkmark$ |  $\checkmark$ |  18,346 |  - |  12 |  173 |  173 |  539 (h) |  $\checkmark$ |  $\checkmark$ |  $\checkmark$

Table 1: The comparison among existing animal behavior understanding video
datasets. Compared to other datasets, MammalNet annotates the animals by
following the scientific mammal taxonomy, focuses on more high-level behavior
recognition, has the largest number of mammal categories, collects the largest
number of animal behavior videos, totalling 539 hours, and also enables
behavior detection tasks.

Animal species are a core component of the world’s ecosystems. Through their
behavior, animals drive diverse ecological processes, including seed
dispersal, nutrient cycling, population dynamics, speciation, and extinction.
Thus, understanding and monitoring the behaviors of animals and their
interactions with their physical and social environments is key to
understanding the complexities of the world’s ecosystems, an objective that is
especially critical now given the ongoing biodiversity crisis [12].

Modern sensors, including camera traps, drones, and smartphones, allow
wildlife researchers, managers, and citizen scientists to collect video data
of animal behavior on an unprecedented scale [43]. However, processing this
data to generate actionable, timely insights remains a major challenge. Manual
human review and annotation of footage to identify and locate species and
behavioral sequences of interest is time-intensive and does not scale to large
datasets. Thus, methods for automated animal and behavioral recognition could
open the door to large-scale behavioral monitoring and speed up the time to
produce usable data, thereby reducing the time to implement management
directives.

The first essential step to creating such an AI system for animal and behavior
recognition is curating a diverse, representative dataset that allows us to
formalize these challenges as computer vision tasks and benchmark potential
solutions. Most previous datasets either only cover a limited number of animal
and behavior types [4, 38], or do not implement animal labeling [36], or
include a small number of videos with insufficient environmental diversity
[48, 4, 38]. Recently, a dataset named “Animal Kingdom” [36] was proposed to
study animal actions and is currently the largest existing behavioral dataset,
to the best of our knowledge. However, it only contains 4,310 videos totaling
50 hours, which might be insufficient for large-scale animal behavior studies
considering its diversity. Furthermore, the authors only focus on the
recognition of atomic actions such as yawning, swimming, and flying. These
basic actions cannot be easily matched to the higher-order behavioral states
that are of primary interest to end users in animal management and
conservation [6]. For example, a cheetah that is running may either be
hunting, escaping, or playing. Finally, and most importantly, they do not
support some important tasks such as animal recognition and behavior detection
which are essential for animal behavior understanding.

To overcome the limitations of previous datasets, we propose a new dataset
called _MammalNet_. We specifically focus on mammals since they, unlike other
animal classes such as birds or insects, usually have more diverse and
distinguishable behavior statuses. MammalNet is comprised of 539 hours of
annotated videos, which is $\sim$10 times longer than that of the largest
available animal behavior dataset. It contains 18,346 videos depicting 12
fundamental high-level behaviors from hundreds of mammal species. Importantly,
it focuses on 12 higher-order animal behaviors that are the focus of previous
animal behavior literature [8, 17, 3, 33], rather than atomic actions.
MammalNet also categorizes animals according to the scientific taxonomy
available in Wikipedia, as we show in Fig. LABEL:fig:teaser; hence the dataset
can be flexibly expanded in the future by following the same protocols. It
includes videos of approximately 800 mammal species in 173 mammal categories.
We establish three benchmarks inspired by ecological research needs - standard
animal & behavior classification, compositional low-shot animal & behavior
recognition, and behavior detection – to promote future study in animal
behavior understanding.

Through our experiments, we find that: (1) Correctly recognizing the animals
and behaviors is a challenging task even for the state-of-the-art models,
especially for less-frequent animals. The top-1 per-class accuracy is 32.5 for
animal recognition, 37.8 for behavior recognition, and 17.8 for their joint
recognition in our best-performing model. (2) Behavior recognition for unseen
animals can be transferred from observations of other seen animals due to
their similar features such as appearance and movement style, which can help
in studies of animals with less available data. However, to achieve more
accurate behavior recognition, having access to videos of the target animals
and behaviors is still crucial.

##  2 Related Work

Automatic animal recognition and behavior detection can help humans monitor
and efficiently process animal behavior data [32, 40, 48, 22, 34, 7, 47]. It
can massively reduce the labour cost from manually collecting and analyzing
animal activities. During the past few years, many datasets [38, 20, 29, 18]
have been introduced to develop foundations for animal behavior research. We
systematically analyze the previous datasets and summarize several important
limitations that prevent them from being used for large-scale animal
recognition and behavior understanding:

Lack of behavior understanding. Many previous datasets only focus on animal
recognition or pose estimation from images, but lack behavior learning. For
example, iNaturalist [45] collects 859,000 images covering more than 5,000
different types of plants and animals. NABird [44] collects 48,562 images of
555 different North American bird species. Also, some works focus on narrow
animal recognition such as dogs [27], birds [50, 46] or cats [13]. On the
other hand, many works also focus on pose estimation [31, 41, 21, 10, 18] and
animal face detection [26, 51, 39]. They are generally not applicable to
learning animal behavior.

Lack of taxonomic diversity for different behaviors. Previous animal behavior
datasets have minimal taxonomic coverage - often containing just a single
animal species. For example, there are existing behavior recognition datasets
for elephants [28], sheep [37], monkeys [5], tigers [16], etc. While useful
for the studied species, these datasets do not enable the exploration of
behavior across species which is necessary to scale up behavior identification
without requiring training examples of every possible combination of species
and behavior.

Lack of taxonomy-guided animal annotation. Previous animal behavior datasets
either do not include animal recognition as a task [36] or group species
according to subjective as opposed to scientific criteria [29, 36]. In
contrast, our dataset collects and annotates the videos according to the
scientific mammal taxonomy. By following this taxonomy we allow exploration of
behavior from an evolutionary perspective, as well as enable standardized and
consistent dataset expansion under the same protocol.

Actions vs. Behaviors. Previous works mostly focus on atomic action
recognition [36, 29, 20]. For example, some of their action classes are _walk,
stand still, fly, etc_. In contrast, MammalNet focuses on higher-level animal
behaviors, such as _hunt, feed baby, etc_. Behavior here denotes the main
activity instance during a period, and it is usually composed of multiple
atomic actions. These complex behaviors are needed to describe and summarize
video activity bouts in a manner which is valuable for ecological research.

We further compare our MammalNet dataset with the other existing animal
behavior understanding video datasets, and summarize the key difference in
Table 1.

##  3 Constructing MammalNet

Figure 1: A subset of the mammal taxonomy of MammalNet. It includes 3 orders,
11 families, and 45 genera.

The goal of _MammalNet_ is to provide a large-scale mammal video dataset that
benchmarks both animal and behavior recognition. In this section, we discuss
our dataset construction protocol, including the choice of scientific animal
taxonomy, crowdsourced annotations, and performing manual quality control
during video collection and annotation. Finally, we describe the statistical
profile of MammalNet.

Figure 2: The examples for the annotated target behavior boundaries. The
frames marked in red boxes denote the annotated temporal boundaries for the
target behavior.

###  3.1 Animal Taxonomy and Behavior Collection

Animal taxonomy construction. Accurate animal recognition is one of the main
goals defined by MammalNet. To ensure this task is scientifically relevant, we
aimed to collect and structure our list of target animals based on the mammal
taxonomy. First, we collected a diverse list of mammals, approximately 800
mammal species and sub-species, from the National Geographic [2] and Animal
A-Z [1] websites. Next, we manually mapped each mammal onto the taxonomic
structure (including class, order, family, genus, tribe, sub-family and
species) in mammal taxonomy from Wikipedia. In total, the classes included in
MammalNet cover 17 orders, 69 families, and 173 mammal categories. A taxonomic
subset is visualized in Fig. 1.

Behavior collection. We aim to enable the study of complex, high-level animal
behavior as opposed to the simpler atomic actions emphasized in previous work
[36, 29]. Behavior here represents the major activity being displayed during a
period of a video, and can be viewed as a series of atomic actions
collectively serving a higher-level purpose. For example, hunting behavior, as
defined in MammalNet, can often be decomposed into running, chasing and
killing actions, etc. Identification of behaviors at this level of definition
is more useful for ecologists and zoologists [8, 17, 3, 33] compared to atomic
actions.

Inspired by previous biological and ecological studies [8, 17, 3], we consider
12 fundamental mammal behaviors under 5 different groups in our study. They
are respectively:

Foraging behaviors: eat food, drink water, hunt.

Reproductive behaviors: mate, feed baby, give birth.

Hygiene behaviors: groom.

Agonistic behaviours: fight.

Maintenance behaviors: urinate, defecate, sleep, vomit.

Figure 3: Number of videos per each mammal category. We rank the categories
according to their trimmed videos frequency. Figure 4: Number of videos per
each behavior. We rank the behavior according to their trimmed videos
frequency.

###  3.2 Video Collection and Quality Assurance

Dataset curation is usually a costly process requiring a lot of manual
annotation by humans. Some datasets are annotated by domain experts to
encourage more reliable labeling, particularly for challenging tasks, but this
expertise comes at a much higher cost and is thus hard to scale. We adapted a
semi-automatic crowdsourcing approach to collect and annotate our datasets,
inspired by many previous works [15, 23, 49, 9].

Online video retrieval. Our goal was to collect videos depicting each animal
in our database performing each of the 12 focal behaviors. To achieve this, we
queried YouTube with the text combinations of each animal common name and
behavior, e.g., _tiger hunting_ , and downloaded videos where the title
included our queried animal name and behavior. However, some animal names are
too rare to generate enough relevant videos, e.g., _Brown hyena_ or _Spotted
hyena_. In such cases, we used a more common name, _hyena_ , to represent the
union of those animals. In order to retrieve more relevant videos from the
search engine, we also expanded each animal with their synonyms as given in
Wikipedia. For example, we expanded the original _artic fox_ with other
equivalent names such as _white fox_ , _polar fox_ and _snow fox_. Each video
was downloaded at its highest available resolution.

Data filtering. During video retrieval, we downloaded the videos that are
accessible for people $<$16 years old to avoid the videos with violent
content. We also prioritized videos that are shorter than 10 minutes in
duration to limit the total storage.

However, some videos might have irrelevant content due to the inaccuracy of
text-based retrieval. For example, the downloaded videos might 1) display
cartoon or toy animals or unrealistic environments such as games or movies, 2)
display a static image instead of a continuous video, 3) involve a lot of
human-animal interaction, e.g., human feeding an animal, as opposed to
focusing on animal behavior, 4) not contain the specified animal and/or
behavior.

To alleviate these issues, we employed Amazon Mechanical Turk workers to
verify the presence of the animal and behavior, and identify other quality
issues. We assigned each video to three different workers and provided them
with the pictures of an animal with its common name, and an expected behavior.
We asked the workers to verify if this animal and behavior indeed appear in
the video. Only the videos that “pass” by all the three workers were kept for
the following behavior localization annotation. Before the workers started the
verification, we first provided them the verification instructions and asked
them to complete a corresponding qualification test with 20 multiple choice
questions to ensure only qualified workers could participate in our task.

###  3.3 Animal Behavior Localization Annotation

The target behavior typically does not span the whole video. To localize the
part where the target behavior is actually occurring in the video, we asked
the AMT workers to manually annotate the respective temporal boundaries.
Namely, we asked five different qualified workers to annotate the start and
end frames for the target behavior in each video. In the end, each video
received at least 5 annotated behavior boundaries. To achieve robust
annotation agreement, we used the complete linkage algorithm [14] to cluster
different temporal boundaries and merge them into one or several more stable
ones that received multiple agreements. Note, that a single video might have
multiple discrete occurrences of a given behavior, and thus have multiple
boundary definitions. We show several examples for annotated target behavior
boundaries in Fig. 2.

###  3.4 Recognition at lowest feasible taxonomic level

We categorize animals according to the lowest and feasible taxonomic
classification, rather than the species level, for the following reasons: 1)
YouTube videos contain the ambiguous and inexpert species labels. 2) We are
not using expert annotators, and even experts often cannot reliably identify
animals at the species or even genus level based on crowdsourced or field
images [25]. Thus, it is more practical to classify animals in our dataset to
the lowest feasible taxonomic level rather than species. As a result, our
taxonomy contains 173 distinguishable taxonomic classification levels
including: sub-family, tribe and genus.

###  3.5 MammalNet Statistics

The final MammalNet dataset contains 18,346 videos (539 hours); after we trim
the videos according to the annotated behavior boundaries, it increases to
20,033 trimmed video instances (394 hours). In total, it covers 173 mammal
categories, 69 families and 17 orders in our dataset, with the total of 173
mammal categories defined for our recognition tasks. MammalNet also contains
12 different common behaviors. The average duration for the untrimmed and
trimmed videos is 106 and 77 seconds, respectively. Over 54% of videos reach
HD resolutions (1280 $\times$ 720). We demonstrate the data distribution in
terms of each animal category and behavior in Fig. 3 and 4. We observe that
the number of collected videos per type follow a long-tail distribution.

| Animal Classification | Behavior Classification | Joint Classification
---|---|---|---
Baselines | Many | Medium | Few | All | Many | Medium | Few | All | Many | Medium | Few | All
| 12 | 28 | 133 | 173 | 4 | 4 | 4 | 12 | 33 | 180 | 823 | 1036
SlowFast [19] | 49.6 | 35.6 | 9.5 | 17.2 | 39.0 | 27.6 | 14.9 | 27.2 | 27.2 | 19.4 | 4.4 | 8.6
C3D [42] | 48.6 | 35.3 | 10.0 | 17.5 | 38.2 | 27.8 | 11.7 | 25.9 | 28.1 | 18.8 | 4.5 | 8.6
I3D [11] | 48.8 | 34.9 | 10.5 | 17.8 | 39.5 | 27.2 | 14.8 | 27.2 | 29.6 | 20.5 | 4.4 | 8.9
MViT V2 [30] | 48.5 | 35.5 | 12.9 | 19.7 | 42.3 | 29.2 | 11.6 | 27.7 | 29.5 | 19.6 | 4.8 | 9.0
SlowFast* | 58.3 | 43.1 | 16.6 | 24.5 | 45.1 | 32.7 | 14.8 | 30.9 | 38.6 | 23.5 | 7.3 | 12.1
C3D* | 58.3 | 45.4 | 19.1 | 26.8 | 44.6 | 36.0 | 15.9 | 32.2 | 38.0 | 26.4 | 8.4 | 13.5
I3D* | 58.6 | 42.9 | 16.9 | 24.8 | 46.3 | 35.0 | 14.8 | 32.1 | 38.3 | 24.5 | 8.6 | 13.3
MViT V2* | 66.7 | 56.0 | 23.4 | 32.5 | 50.9 | 42.4 | 20.0 | 37.8 | 46.2 | 33.0 | 11.8 | 17.8
Table 2: Per-class Top-1 accuracy for animal, behavior and their joint prediction.* denotes the initialization from the model pretrained on Kinetics 400 [24]. Transfer learning from the human action to the animal behavior recognition receives considerable performance gain. Best performance for each split has been highlighted in bold.  | Animal Classification | Behavior Classification | Joint Classification
---|---|---|---
Baselines | Many | Medium | Few | All | Many | Medium | Few | All | Many | Medium | Few | All
| 12 | 28 | 133 | 173 | 4 | 4 | 4 | 12 | 33 | 180 | 823 | 1036
SlowFast | 58.3 | 43.1 | 16.6 | 24.5 | 45.1 | 32.7 | 14.8 | 30.9 | 38.6 | 23.5 | 7.3 | 12.1
C3D | 58.3 | 45.4 | 19.1 | 26.8 | 44.6 | 36.0 | 15.9 | 32.2 | 38.0 | 26.4 | 8.4 | 13.5
I3D | 58.6 | 42.9 | 16.9 | 24.8 | 46.3 | 35.0 | 14.8 | 32.1 | 38.3 | 24.5 | 8.6 | 13.3
MViT V2 | 66.7 | 56.0 | 23.4 | 32.5 | 50.9 | 42.4 | 20.0 | 37.8 | 46.2 | 33.0 | 11.8 | 17.8
Table 3: Per-class Top-1 accuracy for animal, behavior and their joint prediction.* denotes the initialization from the model pretrained on Kinetics 400 [24]. Transfer learning from the human action to the animal behavior recognition receives considerable performance gain. Best performance for each split has been highlighted in bold.  Baselines | Animal | Behavior | Joint
---|---|---|---
SlowFast [19] | 35.4 | 34.2 | 17.4
C3D [42] | 35.0 | 33.5 | 17.1
I3D [11] | 35.2 | 34.3 | 17.9
MViT V2 [30] | 35.6 | 36.8 | 18.0
SlowFast* | 43.0 | 39.4 | 22.8
C3D* | 44.4 | 40.3 | 24.6
I3D* | 43.4 | 41.2 | 24.0
MViT V2* | 52.6 | 46.6 | 30.6
Table 4: Per-example accuracy for animal, behavior and their joint prediction.
* denotes the initialization from the pretrained model.

##  4 Experimental Results

We construct three main tasks on MammalNet: 1) Standard animal and behavior
classification on trimmed videos, 2) Compositional low-shot animal and
behavior recognition on trimmed videos, and 3) Behavior detection on untrimmed
videos. In both the classification and detection tasks, we baseline several
state-of-the-art models [30, 19, 53] that have been successfully applied to
human action recognition and detection. In the following, we describe the
formulation of each challenge and provide baselines and analysis.

###  4.1 Standard Animal and Behavior Classification on Trimmed Videos

This task explores classification of both the primary behavior that occurs in
a trimmed video and the animal that performs the behavior. We report the top-1
per-example and per-class accuracy for all the baseline models.

Many, medium, few splits. To capture the effect of the long-tailed nature of
the MammalNet dataset, we group the animal, behavior, and their composition
classes into _many_ , _medium_ , and _few_ based on their frequency, and
report the average per-class accuracy bands chosen based on the frequency
percentiles. For animal categories this is broken down as _many_ : top 7%
frequent classes, _medium_ : middle 16% classes, and _few_ : the remaining 77%
classes. For behavior, _many_ : top 33% frequent classes, _medium_ : middle
33%, and _few_ : the remaining 33% classes. For joint classification, _many_ :
top 3% frequent classes, _medium_ : middle 17% classes, _few_ : the remaining
80% classes. We show the number of classes per each split in Table 3.

Dataset setup. We randomly split the examples from each animal-behavior
category into 70% for training, 10% for validation, and 20% for testing, and
it results in 14,554 training, 1,638 validation, and 3,841 testing videos,
respectively.

| Compositional Low-Shot Behavior Classification
---|---
Baselines | 0-shot | 1-shot | 5-shot
Per-example | Per-class | Per-example | Per-class | Per-example | Per-class
| A / B | A / B | A / B | A / B | A / B | A / B
C3D* | 27.4 / 23.7 | 18.3 / 16.1 | 29.2 / 25.5 | 21.1 / 19.9 | 33.3 / 29.3 | 25.2 / 23.0
I3D* | 25.3 / 23.9 | 16.7 / 15.2 | 26.8 / 25.7 | 16.9 / 18.3 | 30.5 / 28.3 | 21.7 / 21.9
SlowFast* | 26.2 / 24.8 | 17.9 / 16.3 | 26.5 / 26.3 | 16.7 / 19.0 | 29.6 / 29.2 | 22.5 / 22.4
MViT V2* |  32.2 / 26.2 |  20.7 / 18.1 |  33.7 / 28.9 |  22.9 / 22.5 |  39.3 / 31.7 |  31.0 / 26.0
Table 5: Compositional low-shot animal and behavior recognition. * denotes the
initialization from the model pretrained on Kinetics 400 [24]. “A” denotes the
animal category and “B” denotes the behavior category. The best performance
per each column has been highlighted in bold.

Baselines | mAP
---|---
0.50 | 0.60 | 0.70 | 0.80 | 0.90 | Avg.
CoLA [52] | 26.02 | 22.70 | 18.98 | 13.46 | 3.05 | 15.81
TAGS [35] | 23.09 | 20.97 | 19.09 | 16.98 | 12.56 | 17.63
ActionFormer [53] | 28.48 | 26.14 | 23.17 | 18.69 | 10.48 | 20.07

Table 6: The results for behavior detection. We report mAP at the IoU
thresholds of [0.5:0.1:0.9]. Average mAP is computed by averaging different
tIoU thresholds.

Baselines. We compare SlowFast [19], I3D [11], C3D [42] and MViT V2 [30]
models on our tasks. These models are evaluated in two versions: 1) Training
with random initialization 2) Initializing with weights from a model
pretrained on Kinetics 400 [24]. These methods were originally designed for
human action recognition and hence do not have an ability to predict both an
action and a subject by default. To accommodate them into our joint prediction
setting, we have two task heads, one for animal category recognition and one
for behavior recognition. We compute the joint loss, $\mathcal{L}_{joint}$,
for both animal and behavior classification as shown in Fig. LABEL:fig:teaser.
We tune all the hyper-parameters on the validation data. The final hyper-
parameters for each model are provided in the supplement. The loss is defined
as:

$\mathcal{L}_{joint}=-\frac{1}{M}\sum_{i}^{M}{y^{a}_{i}\text{log}(p^{a}_{i})}-\frac{1}{N}\sum_{j}^{N}{y^{b}_{j}\text{log}(p^{b}_{j})}$
(1)

where $M$ is the number of animal classes, and $N$ is the number of behavior
classes, $p^{a}_{i}$ and $y^{a}_{i}$ denote the animal prediction probability
and ground truth label for the category $i$, $p^{b}_{y}$ and $y^{b}_{j}$
denote the behavior prediction probability and ground truth label for the
category $j$.

Experimental results. The results for per-class and per-example classification
are summarized in Tables 3 and 4, respectively. We find that MViT v2 is able
to achieve competitive results for all the splits. The best top-1 joint per-
class accuracy is 17.8 and per-example accuracy is 30.6, which points to
significant room for improvement on these challenging tasks. We also observe
that transfer learning from the model that is pretrained on Kinetics 400, a
human action recognition dataset, improves both the animal and behavior
classification accuracy (with MViT v2, this corresponds to a per-class
accuracy gain from 19.7 to 32.5 for animal classification, and from 27.7 to
37.8 for behavior classification). Additionally, we find that performance gain
from pretraining is higher for frequently occurring animals and behaviors,
indicated by the results shown in $many$, $medium$, and $few$ splits.
Accurately predicting low-frequency animal and behavior categories remains a
significant challenge.

###  4.2 Compositional Low-shot Animal and Behavior Classification on Trimmed
Videos

It is hard to find a sufficient number of labeled behavior samples for all the
animals in our collected taxonomy. For example, _numbat_ and _florida panther_
have very few behavior annotations. However, it is plausible to imagine that
behaviors can be transferred among different animals that have similar
appearances and movement styles. Also the animal recognition under different
behaviors (hunting, fighting) can be mutually transferred. To investigate
these phenomena, we design the compositional low-shot animal and behavior
classification task.

Dataset setup. We first select the animal-behavior compositional classes that
contain more than 5 examples and allocate 25% of classes to the test set
(4,088 videos). For the remaining classes (the other 75% of classes and those
classes with $\leq$ 5 examples), we randomly designate 90% of classes as the
training set and 10% as the validation set (898 videos). Under the low-shot
scheme, for each compositional class, we randomly sample 5 examples from the
test set and move 0, 1, or 5 of them into the training set for the zero-shot,
1-shot, and 5-shot setup, respectively (14,377, 14,511 and 15,047 training
videos). The train, val and test sets consist of 983, 53, and 134
compositional classes, respectively.

Baselines. We evaluate the compositional low-shot classification with the
SlowFast [19], I3D [11], C3D [42] and MViT V2 [30] models under the joint
loss. We initialize these models with the weights pretrained on Kinetics 400
[24].

Experimental results. We summarize the results in Table 5. It shows that MViT
v2 consistently achieves the best performance under our low-shot setup. The
behavior classification can still achieve 26.2 per-example and 18.1 top-1
accuracy under the zero-shot setting for MViT v2 model. This indicates that
behavior classification is transferable from other animals in some cases.
Additionally, we observe that its performance is improved to 31.7 per-example
and 26.0 per-class top-1 accuracy under 5-shot setup, indicating that training
on more videos with the behaviors from the same animal is still necessary. A
similar phenomenon is observed to the few-shot animal recognition.

Figure 5: The visualization presents various instances of joint animal and
behavior classification. In the third column, accurate predictions are
displayed, while the fourth, fifth, and sixth columns showcase mispredicted
examples where either the animal or the behavior does not correspond to the
correct prediction.

###  4.3 Behavior Detection on Untrimmed Videos

This task is to detect the behavior in untrimmed videos. The behavior
detection algorithm should correctly detect the temporal range for the primary
activity presented in the video. We follow previous temporal action
localization works [9, 53] to benchmark this task. We report the mean Average
Precision (mAP) with different temporal intersections over the union (tIoU)
thresholds [0.5:0.1:0.9] as our evaluation metric. We also report the average
mAP averaging across different tIoUs.

Figure 6: The visualization for behavior detection examples.

Dataset setup. Similar to the previous standard animal and behavior
classification, we follow the [train:0.7, val:0.1, test:0.2] ratios to split
the untrimmed videos at the animal-behavior composition level. This results in
13,318 videos for training, 1,486 for validation and 3,542 for testing. We
tune all the hyper-parameters based on the validation set.

Baselines. We evaluate our datasets with the baseline models such as
ActionFormer [53], TAGS [35], and CoLA [52]. To produce the features for our
MammalNet videos, we first finetune a two-stream I3D [11] model, that is
originally pretrained on ImageNet [15] and Kinetics 400 [24], on our dataset,
and then extract the RGB and optical flow features for each video. We
concatenate these two features together as the model input.

Experimental results. We show the behavior detection results in Table 6. Among
all the baselines, ActionFormer demonstrates the most competitive performance
with an average mAP of 20.07, and also achieves 28.48 mAP for the threshold of
0.5. Overall, it is clear to see that the behavior detection task is still
very challenging for current methods.

##  5 Analysis

Demonstration of animal and behavior classification. We sample some prediction
examples from MViT v2 [30] for the joint categories _Acinonyx hunt_ , _Canis
eat_ and _Antilopini fight_. We demonstrate one correct prediction and three
mispredicted examples where the model mis-recognizes the animal or behavior or
both of them in Fig. 5.

Demonstration of behavior detection: We visualize behavior detection results
in Fig. 6 from ActionFormer [53]. The top example shows correctly predicted
behavior with a proposal closely aligned with the ground-truth. The bottom
example shows a misclassified behavior, and it mistakenly predicts the _hunt_
behavior as _fight_.

Joint animal and behavior recognition vs. separate recognition. We also train
the model for recognizing the animal and behavior separately and provide the
results in the Table 1 of the supplementary. Comparing with the joint
recognition, we find that training a system to recognize the animal and
behavior together can improve the behavior recognition under separate training
by $\sim$2.2 per-class accuracy. This indicates that being capable of
understanding the animal types can benefit behavior prediction. However, the
results also indicate that predicting animal only can be more useful in
recognizing animal types.

Dataset bias. Our dataset is downloaded from YouTube channels with diverse
backgrounds and video quality, and it might differ from video datasets
collected for behavioral or ecological studies. The videos on YouTube might
also exhibit potential biases:

1) Over-representation of some behaviors and under-representation of others.
For example, people might prefer watching videos with _fight_ or _eat food_
activities, and these two behaviors are more likely to be over-represented,
while _urinate_ and _defecate_ behaviors are less interesting to humans and
hence are underrepresented on YouTube. However, they are still important in
informing conservation-related actions to protect the environment.

2) Bias towards captive animals or wild animals that are habituated to humans.
We find that many videos are shot at zoos, farms, and homes, etc. These
animals may display different behaviors than wild or non-habituated animals.

##  6 Conclusion

We introduced MammalNet, a large-scale video dataset for mammal recognition
and behavior understanding. We have collected videos for hundreds of different
mammals and structured them by following the scientific mammal taxonomy.
MammalNet consists of 18,346 untrimmed videos covering 173 mammal categories
and 12 common behaviors. We established three challenges: standard animal and
behavior classification, compositional low-shot animal and behavior
classification, and behavior detection. Through our experiments, we found that
the accurate recognition of animals at scale and their common behaviors is
very challenging even with current state-of-the-art models, especially when
the dataset has a long-tail distribution. We also found that learning to
recognize the behavior of unseen animals is possible via transfer from the
other seen animals. To promote further research and development in the field
of animal behavior study, we have open-sourced all of our data and code to the
research community.

Acknowledgement: This work is supported by KAUST BAS/1/1685-01-01 and KAUST
FCC/1/1973-58-01 (Red Sea Research Center), DARPA’s SemaFor and PTG programs,
the Caltech Resnick Sustainability Institute, and Germany’s Excellence
Strategy–‘Centre for the Advanced Study of Collective Behaviour’ EXC
2117-422037984.

## References

  * [1] A-z animals.  https://a-z-animals.com/.
  * [2] National geographics.  https://www.nationalgeographic.com/.
  * [3] John Alcock.  Animal behavior: An evolutionary approach.  Sinauer Associates, 2009.
  * [4] David J Anderson and Pietro Perona.  Toward a science of computational ethology.  Neuron, 84(1):18–31, 2014.
  * [5] Praneet C Bala, Benjamin R Eisenreich, Seng Bum Michael Yoo, Benjamin Y Hayden, Hyun Soo Park, and Jan Zimmermann.  Automated markerless pose estimation in freely moving macaques with openmonkeystudio.  Nature communications, 11(1):1–12, 2020.
  * [6] Oded Berger-Tal, Daniel T Blumstein, Scott Carroll, Robert N Fisher, Sarah L Mesnick, Megan A Owen, David Saltz, Colleen Cassady St Claire, and Ronald R Swaisgood.  A systematic survey of the integration of animal behavior into conservation.  Conservation Biology, 30(4):744–753, 2016.
  * [7] Cigdem Beyan and Robert B Fisher.  Detection of abnormal fish trajectories using a clustering based hierarchical classifier.  In BMVC, 2013.
  * [8] Michael D Breed and Janice Moore.  Animal behavior.  Academic Press, 2021.
  * [9] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015.
  * [10] Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai.  Cross-domain adaptation for animal pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9498–9507, 2019.
  * [11] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [12] Gerardo Ceballos, Paul R Ehrlich, and Peter H Raven.  Vertebrates on the brink as indicators of biological annihilation and the sixth mass extinction.  Proceedings of the National Academy of Sciences, 117(24):13596–13602, 2020.
  * [13] Yu-Chen Chen, Shintami C Hidayati, Wen-Huang Cheng, Min-Chun Hu, and Kai-Lung Hua.  Locality constrained sparse representation for cat recognition.  In International Conference on Multimedia Modeling, pages 140–151. Springer, 2016.
  * [14] Daniel Defays.  An efficient algorithm for a complete link method.  The Computer Journal, 20(4):364–366, 1977.
  * [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [16] Rod K Dishman, Andrew S Jackson, and Molly S Bray.  Self-regulation of exercise behavior in the tiger study.  Annals of Behavioral Medicine, 48(1):80–91, 2014.
  * [17] Lee Alan Dugatkin.  Principles of animal behavior.  University of Chicago Press, 2020.
  * [18] Cheng Fang, Tiemin Zhang, Haikun Zheng, Junduan Huang, and Kaixuan Cuan.  Pose estimation and behavior classification of broiler chickens based on deep neural networks.  Computers and Electronics in Agriculture, 180:105863, 2021.
  * [19] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019.
  * [20] Liqi Feng, Yaqin Zhao, Yichao Sun, Wenxuan Zhao, and Jiaxi Tang.  Action recognition using a spatial-temporal network for wild felines.  Animals, 11(2):485, 2021.
  * [21] Jacob M Graving, Daniel Chae, Hemal Naik, Liang Li, Benjamin Koger, Blair R Costelloe, and Iain D Couzin.  Deepposekit, a software toolkit for fast and robust animal pose estimation using deep learning.  Elife, 8:e47994, 2019.
  * [22] Shilpi Gupta, Prerana Mukherjee, Santanu Chaudhury, Brejesh Lall, and Hemanth Sanisetty.  Dftnet: Deep fish tracker with attention mechanism in unconstrained marine environments.  IEEE Transactions on Instrumentation and Measurement, 70:1–13, 2021\.
  * [23] Fabian Caba Heilbron and Juan Carlos Niebles.  Collecting and annotating human activities in web videos.  In Proceedings of International Conference on Multimedia Retrieval, pages 377–384, 2014.
  * [24] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [25] Roland Kays, Monica Lasky, Maximilian L Allen, Robert C Dowler, Melissa TR Hawkins, Andrew G Hope, Brooks A Kohli, Verity L Mathis, Bryan McLean, Link E Olson, et al.  Which mammals can be identified from camera traps and crowdsourced photographs?  Journal of Mammalogy, 2022.
  * [26] Muhammad Haris Khan, John McDonagh, Salman Khan, Muhammad Shahabuddin, Aditya Arora, Fahad Shahbaz Khan, Ling Shao, and Georgios Tzimiropoulos.  Animalweb: A large-scale hierarchical dataset of annotated animal faces.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6939–6948, 2020.
  * [27] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.  Novel dataset for fine-grained image categorization: Stanford dogs.  In Proc. CVPR workshop on fine-grained visual categorization (FGVC), volume 2. Citeseer, 2011.
  * [28] Nicole Laws, Andre Ganswindt, Michael Heistermann, Moira Harris, Stephen Harris, and Chris Sherwin.  A case study: fecal corticosteroid and behavior as indicators of welfare during relocation of an asian elephant.  Journal of Applied Animal Welfare Science, 10(4):349–358, 2007\.
  * [29] Weining Li, Sirnam Swetha, and Mubarak Shah.  Wildlife action recognition using deep learning.  2020\.
  * [30] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4804–4814, 2022.
  * [31] Alexander Mathis, Thomas Biasi, Steffen Schneider, Mert Yuksekgonul, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis.  Pretraining boosts out-of-domain robustness for pose estimation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1859–1868, 2021.
  * [32] Mackenzie Weygandt Mathis and Alexander Mathis.  Deep learning tools for the measurement of animal behavior in neuroscience.  Current opinion in neurobiology, 60:1–11, 2020.
  * [33] Joy Mench.  Why it is important to understand animal behavior.  ILAR journal, 39(1):20–26, 1998.
  * [34] Jiteng Mu, Weichao Qiu, Gregory D Hager, and Alan L Yuille.  Learning from synthetic animals.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12386–12395, 2020.
  * [35] Sauradip Nag, Xiatian Zhu, Yi-Zhe Song, and Tao Xiang.  Temporal action detection with global segmentation mask learning.  European Conference on Computer Vision, 2022.
  * [36] Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu.  Animal kingdom: A large and diverse dataset for animal behavior understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19023–19034, 2022.
  * [37] Kehinde Owoeye and Stephen Hailes.  Online collective animal movement activity recognition.  arXiv preprint arXiv:1811.09067, 2018.
  * [38] Shah Atiqur Rahman, Insu Song, Maylor KH Leung, Ickjai Lee, and Kyungmi Lee.  Fast action recognition using negative space features.  Expert Systems with Applications, 41(2):574–587, 2014.
  * [39] Maheen Rashid, Xiuye Gu, and Yong Jae Lee.  Interspecies knowledge transfer for facial keypoint detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6894–6903, 2017.
  * [40] Lianne Robinson and Gernot Riedel.  Comparison of automated home-cage monitoring systems: emphasis on feeding behaviour, activity and spatial learning following pharmacological interventions.  Journal of neuroscience methods, 234:13–25, 2014.
  * [41] Moira Shooter, Charles Malleson, and Adrian Hilton.  Sydog: A synthetic dog dataset for improved 2d pose estimation.  arXiv preprint arXiv:2108.00249, 2021.
  * [42] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 4489–4497, 2015.
  * [43] Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank van Langevelde, Tilo Burghardt, et al.  Perspectives in machine learning for wildlife conservation.  Nature communications, 13(1):1–15, 2022.
  * [44] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 595–604, 2015.
  * [45] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie.  The inaturalist species classification and detection dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8769–8778, 2018.
  * [46] Grant Van Horn, Rui Qian, Kimberly Wilber, Hartwig Adam, Oisin Mac Aodha, and Serge Belongie.  Exploring fine-grained audiovisual categorization with the ssw60 dataset.
  * [47] Grant Van Horn, Rui Qian, Kimberly Wilber, Hartwig Adam, Oisin Mac Aodha, and Serge Belongie.  Exploring fine-grained audiovisual categorization with the ssw60 dataset, 2022.
  * [48] Lukas von Ziegler, Oliver Sturman, and Johannes Bohacek.  Big behavior: challenges and opportunities in a new era of deep behavior profiling.  Neuropsychopharmacology, 46(1):33–44, 2021.
  * [49] Carl Vondrick, Donald Patterson, and Deva Ramanan.  Efficiently scaling up crowdsourced video annotation.  International journal of computer vision, 101(1):184–204, 2013\.
  * [50] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona.  Caltech-ucsd birds 200.  2010\.
  * [51] Heng Yang, Renqiao Zhang, and Peter Robinson.  Human and sheep facial landmarks localisation by triplet interpolated features.  In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1–8. IEEE, 2016.
  * [52] Can Zhang, Meng Cao, Dongming Yang, Jie Chen, and Yuexian Zou.  Cola: Weakly-supervised temporal action localization with snippet contrastive learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16010–16019, June 2021.
  * [53] Chenlin Zhang, Jianxin Wu, and Yin Li.  Actionformer: Localizing moments of actions with transformers.  In European Conference on Computer Vision, 2022.