# Navigating Decision Landscapes: The Impact of Principals on Decision-Making
Dynamics

Lu Li  Huangxing Li

###### Abstract

We explored decision-making dynamics in social systems, referencing the ’herd
behavior’ from prior studies where individuals follow preceding choices
without understanding the underlying reasons. While previous research
highlighted a preference for the optimal choice without external influences,
our study introduced principals or external guides, adding complexity to the
decision-making process. The reliability of these principals significantly
influenced decisions. Notably, even occasional trust in an unreliable
principal could alter decision outcomes. Furthermore, when a principal’s
advice was purely random, heightened trust led to more decision errors. Our
findings emphasize the need for caution when placing trust in decision-making
contexts.

###### keywords:

herd behavior , sequential decision-making

††journal:

[inst1]organization=Microsoft,addressline=One Microsoft Way, city=Redmond,
postcode=98052, state=WA, country=US

##  1 Introduction

Sequential decision-making is a cornerstone of cognitive science, economics,
and information theory. It encapsulates the intricate process by which
individuals navigate multifaceted environments. In today’s digital age, where
we are constantly bombarded with vast amounts of data and information from
various sources, decisions are seldom made in a vacuum. Instead, they often
result from a series of preceding choices, both personal and those observed
from others.

A pivotal element in this decision-making process is the role of principal
signals. These are vital cues or pieces of information that steer the
decision-making trajectory. For instance, consider the financial market. Here,
a principal could be a renowned financial analyst or institution whose
forecasts and recommendations significantly influence investor behavior. When
this analyst predicts a bullish trend for a particular stock, many investors,
especially those who trust or follow this analyst, might decide to buy the
stock based on this principal signal, even if they haven’t done their own
thorough research. Such principal signals, whether they originate from expert
opinions, media reports, or influential figures, play a crucial role in
shaping public opinion and actions.

The ability to discern these signals, coupled with insights from prior
decisions, paves the way for more strategic and informed choices. Thus, the
significance of sequential decision-making is profound, offering insights into
the complex interplay of cognition, information, and social influences in
shaping individual choices.

In this study, we embark on an exhaustive journey, addressing several pivotal
research questions that illuminate the intricacies of decision-making under
diverse scenarios. Our initial exploration probes the influence of principals
on decision convergence. Do certain guiding signals from principals lead to
more harmonized decisions? We then investigate how varied principals’
settings, their reliability, and the trust placed in them sculpt the decision-
making path.

Our research findings, derived from extensive simulations, reveal a nuanced
landscape of decision-making influenced by principal signals. We observed that
the presence of a reliable principal can expedite convergence towards optimal
decisions. However, when the principal’s guidance is inconsistent or
misleading, it can introduce significant variability into the decision-making
process, sometimes leading to suboptimal outcomes. Furthermore, the degree of
trust vested in the principal emerged as a critical factor. Even occasional
trust in a predominantly unreliable principal can profoundly alter decision
trajectories, underscoring the importance of discernment in decision-making
trust allocation. Conversely, when the principal’s guidance was statistically
random, an intriguing inverse relationship between trust in the principal and
decision accuracy was noted.

These findings have broader implications for various sectors, from finance to
public policy, emphasizing the need for critical evaluation of external cues
and the potential consequences of misplaced trust. In a world where decisions
can have cascading effects, understanding the dynamics of sequential decision-
making becomes paramount.

##  2 Background

Observational learning is crucial in deciphering the nuances of option
adoption behaviors. Song and Walden’s work on the adoption of peer-to-peer
technologies underscores the importance of observational learning [1, 2].
Walden and Browne’s research further elucidates the impact of observational
learning on investor behaviors, particularly in their reactions to electronic
commerce announcements [3]. Delving into the realm of herd behavior in
observational learning, Çelen and Shachar spotlighted the phenomenon of
information cascades [4]. This aligns with the broader framework of technology
adoption decision-making models, which offer insights into the dynamics of
technology adoption and the factors that drive herd behaviors [5].

The nuanced interplay between principals and decision-making processes has
garnered significant attention in academic circles. Kamenica and Gentzkow’s
introduction of Bayesian persuasion is a cornerstone in this field,
emphasizing optimal signaling in the face of incomplete information in static
contexts [6]. This model has found resonance across various domains. For
instance, implications of this model in the realms of security protocols and
voting mechanisms have been explored by Rabinovich et al. [7]. Concurrently,
Xu et al. and Goldstein and Leitner delved into its ramifications on
investment strategies and market trends [8, 9].

As Bayesian persuasion evolved, it paved the way for dynamic models.
Foundational works by Ely and Renault et al. set the stage for this
progression [10, 11]. Cutting-edge research by Celli et al. and Castiglioni et
al. tackled the algorithmic intricacies inherent in these dynamic models [12,
13, 14]. A fascinating tangent to this is the concept of automated mechanism
design, a topic broached by Conitzer and Sandholm [15, 16]. Building on this,
Zhang and Conitzer offered a fresh perspective, accentuating the dynamism
inherent in principal-agent interactions [17].

##  3 Principle-WB model

###  3.1 Assumptions

For research in binary adoption processes, it’s crucial to grasp the influence
of a principle’s signal on sequential decision-making. We’ve built upon the
model assumptions set forth by [5] and [18], introducing additional
assumptions to account for the presence of a principle. This enhanced model is
denoted as the P-WB model. The following are our assumptions:

  1. 1.

Decision-makers are presented with two distinct options: $A$ and $B$.

  2. 2.

An unbounded sequence of decision-makers confronts the identical challenge of
choosing between $A$ and $B$ under comparable situations. These decisions
occur sequentially and at distinct time intervals. Furthermore, the decision-
making order is exogenous, ensuring that one decision-maker’s choice does not
influence the sequence or choices of others.

  3. 3.

For every decision-maker, the benefits of options $A$ and $B$ remain
consistent and are unaffected by the timing of the decision-making process.

  4. 4.

Each decision maker possesses confidential data regarding the advantages of
options $A$ and $B$. Specifically, they obtain an objective signal that
follows a Gaussian distribution. Given that options $A$ and $B$ excel under
distinct scenarios, the objective signals they receive originate from separate
distributions. If technology $A$ surpasses $B$, the associated signal stems
from $N(\mu_{A},\sigma^{2})$. Conversely, if option $B$ outperforms option
$A$, the signal stems from $N(\mu_{B},\sigma^{2})$. Assuming, for simplicity,
that $\mu_{A}>\mu_{B}$, we use $d=\frac{\mu_{A}-\mu_{B}}{\sigma}$ to gauge the
disparity between these distributions. For an informed decision, the decision
maker must discern which distribution the received objective signal is more
likely derived from.

  5. 5.

Apart from the objective signal, every decision maker also obtains a signal
from a principal. This principal signal carries an inherent bias. To
streamline our discussion, we represent this bias as the probability
$P_{\text{bias}}$ associated with the principal opting for option $A$. If the
principal selects $A$, the principal signal originates from
$N(\mu_{A},\sigma_{\text{P}}^{2})$. Conversely, if option $B$ is chosen, the
principal signal originates from $N(\mu_{B},\sigma_{\text{P}}^{2})$.

  6. 6.

In addition to the objective signal and the principal signal, decision-makers
are aware of the initial conditions and can monitor the choices made by those
before them. These initial conditions specify the prior probabilities of
option A being superior to B and vice versa, represented as $P_{\mu_{A}}$ and
$P_{\mu_{B}}$ respectively. For simplicity, we set
$P_{\mu_{A}}=P_{\mu_{B}}=0.5$, suggesting an absence of pre-existing bias.
While decision-makers can observe others’ actions, the underlying motivations
remain opaque.

  7. 7.

Decision-makers, driven by rationality, ground their choices in available
information. The decisions don’t diverge from this data-driven approach due to
mere personal inclinations. This aligns with the rationality assumption.

Given the assumptions enumerated above, our P-WB model provides a nuanced
representation of the decision-making process faced by sequential decision-
makers operating under uncertainty. These assumptions encapsulate the
necessity of leveraging both objective and subjective signals, as well as
previous decisions made by others in determining the optimal choice between
options $A$ and $B$.

As we proceed, we will delve deeper into how these assumptions play out in
real-world scenarios, the potential pitfalls they might present, and the
strategies decision-makers can employ to navigate these challenges
effectively.

##  4 Model Formalization with Principal Influence

In our refined model, decision-makers receive a continuous objective signal
$s_{o}$ from a Gaussian distribution, with a mean represented as either
$\mu_{A}$ or $\mu_{B}$. This distinction signifies the superiority of option
$A$ over option $B$ or vice versa. In addition to this objective signal,
decision-makers also receive a principal signal $s_{p}$ emanating from a
Gaussian distribution with identical mean values, $\mu_{A}$ or $\mu_{B}$, but
different standard deviations, denoted as $\sigma_{p}$. This principal
influence can potentially modulate the decision-making trajectory.

To make an informed choice, decision-makers need to discern from which
Gaussian distribution their signal most likely originates, taking into account
both the objective evidence and insights obtained from the choices of their
predecessors. The principal’s signal, with its unique standard deviation,
introduces another layer of complexity to this evaluation.

A decision-maker will be inclined towards option $A$ if the following
relationship is satisfied:

$\frac{p(s_{o}|\mu_{A},s_{p})}{p(s_{o}|\mu_{B},s_{p})}\geq\beta$

Using the Gaussian distribution density function, a decision threshold for
$s_{o}$ is derived as:

$r(\beta)=\frac{\sigma^{2}}{\mu_{A}-\mu_{B}}\ln\beta+\frac{\mu_{A}+\mu_{B}}{2}$

When $s_{o}$ is greater than $r(\beta)$, the decision-maker chooses option $A$
and When $s_{o}$ is less than $r(\beta)$, the decision-maker chooses option
$B$. Consequently, the following decision rule, termed the G-WB criterion, is
established:

$\displaystyle s_{o}$ $\displaystyle\geq r(\beta)$
$\displaystyle\Rightarrow\text{choosing }A$ $\displaystyle s_{o}$
$\displaystyle<r(\beta)$ $\displaystyle\Rightarrow\text{choosing }B$

This implies:

  * 1.

When $s_{o}$ exceeds $r(\beta)$, the decision-maker chooses $A$.

  * 2.

Otherwise, the decision-maker selects $B$.

The optimal threshold is selected by balancing the benefits of each technology
and the decision maker’s information regarding the probability that $A$ is
better than $B$ and that $B$ is better than $A$. In this case, the optimal
value of $\beta$ is

$\beta=\frac{p_{\mu_{B}}(\text{Benefit}(b|\mu_{B})-\text{Benefit}(a|\mu_{B}))}{p_{\mu_{A}}(\text{Benefit}(a|\mu_{A})-\text{Benefit}(b|\mu_{A}))}=\frac{p_{\mu_{B}}}{p_{\mu_{A}}}k$
(1)

where

$k=\frac{\text{Benefit}(b|\mu_{B})-\text{Benefit}(a|\mu_{B})}{\text{Benefit}(a|\mu_{A})-\text{Benefit}(b|\mu_{A})}$
(2)

$k$ represents the relative difference between the values of the two
technologies in different states of the world. The numerator is the difference
between the benefit of choosing $B$ and the benefit of choosing $A$ in the
state where the technology $B$ is better than $A$. The denominator is the
difference between the benefit of choosing $A$ and the benefit of choosing $B$
in the state where the technology $A$ is better than $B$. According to the
assumption #3, both the numerator and denominator are positive numbers and $k$
is a constant.

Incorporating this signal into the Bayesian update:

$\displaystyle p_{\mu_{A}\cdot(t+1)}$
$\displaystyle=p(\mu_{A}|D_{t},D_{t-1},\dots,D_{1},s_{p})=\frac{p_{\mu_{A}}\cdot
p(D_{t},D_{t-1},\dots,D_{1}|\mu_{A})p(s_{p}|\mu_{A})}{p(D_{t},D_{t-1},\dots,D_{1})p(s_{p})}$
(3) $\displaystyle p_{\mu_{B}\cdot(t+1)}$
$\displaystyle=p(\mu_{B}|D_{t},D_{t-1},\dots,D_{1},s_{p})=\frac{p_{\mu_{B}}\cdot
p(D_{t},D_{t-1},\dots,D_{1}|\mu_{B})p(s_{p}|\mu_{B})}{p(D_{t},D_{t-1},\dots,D_{1})p(s_{p})}$
(4)

Where $p_{\mu_{A}\cdot(t+1)}=p(\mu_{A}|s_{p},D_{t},D_{t-1},\dots,D_{1})$ and
$p_{\mu_{B}\cdot(t+1)}=p(\mu_{B}|s_{p},D_{t},D_{t-1},\dots,D_{1})$. $s_{p}$ is
independent with previous observed options $D_{t},D_{t-1},\dots,D_{1})$ , Thus

$\beta_{t+1}=\frac{p_{\mu_{B}\cdot(t+1)}}{p_{\mu_{A}\cdot(t+1)}}k=\frac{p(D_{t},D_{t-1},\dots,D_{1}|\mu_{B})p(s_{p}|\mu_{B})}{p(D_{t},D_{t-1},\dots,D_{1}|\mu_{A})p(s_{p}|\mu_{A})}\left[\frac{p_{\mu_{B}}}{p_{\mu_{A}}}k\right]$
(5)

and define $A_{t}$ as $A_{t}=\\{D_{t-1},D_{t-2},\dots,D_{1}\\}$, the set of
all prior decisions for all $t>1$, then

$\displaystyle p(D_{t},D_{t-1},\dots,D_{1}|\mu_{A})$
$\displaystyle=p(D_{t}|A_{t},\mu_{A})p(D_{t-1},\dots,D_{1}|\mu_{A})$ (6)
$\displaystyle p(D_{t},D_{t-1},\dots,D_{1}|\mu_{B})$
$\displaystyle=p(D_{t}|A_{t},\mu_{B})p(D_{t-1},\dots,D_{1}|\mu_{B})$ (7)

Thus, we obtain the iterative formula:

$\beta_{t+1}=\frac{p(D_{t}|A_{t},\mu_{B})p(D_{t-1},\dots,D_{1}|\mu_{B})}{p(D_{t}|A_{t},\mu_{A})p(D_{t-1},\dots,D_{1}|\mu_{A})}\left[\frac{p(s_{p}|\mu_{B})p_{\mu_{B}}}{p(s_{p}|\mu_{A})p_{\mu_{A}}}k\right]=\frac{p(D_{t}|A_{t},\mu_{B})p(s_{p}|\mu_{B})}{p(D_{t}|A_{t},\mu_{A})p(s_{p}|\mu_{A})}\beta_{t}$
(8)

In this manner, we can obtain the threshold of the t-th decision maker and the
probability of the t-th decision maker’s choice after observing the decisions
made by the previous decision-makers and the signal from the principle.

##  5 Data and Results

###  5.1 Decision Dynamics Absent of Principal Influence

Within the realm of social systems, decision-making dynamics often manifest
complex patterns. A notable example is the ’herd behavior’, where, over time,
a majority of participants converge towards a single choice. It’s noteworthy
that subsequent decision-makers often base their choices solely on the
observed decision, without access to the underlying rationale that informed
it.

For a clearer understanding of this dynamic, we set our model parameters as:
$\mu_{A}=1$, $\mu_{B}=0$, $\sigma=1$, and $k=1$. We assumed that the objective
signal is uniformly drawn from $N(\mu_{A},\sigma^{2})$. The outcomes from this
configuration align with the foundational works of [5]. Specifically, there
was a consistent rise in the proportion of participants opting for choice $A$.
This aligns with our model’s design, where the decision threshold, denoted by
$r(\beta)$, progressively shifts towards $\mu_{B}$, diminishing the influence
of the objective signal in subsequent decisions.

Fig.1 vividly portrays the proportion of decision-makers selecting the correct
choice, as a function of decision count. This pattern of convergence towards
the correct choice, which intensifies with an increasing number of decision-
makers, echoes the findings of [18, 5].

Figure 1: Convergence trends in the WB model without principal influence.

###  5.2 Decision Dynamics Influenced by Unwaveringly Trusted Principals

Introducing principals into the decision-making framework adds layers of
complexity. Notably, in our model, principals aren’t necessarily reliable
sources of the correct signal, here denoted as option $A$. We utilize $p_{b}$
to represent the likelihood that a principal’s advice originates from the
correct signal distribution $N(\mu_{A},\sigma^{2})$. If not, the advice aligns
with the incorrect signal distribution $N(\mu_{B},\sigma^{2})$.

Fig.2 offers a comparative study across various $p_{b}$ values. The influence
of the principal is palpable, as the decision-makers’ convergence trajectory
seems significantly influenced by the principal’s signal accuracy. This
influence remains pronounced even when private signals consistently arise from
$N(\mu_{A},\sigma^{2})$. For a comprehensive comparison, the principal-free
scenario from Sec.5.1 is also depicted.

Figure 2: Dynamics under the influence of the P-WB model.

A thorough assessment reveals several insights:

  1. 1.

Principals with a high accuracy rate expedite convergence towards the optimal
decision.

  2. 2.

In contrast, principals prone to errors can misguide decisions towards less
optimal outcomes.

  3. 3.

When the principal’s accuracy surpasses a critical 50% threshold, the initial
convergence might be gradual, but eventually surpasses the baseline,
highlighting the principal’s dominant influence.

  4. 4.

Interestingly, at $p_{b}=0.5$, decision patterns stabilize, showing no evident
convergence.

###  5.3 Decision Dynamics Influenced by Conditionally Trusted Principals

In Sec.5.2, we considered a principal that consistently garners trust from the
decision-maker, illustrating how decision convergence is largely governed by
the principal’s influence. Here, we delve into a more intricate scenario: the
principal isn’t invariably trusted, irrespective of its signal’s accuracy. In
this context, $p_{t}$ denotes the likelihood of the decision-maker trusting
the principal’s signal. When trusted, the decision boundary is adjusted as per
Eq.8. If not, the decision-maker disregards the principal’s signal, updating
the decision boundary based on [18]:

$\beta_{t+1}=\frac{p(D_{t}|A_{t},\mu_{B})}{p(D_{t}|A_{t},\mu_{A})}\beta_{t}$
(9)

Figure 3: Decision dynamics under varying trust levels.

Our analysis first contrasts results across different $p_{t}$ values, given
$p_{b}=0.3$. This setting implies the principal’s guidance is accurate 30% of
the time, and even so, is trusted only 10% of the time by decision-makers. The
outcomes, as visualized in Fig.3, are compelling. Initially, there’s a rise in
the percentage of correct decisions, hinting at decision-makers benefiting
from prior accurate decisions. Yet, as decisions accumulate, this trend
inverts, leading to a decline in decision accuracy. This unexpected result
emphasizes a crucial observation: even a predominantly misleading principal,
if occasionally trusted, can profoundly alter decision-making trajectories.
This effect can be traced back to prior studies indicating that accurate
guidance from a trusted principal reinforces convergence towards the correct
decision. However, in this context, as more decisions are swayed by the
principal’s erroneous guidance, the overall decision accuracy wanes.

This finding aligns with our study’s overarching theme, highlighting the
significant influence of external entities, like principals, on sequential
decision-making. Even when such influences are intermittent and potentially
unreliable, they can shape collective decision-making patterns, emphasizing
the importance of discernment in decision-making trust allocation.

In a subsequent phase of our analysis, we set $p_{b}$ to 0.5, suggesting that
the principal’s guidance is accurate half the time, making it statistically
random. Fig.4 charts decision-making accuracy against diverse $p_{t}$ values.
The results reveal an unexpected inverse correlation: as trust in the
principal grows, decision accuracy recedes. This suggests that increased trust
in a statistically random principal can inadvertently misguide decision-
makers.

While one might anticipate a neutral impact from a random principal on
decision-making, the data indicates otherwise. Entrusting an external source,
even one offering statistically random guidance, can introduce variability
into the decision-making process. This variability, when compounded over
successive decisions, can lead to deviations from optimal decision pathways.

This phase of the analysis underscores the intricate interplay between trust
and decision-making accuracy. Even without a discernible bias in external
guidance, the mere act of entrusting it can introduce uncertainties, affecting
the broader decision-making framework. This emphasizes the importance of
prudence in trust allocation, especially when the reliability of external
guidance is ambiguous or unverified.

Figure 4: Decision dynamics under varying trust levels with a 50% bias
parameter.

##  6 Summary

Our comprehensive analysis delved into the intricate dynamics of decision-
making, especially within social systems. We explored the phenomenon of ’herd
behavior’, as described in previous works, where participants tend to converge
towards a singular choice over time. This behavior was characterized by
subsequent decision-makers basing their choices solely on the observed
decision, without access to the underlying rationale.

In scenarios devoid of principal influence, our model’s outcomes aligned with
foundational works, showcasing a consistent preference for optimal option.
This was attributed to the model’s design, where the decision threshold
progressively shifts, reducing the influence of the objective signal in
subsequent decisions.

However, the introduction of principals, entities that may or may not provide
reliable signals, added layers of complexity. The decision-making trajectory
was significantly influenced by the principal’s signal accuracy. Even when
private signals were consistently accurate, the principal’s influence remained
dominant.

In scenarios where the principal wasn’t consistently trusted, even occasional
trust in a predominantly misleading principal could profoundly alter decision-
making trajectories. This emphasized the significant influence of external
entities on sequential decision-making and the importance of discernment in
decision-making trust allocation.

Interestingly, when the principal’s guidance was statistically random, an
inverse relationship between trust in the principal and decision accuracy
emerged. This highlighted the potential pitfalls of placing undue trust in
external sources, especially when their reliability is uncertain.

In essence, our study underscores the nuanced interplay between trust,
external influences, and decision-making accuracy. It emphasizes the
importance of prudence in trust allocation and the potential consequences of
misplaced trust, especially in complex decision-making landscapes.

## References

  * [1] J. Song, E. Walden, Consumer behavior in the adoption of peer-to-peer technologies: an empirical examination of information cascades and network externalities (2003).
  * [2] L. Li, Z. He, X. Zhou, D. Yu, How to evaluate the next system: Automatic dialogue evaluation from the perspective of continual learning (2019).  arXiv:1912.04664.
  * [3] E. Walden, G. Browne, Rational fads in investor reactions to electronic commerce announcements: an explanation of the internet bubble, Electron. Commer. Res. Appl. 7 (2008) 44–54.
  * [4] B. Çelen, K. Shachar, Observational learning under imperfect information, Games Econ. Behav. 47 (2004) 72–86.
  * [5] J. Gu, L. Li, Z. Xu, H. Fujita, Construction of a technology adoption decision-making model and its extension to understanding herd behavior, Knowledge-Based Systems 89 (2015) 471–486.  doi:https://doi.org/10.1016/j.knosys.2015.08.014.
URL https://www.sciencedirect.com/science/article/pii/S0950705115003251

  * [6] E. Kamenica, M. Gentzkow, Bayesian persuasion, American Economic Review 101 (6) (2011) 2590–2615.
  * [7] Z. Rabinovich, et al., Information disclosure as a means to security (2015) 645–653.
  * [8] H. Xu, et al., Exploring information asymmetry in two-stage security games, Manage. Inform. Syst. Quart. 33 (2015) 23–48.
  * [9] I. Goldstein, Y. Leitner, Stress tests and information disclosure, Journal of Economic Theory 177 (2018) 34–69.
  * [10] J. Ely, Beeps, American Economic Review 107 (1) (2017) 31–53.
  * [11] J. Renault, E. Solan, N. Vieille, Optimal dynamic information provision, Games and Economic Behavior 104 (2017) 329–349.
  * [12] A. Celli, et al., Private bayesian persuasion with sequential games, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 1886–1893.
  * [13] M. Castiglioni, et al., Online bayesian persuasion, Advances in Neural Information Processing Systems 33 (2020) 16188–16198.
  * [14] M. Castiglioni, et al., Multi-receiver online bayesian persuasion, in: Proceedings of 38th International Conference on Machine Learning, 2021, pp. 1314–1323.
  * [15] V. Conitzer, T. Sandholm, Complexity of mechanism design, in: Proceedings of the 18th Conference in Uncertainty in Artificial Intelligence, 2002, pp. 103–110.
  * [16] V. Conitzer, T. Sandholm, Self-interested automated mechanism design and implications for optimal combinatorial auctions, in: Proceedings of the 5th ACM Conference on Electronic Commerce, 2004, pp. 132–141.
  * [17] H. Zhang, V. Conitzer, Automated dynamic mechanism design, Advances in Neural Information Processing Systems 34 (2021).
  * [18] E. Walden, G. Browne, Sequential adoption theory: A theory for understanding herding behavior in early adoption of novel technologies, J. AIS 10 (01 2009).  doi:10.17705/1jais.00181.


# THORN: Temporal Human-Object Relation Network for Action Recognition

Mohammed Guermal, Rui Dai, and François Brémond  Inria, Université Côte
d’Azur, 2004 Route des Lucioles, 06902 Valbonne
{mohammed.guermal, rui.dai<EMAIL_ADDRESS>
###### Abstract

Most action recognition models treat human activities as unitary events.
However, human activities often follow a certain hierarchy. In fact, many
human activities are compositional. Also, these actions are mostly human-
object interactions. In this paper we propose to recognize human action by
leveraging the set of interactions that define an action. In this work, we
present an end-to-end network: THORN, that can leverage important human-object
and object-object interactions to predict actions. This model is built on top
of a 3D backbone network. The key components of our model are: 1) An object
representation filter for modeling object. 2) An object relation reasoning
module to capture object relations. 3) A classification layer to predict the
action labels. To show the robustness of THORN, we evaluate it on EPIC-
Kitchen55 and EGTEA Gaze+, two of the largest and most challenging first-
person and human-object interaction datasets. THORN achieves state-of-the-art
performance on both datasets.

##  I Introduction

Human activity recognition in video is a fundamental problem in computer
vision, due to its large field of applications, such as human-computer
interaction [1] or video surveillance [2]. Machine learning and computer
vision models have achieved interesting results in this field. Unfortunately,
most of the State-of-the-art methods focus on simple activities such as
walking or drinking, while the recognition of longe-term, complex, and
composite activities such as assembling furniture or food preparation has been
rarely addressed. These methods make use of end-to-end models that produce a
video level label, and do not explicitly decompose the action into a
hierarchical set of sub-actions or interactions. Moreover, neuroscience [3, 4]
has shown that the human perception of action is actually based on decomposing
an action into different groups of interactions which enables him/her to
understand other human behaviors. In this paper we decide to visit this
composite actions, that we refer to as actions of Human-Object Interaction
(HOI). Not only that we also focus on first-person view HOI action
recognition.

first-person action recognition also comes with its challenges, one of which
is the narrow field of view that makes actions sometimes happen outside the
video viewing range. Also, the huge ego-motion caused by the sharp movements
of the camera can make it harder to recognize actions. Finally, in ego-vision,
the field of view usually covers the human hands and an ensemble of objects.
In this case, actions are generally involving interactions between the human
and objects. Hence the challenge is also to recognize which of these objects
are relevant to the action and which are distractors.

A HOI action can be seen as combination of verbs and nouns, for instance the
action cutting bread with knife is the combination of the verb cut and the
nouns knife and bread. Hence recognizing an action of HOI, is a class of
visual relationship detection, where the task is to not only recognize the
objects (the noun), but also to infer the relation and motion (the verb)
between different objects and the human. Fig. 1 represents an example of an
object-based action: wash plate. Such action requires highlighting objects
like the hand, the plate and the tap while giving less attention to other
objects that are not important to the action.

Figure 1: An example of the Human-Object Interactions of wash plate in an
first-view video. Green arrows represent interactions at the same time step
(i.e., spatial relation) while black arrows represent interactions across
time. In practice, the model captures all the objects detected. For
simplicity, here we highlight only the relevant objects to wash plate.

Previous works such as two-stream CNNs [5],[6],[7] or 3D CNNs [8] [9] [10]
have achieved very good results on third view and video level label datasets
[11] [12] [13] [14]. However, when it comes to HOI actions they still lack in
performance. That is due mainly to the fact that CNNs capture shareable local
features in the image/videos, and they can not handle complex or fine-grained
actions. Another major challenge is the fact that such activities can often be
performed in a wide variety, making it harder for CNNs to learn significant
patterns.

Thus, our intuition is to build a model that can, extract detailed and object
specific semantics in the videos, as well as explore the cross-object relation
at different time-steps. By doing so we can firstly, improve object
recognition in actions of HOI (the noun). Moreover, we can refine the motion
recognition (the verb) by having a clearer idea about the interaction of these
objects and their roles in the action. Finally, by encoding the scenes into a
graph of objects interactions, we make it easier to learn patterns for actions
even if they have many variations, since the interactions are usually the
same.

To step-up to the aforementioned challenges, we propose a new module built on
top of 3D-CNNs, this module is divided into two sub parts. Firstly, we design
an Object Representation filter. This first sub-module acts as a filter that
retrieves specific and object-related semantics from the overall and mixed
representation (extracted from the 3D-CNN). Secondly, we add an Object
Relation Reasoning module that uses the detailed representations to explore
cross-objects relations (interactions). Finally, we obtain an object-centric
model that can predict actions of HOI by exploring human-object and object-
object interactions.

To summarize, our main contributions are:
1\. A model that can find and extract detailed semantics of specific objects;
2- A graph-based module capable of exploring interactions between different
objects.

##  II Related work

Human-object interaction action recognition became the focus of many research
subjects lately, especially with the development of important datasets such as
[15, 16, 17]. Several approaches have been proposed to tackle this
problematic. In the following, we review some of these approaches.

###  II-A 3D-CNNs

3D-CNNs methods focus on getting the overall appearance of the videos without
considering the objects interactions. Since these methods cannot capture
specific or detailed semantics, they are still limited in case of actions of
HOI. Making this architectures more adequate to video level labels. We cite as
an example I3D [9]. Although it achieves good results on many action
recognition datasets, its performance is still poor on actions of HOI. To
improve the performances on these 3D-CNNs, Long Features Bank [18] for
instance, tries to capture HOI actions by extracting and fusing features from
local clips as well as globally from the whole video. This method uses object
detection and ROI-Align to capture detected object features. And though they
successfully capture richer features and more temporal information, they fail
to do any object interaction modeling. Hence, they cannot improve much on HOI
actions. In the same direction, Temporal Binding Networks (TBN) [19] proposes
to capture local clip features from different clips and fuse them for later
prediction. In addition to that, TBN uses multi-modalities as they capture
audio-visual features using audio, RGB, and optical flow. However, we believe
that this multi-modality will not always bring much information about the
objects. sounds can be very noisy and very similar which can confuse the
prediction. Moreover, fusing multi-modalities can be hard and requires lot of
efforts the may not lead to significant improvements. Finally, other works
such as [20] use also multi-modality reasoning. However, we argue that HOI
actions recognition requires more focus on objects and their interactions.

###  II-B Graph’s Convolutions

Recently, graphs have also been considered a way for solving action
recognition [21, 22, 23, 24].

As for human-object interaction, videos as a space-time region graph [21]
propose to model the interaction between objects and humans in two steps as
they build two different graphs. This allows to correlate objects across
space-time. Similarly, in [25], the authors construct the nodes of the graph
with consideration to the node class. For instance, the node for the scene is
computed using the aforementioned I3D. While for objects, they use the Faster-
RCNN network [26] trained on MS COCO. All these methods mentioned above try to
define their nodes by using ROI-Align. However, this is not optimal as,in most
cases, multiple objects are present at the scene and some of them are too
close to each other. In this case, the projected coordinates of different
objects tend to be in the same set of pixels. Therefore, extracting an
object’s specific feature from a feature map with low resolution becomes
difficult. Not only that these methods rely on pre-trained object detetctors,
hence they can not leverage only objects relevant to the action. Whereas in
our work, we learn to filter only relevant objects and learn specific
representation to different object-classes in an end-to-end way.

In the domain of semantic modelling, Class Temporal Relational Network (CTRN)
[24] is proposed for the action detection tasks. However, CTRN is a two steps
method, which is built on top of pre-extracted flattened 1-dimensional
features. The dissociation between the visual encoder and temporal module
makes the model overlook the appearance and spatial information in the video,
while such information is critical to the HOI action recognition. In this
work, we propose a one-step method THORN for HOI action recognition. Different
from CTRN, our method leverages the object detector to extract the object
semantics directly from the spatio-temporal features. After that, graph
reasoning is applied to refine the object representation and to jointly model
inter-object relations. This design allows the model to capture the latent
relations among the objects in the videos, which results in higher accuracy in
HOI action recognition.

Figure 2:  THORN architecture contains three main components: (1) a Visual
encoder (i.e., X3D) encodes the input RGB clip into a primary spatio-temporal
representation. (2) The obtained representation is fed to the Object
Representation Filter, which maps the previous representation into object-
class representation. To ensure a discriminative object representation, an
object classifier is added on top of the object-class representation. This
classifier is trained with the pseudo-object ground truth provided by an
object detector. (3) The object-class representation is also sent to the
Object Relation Reasoning module to model the temporal-object relation in a
dissociated manner. Finally, two classifiers are used to predict the verbs and
nouns relevant to the action.

##  III Proposed Method

In this section, we detail each sub-part of the proposed model, THORN. The
main components in this model are: a 3D Visual Encoder which encodes the video
into a spatio-temporal embedding. Then, the previously extracted embeddings
are passed to the Object Representation Filter (ORF). This filter extracts
class-specific features. Finally, the Object Relation Reasoning module
computes the relation between the different objects to predict the action.
Fig. 2 provides an overview of the model.

###  III-A Visual Encoder

We start by using a visual encoder to extract an embedding that serves as a
full understanding of the scene, and carries the global information of the
input frames. We choose X3D [27] as our visual encoder. X3D has many
advantages as it does not do any temporal pooling and keeps the full temporal
information, providing richer temporal information. Moreover, X3D is a lighter
model compared to other architectures such as I3D [9]. The input to the 3D
encoder is a set of video-clip frames. The output is a spatio-temporal
representation $F$ of shape $(T\times H^{\prime}\times
W^{\prime}\times{D_{1}})$, where: $H^{\prime}$ = $W^{\prime}$ = 7, $D_{1}$ =
432, while $T$ is the same as the input.
This embedding carries both spatial and temporal information. The spatial
information is important, as it provides object related information, such as
its appearance, shape and position (e.g. drawers usually appear at the bottom
of the image). That is why instead of using the X3D final output of shape
$(T\times 2048)$ to construct our nodes, we use a finer spatial representation
of shape $(T\times 7\times 7\times 432)$, making nodes of our graph contain
more and finer information about the objects. We provide more details on this
in the ablation study, by comparing both settings. Finally, as X3D is a light-
weighted model it is easier to train the Visual Encoder jointly with the
following modules.

###  III-B Object Representation Filter

Figure 3: Representation of our Object Representation Filter (ORF). The input
is the feature map from the 3D encoder reshaped to $T\times
H^{\prime}W^{\prime}D$ and the duplicated $C_{o}$ times, where $C_{o}$ is the
number of classes. Finally, we have a representation specific to each object
class.

Our main objective through this work is to have object-based reasoning. Hence
the first step is to obtain object in scene representations. Therefore, we
developed the Object Representation Filter module, capable of extracting
semantic representation specific to each object class from the previous
overall representation. This module serves as a filter to obtain the object-
specific representation from the output of visual encoder. In practice,
firstly, we reshape the representation $F$ from the visual encoder to shape
$(T\times H^{\prime}W^{\prime}{D_{1}})$. After that, we duplicate the reshaped
features $F^{\prime}$ for $C_{o}$ times, where $C_{o}$ indicates the number of
object classes in the dataset. For each class, we use a channel-mixer MLP
(i.e., linear transformation layer), followed by non-linear activation and
dropout. In Fig. 3, we show an overview of the ORF module. We argue that each
MLP layer learns to filter features specific to a certain object class. The
equations in this module can be formulated as:

$F^{\prime}_{i}=ReLU(MLP(F))$ (1)
$F^{\prime}=DropOut([F^{\prime}_{1},F^{\prime}_{2},F^{\prime}_{3}....,F^{\prime}_{C_{o}}])$
(2)

With $F^{\prime}$ $\in$ $\mathbb{R}^{T\times C_{o}\times D_{2}}$. Where
$D_{2}$ is smaller than $D_{1}$ to shallow the channel size. Finally, we add
another MLP layer on top of $F^{\prime}$ that would represent the object
classifier in Fig. 2.

$F^{\prime\prime}=ReLU(MLP(F^{\prime}))$ (3)

Here $F^{\prime\prime}$ $\in$ $\mathbb{R}^{T\times C_{o}\times 1}$. To ensure
the object-specific representation, we add a frame-level object classifier on
$F^{\prime\prime}$. As the frame-level object label is not provided by the
dataset, the object classifier is trained with the pseudo label provided by an
object detector (i.e. Fast-RCNN [26]). In the video, multiple objects can
appear in a frame, thus, we train the object classifier with binary cross-
entropy loss: $\mathcal{L}_{clip-objects}$. Finally the ORF module outputs a
representation for each object-class. However, we still need to correlate and
refine these object representations to explore their interactions and model
the actions. To do so, we introduce the next module of our pipeline in the
next section.

###  III-C Object Relation Reasoning Module

To correlate between the aforementioned representations in the previous
section, we introduce the Object Relation Reasoning Module.
In order to extract the relations between the filtered object classes, we
propose to make use of graph convolutions. In the previous section, we
transform the clip representation into a class-specific representation. Then,
we map it to a graph-like structure, where each vertex of the graph represents
an object class at a time step; the vertex would be the previously extracted
embedding of a certain class. In total, the graph consists of $C_{o}\times T$
nodes whose topology is defined by its vertex and an adjacency matrix
$A^{\prime}_{C_{o}}$. The adjacency matrix represents the connectivity or
relation between the different nodes (objects) and its weights represent how
strong their relationship is at different time steps. Fig. 4 represents an
overview of this module.

####  III-C1 Graph reasoning

The graph reasoning aims to do cross-class reasoning on the previously
constructed graph. The objects relations are video dependent, and so multiple
GCN blocks are stacked to learn multiple levels of semantics. Moreover, the
adjacency matrix is also parameterized so that it can be learned and optimized
with the pipeline during the training phase. Moreover, it can learn to adapt
to the data itself. We also make use of self-attention mechanisms.
Consequently, our adjacency matrix learns better to differentiate class
relations owing to different videos. Fig. 4 represents a block of the graph
convolution reasoning.

As the object relations are complex, it is hard to predefine the inter-object
relations for each video. Therefore, by leveraging the self-attention
mechanism [28, 22], our graph adjacency matrix is learnable and can vary with
the videos. In practice, the adjacency matrix $A_{C_{o}}$ is initialized with
a fully connected matrix. Finally, the full topology of our graph is
$A_{C_{o}}$ $\in$ $\mathbb{R}^{C_{o}\times C_{o}}$ and the vertexes
representation $G_{in}$ $\in$ $\mathbb{R}^{D_{2}\times T\times C_{o}}$. First,
we embed the input $G_{in}$ using bottleneck convolutional layer (i.e. 1 × 1),
then the output feature maps are rearranged into $\mathbb{R}^{D_{2}\times
T\times C_{o}}$ and $\mathbb{R}^{C_{o}\times D_{2}\times T}$ followed by a
matrix multiplication. The value of the resultant matrix is then normalized by
a softmax activation. Now, the superimposed adjacency matrix
$A^{\prime}_{C_{o}}$ can be formulated as:

$A^{\prime}_{C_{o}}=A_{C_{o}}+softmax(W_{1}^{T}G_{in}^{T}W_{2}G_{in})$ (4)

Where $W_{1}$ and $W_{2}$ are learnable weights of the bottleneck
convolutions, and $G_{in}$ being $F^{\prime}$ the stacked class
representations in section B. $G_{out}$, the output of the graph layer is
passed to the next graph layer and follows the same equations. In this work,
we use 5 blocks of graph convolutions. As for the $A^{\prime}_{C_{o}}$, each
value represents an edge between two nodes (objects). We learn a graph that is
shared across different time-steps but depends on each layer and for each
video, as we said earlier we learn different semantics at each level.
After bottleneck convolutions, we do the graph convolution operation with the
formulation in [29]:

$G_{out}=A^{\prime}_{C_{o}}G_{in}W_{3}$ (5)

$W_{3}$ is a learnable parameter where $W_{3}$ $\in$ $\mathbb{R}^{D_{2}\times
D_{2}}$. The equation 5 represents the message passing and node feature
updating, and finally $G_{out}$ is rearranged to $\mathbb{R}^{D_{2}\times
T\times C_{o}}$.
From equation 5, we can understand how graph convolutions work. The graph
convolutional layer represents each node as an aggregate of its neighborhood,
hence each node gathers information from its neighborhood and adapts itself
accordingly. In other words, at each graph block, each object collects
information about other objects and finally finds to which it is most
correlated, and thus whether there is an interaction or not. That is why we
judge that the use of graphs is a promising idea in this domain.

####  III-C2 TCN

stands for Temporal Convolution Network. The graph reasoning is capable of
extracting the relation between objects. However, in our study, we aim at
modelling the spatio-temporal interaction in a large time span. To do so, we
add a 1D convolution layer on top of the previous output of the graph
reasoning (i.e., $G_{out}$). As shown in Fig. 4, each Object Relation
Reasoning Module contains a TCN. This 1D-convolution layer is used to
aggregate the information across time. While stacking multiple object relation
reasoning blocks, each block is used to model the object relation in a
specific temporal scale. Finally, the output of the Object Relation Reasoning
Module is:

$G_{out}=Conv1D(G_{out})+G_{in}$ (6)

As mentioned earlier, the output of each block $G_{out}$ is the input $G_{in}$
to the next block.

Figure 4: Overview of one layer of the Object Relation Reasoning module, using
a graph architecture [22]. As we can see, the input is a graph representation
between different classes and the output is an updated representation of the
graph. The $\times$ $N_{block}$ stands for the number of blocks used in total,
while the $\times$ 3 at the bottom in blue stands for the number of used
multi-head attentions.

###  III-D Predictions

TABLE I: Ablation study on different settings. This evaluation is on EPIC-KITCHEN dataset. Temporal nodes means using the final output of X3D of size $T\times 2048$ to create nodes, while spatio-temporal nodes means using a mid layer of size $T\times 7\times 7\times 432$ with more spatial information. Finally ADJ-matrix stands for using the adjacency matrix for predicting the verbs instead of using only nodes for nouns and verbs. | verbs | nouns | actions
---|---|---|---
| top1 | top5 | top1 | top5 | top1 | top5
X3D | 46.5 | 79.8 | 34.3 | 65.3 | 21.0 | 38.7
THORN/temporal nodes | 55.8 | 82.86 | 39.9 | 66.37 | 26.8 | 44.0
THORN/temporal nodes + ADJ-matrix | 60.3 | 86.0 | 41.1 | 66.9 | 30.1 | 47.3
THORN/spatio-temporal nodes + ADJ-matrix | 61.0 | 85.9 | 42.9 | 67.9 | 30.5 | 47.5

Predictions are based on the learned nodes and adjacency matrix. However,
since in our case our actions are composed of verbs and nouns, we show that
using the adjacency matrix for predicting the verb and the object feature
representation for noun prediction is more effective. This makes sense since
the adjacency carries more information about how different objects interact
with each others, while the nodes carry a refined object representations,
after been processed through the different graph convolutions blocks. Our
final layers are two fully-connected layers one projecting $G_{out}$ from
$\mathbb{R}^{D_{2}\times C_{o}}$ to $\mathbb{R}^{1\times C_{o}}$, and the
other fully-connected layer projecting $A^{\prime}_{C_{o}}$ from
$\mathbb{R}^{C_{o}\times C_{o}}$ into $\mathbb{R}^{1\times C_{v}}$, where
$C_{o}$ and $C_{v}$ stand for the number of object classes and verb classes
respectively.
Since we have 3 outputs, our loss is a sum of three losses and can be
formulated as :

$\mathcal{L}=\mathcal{L}_{verbs}+\mathcal{L}_{nouns}+\mathcal{L}_{clip-
objects}$ (7)

Where $\mathcal{L}_{verbs}$ and $\mathcal{L}_{nouns}$ are the negative log-
likelihood losses (since each action is composed of one verb and one noun). As
described earlier, the $\mathcal{L}_{clip-objects}$ is the loss to ensure the
semantic of the object representation.

##  IV Experiments

Dataset. We have evaluated our model on two of the largest and challenging
datasets for first-view and human-object interaction action recognition. Epic-
Kitchen55 [30] contains 55 hours of recording of 32 different kitchens in 4
cities. This dataset has a total of 125 verbs and 352 nouns. EGTEA Gaze+ [31]
contains 28 hours of cooking activities from 86 unique sessions of 32
subjects, with over 10k video clips of 106 fine-grained egocentric activities.
In both datasets, each action is a combination of a verb and a noun. Actions
are relevant to different steps of preparing food (e.g. cleaning the kitchen,
cutting vegetables, preparing table).
Implementation. We implement our method using X3D as the visual encoder where
$D_{1}$ = 432, $H^{\prime}$= $W^{\prime}$= 7 and $D_{2}$ is 128. We input a
clip of 16 RGB frames for Epic-Kitchen and 25 frames for EGTEA Gaze+. We use a
dropout probability of 0.3. For the object relation reasoning module,
$N_{Block}$ is 5 blocks.

For the temporal convolution network, we run our model with different values
of the kernel size. As there was no impact on the results, we kept a kernel
size of 9. In training phase, we utilized Adam [32] to optimize the model with
an initial learning rate of 0.00005. We scaled the learning rate by a factor
of 0.1 with the patience of 5 epochs. The network was trained on a 4-GPU
machine for 30 epochs. We evaluated our model using top1 and top5 accuracy on
verbs and nouns for Epic-Kitchen, while for EGTEA Gaze+ we evaluated directly
on actions using top 1 accuracy.

###  IV-A Ablation Study

In this section, we validate our model design for the modules in the THORN.
The evaluation is conducted on the EPIC-Kitchen dataset. We propose different
settings, and see how each setting can improve the performance. In table I, we
can notice different results:

Firstly, we compare our baseline model X3D with THORN. Note that, in THORN,
the graph nodes can be constructed either using the output of the last layer
of X3D (temporal nodes) or using its intermediate layer (spatio-temporal
nodes). Here, we first compared X3D with THORN (temporal nodes), i.e., we
construct the nodes by the features in shape $T\times 2048$. In this setting,
nodes would serve to predict both verbs and nouns. In this scenario, we
improve nouns prediction by +5.6%, while, the verbs accuracy increased by
+9.3% . Proving the importance of the cross-object reasoning, compared to only
capturing visual information from 3D-CNNs.
Secondly, we study the importance of the adjacency matrix for predicting the
verbs. To do so, we use the adjacency matrix (ADJ-matrix) to predict verbs,
while keeping the nodes to predict the nouns. In this setting, the verb
prediction improves by +4.5% compared to the previous setting and by +13.8% to
the baseline X3D. This is because the adjacency matrix captures the object
interaction, hence, it is more suitable for verb prediction.
Thirdly, we study the effect of changing the temporal nodes with the spatio-
temporal nodes. Spatio-temporal nodes are the nodes constructed by the middle
layer of X3D which contains the spatial information ($T\times 7\times 7\times
432$). With spatio-temporal nodes, THORN improves +1.8% on nouns. This is
because, with spatial dimensions, the ORF can better capture the object
relative locations and the size of the object, then embed them in the node
representation. As a result, the noun accuracy improves. This setting also
brings +0.7% improvement on verbs.
Our overall architecture obtains +13.8% more accuracy on verbs and +8.6% on
nouns w.r.t. vanilla X3D. This reflects the importance of our proposed modules
in THORN and how an object-centric method can improve results on human-object
interaction actions.

TABLE II: Ablation study on fusing the scores of THORN with the scores from the object detector (Faster RCNN). This evaluation is on EPIC-KITCHEN dataset. Fusing both scores brings significant improvement on top-1 accuracy. For the object detector, we use an average pooling on all the video clip frames object detection scores and add a thresh-hold of 0.3  Faster-RCNN scores | THORN | Nouns
---|---|---
| $\times$ | 31.5
$\times$ |  | 32.8
|  | 42.9

We then study the components for predicting the nouns in our model. In table
II, we show that fusing scores of object detection and the scores obtained by
the THORN nodes representation works better than using only one of them. We
also find that predictions using only our model are better than the object
detector itself. This shows that our model can refine the objects represented
by the other objects (nodes) using our graph-based module.

###  IV-B Comparison with the State-of-the-Art

We then compare our proposed method with the state-of-the-art methods on EPIC-
Kitchen and EGTEA Gaze+ in table III and IV.

In Table III, we compare our results with the state-of-the-art methods. Among
these methods, Long Features Bank (LFB) [18] proposes to use global as well as
local features for action recognition. To do so, they extract features on both
clip and video levels, and combine them to have a better understanding of the
scene. Nevertheless, this method still lacks accuracy for the objects.
Moreover, LFB is a two step method which trains separately an object and verb
recognizer modules. For our THORN, we train a single model for predicting both
entities. As a result, we have a +8.5% improvement on top 1 nouns and a +4.9%
w.r.t. LFB on action recognition.

Our method achieves the overall best performance. We claim that AssembleNet++
utilizes additional modality such as optical flow in both training and
inference time. Even though, we still have the lead in top 1 accuracy for the
verbs, nouns and actions, which proves again that having an object-centric and
specific reasoning on object interactions is a key solution for having a
better action recognition on HOI datasets. Finally, our results prove that
using only RGB with an object-centric model achieves better or similar results
compared to methods relying on heavy multi-modality reasoning.

In table IV, we compare our method with the state-of-the-art on EGTEA Gaze+
dataset. We have the best accuracy w.r.t. the others methods, which shows the
generalization and robustness of our model on actions of HOI.

To sum up, compared to other methods, ours is lightly weighted as we use X3D,
while other methods rely on heavy 3D-CNNs such as I3D. THORN is trained
jointly on nouns and verbs as opposed to other methods such as LFB [18], and
we only need RGB frames and object classes per-frame.

TABLE III: Comparing THORN model with other state-of-the-art methods on the
validation set. Even though some of these comparisons are not fair since these
models are using multi-modalities, we still hold the best accuracy on actions
and nouns, which shows the strength of our model

Model | Obj | RGB | Flow | Audio | Verbs | Nouns | Actions
---|---|---|---|---|---|---|---
|  |  |  |  | top1 | top1 | top1
Baradel[33] | $\times$ |  | $\times$ |  | 40.9 | - | -
3D-CNN | $\times$ |  | $\times$ | $\times$ | 49.8 | 26.1 | 19.0
STO[18] |  |  | $\times$ | $\times$ | 51.0 | 26.6 | 19.5
LFB[18] |  |  | $\times$ | $\times$ | 52.6 | 31.5 | 22.8
AssembleNET++ ODF+SDF[20] |  |  |  | $\times$ | 60.0 | 37.1 | 25.2
THORN |  |  | $\times$ | $\times$ | 61.0 | 42.9 | 30.5

TABLE IV: Comparing THORN model with other state-of-the-art methods on EGTEA
Gaze+ split1. We hold the best accuracy on actions

| Two-stream | I3D [9] | TSN [34] | ego-rnn [35] | LSTA [36] | SAP [37] | THORN
---|---|---|---|---|---|---|---
ACC % | 43.8 | 54.2 | 58.0 | 62.1 | 62.0 | 64.1 | 67.5

###  IV-C Qualitative Study

In this section, we conduct a qualitative study of THORN.

Figure 5: Accuracy improvement on nouns (right) and verbs (left) w.r.t X3D.

In Fig. 5, we show the impact on some classes after adding our proposed module
w.r.t. vanilla X3D. In EPIC-Kitchen, we significantly improve accuracy on 28
verb classes. Only the accuracy of 3 out of 125 verbs decreases, while the
decrease is negligible. This improvement on verbs shows that understanding the
inter-relation of different objects is important for HOI.

For noun recognition, it is interesting to find that THORN can now predict
some classes such as water and wall. These classes are barely detected with
the object detector. This is a result of the reasoning process on cross object
classes, which refines the nodes and can finally predict overlooked object
classes.

###  IV-D Qualitative Study on Learned Adjacency Matrix

Figure 6: Example of the learned adjacency matrix of the action from Epic-
Kitchen55 dataset. We notice a strong correlation between the classes knife
and water for the action wash knife. Thus, we are able to collect high inter-
class relation to recognize the right verb and its relevant objects. Moreover,
the irrelevant classes such as fish are not activated, showing robustness of
the learned attention.

In this section, we provide more insight of our THORN model. We show the
strength of using the adjacency matrix and the attention mechanism.

In Fig. 6 we show an example of the learned adjacency matrix for the action
wash knife. In this figure, we find that there is a high correlation between
the classes knife and water in both directions. Whereas the classes tap, fish
and sponge are only correlated to themselves since they are not directly
relevant to the objective action class wash knife. This example shows the
effectiveness of THORN to capture the inter-object relations in the clipped
HOI videos.

###  IV-E Qualitative Study on the Object Representation Filter

The object representation filter is one of the main parts of our architecture
as it allows extraction of a good representation for different objects related
to the action. To make sure our filtering work, we extract the activation maps
for the different object and see what do they highlight in the scene.
Figures 7, 8, 9, represent different actions with their Class Activation Map
(CAM). The example in Fig. 7 represents the action wash leaf, when looking at
the output of the object representation filter the highest activation where on
the classes leaf and tap. As specified in the main paper, we want to learn
features specific to each class. The CAM of tap and leaf in this example
clearly shows that only the pixels relevant to the object were highlighted,
hence, the feature in the nodes are more representative of the objects of
interest.
Moreover, this result shows that our work does similar work to unsupervised
object segmentation. Hence, unlike other methods that rely on pre-trained
object detectors and tracking methods to extract object and then use ROI-Align
to extract objects features, our method is capable of yielding the same result
in a unsupervised manner and in a more simplified way. Besides that, our THORN
model learns to only focus on objects of interest.

Figure 7: Example of action washing leaf. the highest activated classes were
leaf and tap and when inferring the class activation map we can see that most
activated pixels are around the objects of interest. Hence, the features
extracted are more significant which makes it easier to predict the right
action. Figure 8: Example of action put leaf. In this example the most
activated object was leaf and its activation map shows that the focused-on
pixels actually belongs the leaf, proving the strength and robustness of our
approach.

Figure 9: The action in this figure is mix meat, and looking at the figure we
notice that the highlighted pixels are the ones corresponding to the spatula
and the meat. Therefore, it is easier to predict the right action.

##  V Conclusion

First-view action recognition relies on capturing the visual relationships
between different objects and the human. In this work, we propose an object-
centric model, which projects the standard CNN features into object class-
specific features. After that, we compute the inter-object relations in graph
reasoning, where each node corresponds to an object class and each edge
represents the relation between two different objects. We evaluate our model
on two large and challenging datasets. THORN achieves state-of-the-art
performance on both datasets, which shows the effectiveness and robustness of
our method. As our method relies on object detection precision, our future
work aims at developing an architecture that can combine object detection and
action recognition tasks. We also want to extend our model for first-view
action detection for untrimmed video.

## References

  * [1] X. Jiang, K. Xu, and T. Sun, “Action recognition scheme based on skeleton representation with ds-lstm network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 7, pp. 2129–2140, 2019.
  * [2] T. V. Nguyen and B. Mirza, “Dual-layer kernel extreme learning machine for action recognition,” _Neurocomputing_ , vol. 260, pp. 123–130, 2017.
  * [3] R. G. Barker and H. F. Wright, “One boy’s day; a specimen record of behavior.” 1951.
  * [4] ——, “Midwest and its children: The psychological ecology of an american town.” 1955.
  * [5] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” _arXiv preprint arXiv:1406.2199_ , 2014.
  * [6] C. Feichtenhofer, A. Pinz, and R. P. Wildes, “Spatiotemporal multiplier networks for video action recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4768–4777.
  * [7] C. Feichtenhofer, A. Pinz, and A. Zisserman, “Convolutional two-stream network fusion for video action recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1933–1941.
  * [8] S. Ji, W. Xu, M. Yang, and K. Yu, “3d convolutional neural networks for human action recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 35, no. 1, pp. 221–231, 2012.
  * [9] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6299–6308.
  * [10] X. Wang, R. Girshick, A. Gupta, and K. He, “Non-local neural networks in: Proceedings of the ieee conference on computer vision and pattern recognition,” 2018.
  * [11] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev _et al._ , “The kinetics human action video dataset,” _arXiv preprint arXiv:1705.06950_ , 2017.
  * [12] H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre, “Hmdb: a large video database for human motion recognition,” in _2011 International conference on computer vision_. IEEE, 2011, pp. 2556–2563.
  * [13] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1725–1732.
  * [14] K. Soomro, A. R. Zamir, and M. Shah, “Ucf101: A dataset of 101 human actions classes from videos in the wild,” _arXiv preprint arXiv:1212.0402_ , 2012\.
  * [15] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price _et al._ , “Scaling egocentric vision: The epic-kitchens dataset,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 720–736.
  * [16] K. Alahari, “Actor and observer: Joint modeling of first and third-person videos,” in _Proceedings of the 1st Workshop and Challenge on Comprehensive Video Understanding in the Wild_ , 2018, pp. 3–3.
  * [17] A. Fathi, Y. Li, and J. M. Rehg, “Learning to recognize daily actions using gaze,” in _European Conference on Computer Vision_. Springer, 2012, pp. 314–327.
  * [18] C.-Y. Wu, C. Feichtenhofer, H. Fan, K. He, P. Krahenbuhl, and R. Girshick, “Long-term feature banks for detailed video understanding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 284–293.
  * [19] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen, “Epic-fusion: Audio-visual temporal binding for egocentric action recognition,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 5492–5501.
  * [20] L. Wang and P. Koniusz, “Self-supervising action recognition by statistical moment and subspace descriptors,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 4324–4333.
  * [21] X. Wang and A. Gupta, “Videos as space-time region graphs,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 399–417.
  * [22] L. Shi, Y. Zhang, J. Cheng, and H. Lu, “Two-stream adaptive graph convolutional networks for skeleton-based action recognition,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 12 026–12 035.
  * [23] S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in _Thirty-second AAAI conference on artificial intelligence_ , 2018.
  * [24] R. Dai, S. Das, and F. Bremond, “CTRN: Class Temporal Relational Network For Action Detection,” in _The British Machine Vision Conference_ , Virtual, United Kingdom, Nov. 2021.
  * [25] P. Ghosh, Y. Yao, L. Davis, and A. Divakaran, “Stacked spatio-temporal graph convolutional networks for action segmentation,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2020, pp. 576–585.
  * [26] Y. Chen, W. Li, C. Sakaridis, D. Dai, and L. Van Gool, “Domain adaptive faster r-cnn for object detection in the wild,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3339–3348.
  * [27] C. Feichtenhofer, “X3d: Expanding architectures for efficient video recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 203–213.
  * [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [29] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [30] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price _et al._ , “Scaling egocentric vision: The epic-kitchens dataset,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 720–736.
  * [31] Y. Li, M. Liu, and J. M. Rehg, “In the eye of beholder: Joint learning of gaze and actions in first person video,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 619–635.
  * [32] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [33] F. Baradel, N. Neverova, C. Wolf, J. Mille, and G. Mori, “Object level visual reasoning in videos,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 105–121.
  * [34] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in _European conference on computer vision_. Springer, 2016, pp. 20–36.
  * [35] S. Sudhakaran and O. Lanz, “Attention is all we need: Nailing down object-centric attention for egocentric activity recognition,” _arXiv preprint arXiv:1807.11794_ , 2018.
  * [36] S. Sudhakaran, S. Escalera, and O. Lanz, “Lsta: Long short-term attention for egocentric action recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9954–9963.
  * [37] X. Wang, Y. Wu, L. Zhu, and Y. Yang, “Symbiotic attention with privileged information for egocentric action recognition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 12 249–12 256.