# Learning Affordance Grounding from Exocentric Images

Hongchen Luo1$\ddagger$ Wei Zhai1$\ddagger$ Jing Zhang3 Yang
Cao1,422footnotemark: 2 Dacheng Tao2,3
1 University of Science and Technology of China
3 JD Explore Academy 2 The University of Sydney
4 Institute of Artificial Intelligence, Hefei Comprehensive National Science
Center
{lhc12, wzhai056}<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>This work was done during an
internship at JD Explore Academy.Corresponding author. ‡ Equal contribution.

###### Abstract

Affordance grounding, a task to ground (i.e., localize) action possibility
region in objects, which faces the challenge of establishing an explicit link
with object parts due to the diversity of interactive affordance. Human has
the ability that transform the various exocentric interactions to invariant
egocentric affordance so as to counter the impact of interactive diversity. To
empower an agent with such ability, this paper proposes a task of affordance
grounding from exocentric view, i.e., given exocentric human-object
interaction and egocentric object images, learning the affordance knowledge of
the object and transferring it to the egocentric image using only the
affordance label as supervision. To this end, we devise a cross-view knowledge
transfer framework that extracts affordance-specific features from exocentric
interactions and enhances the perception of affordance regions by preserving
affordance correlation. Specifically, an Affordance Invariance Mining module
is devised to extract specific clues by minimizing the intra-class differences
originated from interaction habits in exocentric images. Besides, an
Affordance Co-relation Preserving strategy is presented to perceive and
localize affordance by aligning the co-relation matrix of predicted results
between the two views. Particularly, an affordance grounding dataset named
AGD20K is constructed by collecting and labeling over 20K images from 36
affordance categories. Experimental results demonstrate that our method
outperforms the representative models in terms of objective metrics and visual
quality. Code: github.com/lhc1224/Cross-View-AG.

##  1 Introduction

The goal of affordance grounding is to locate the region of “action
possibilities” of an object. For an intelligent agent, it is necessary to know
not only what the object is but also to understand how it can be used [11].
Perceiving and reasoning about possible interactions in local regions of
objects is the key to the shift from passive perception systems to embodied
intelligence systems that actively interact with and perceive their
environment [1, 33, 34, 38]. It has a wide range of applications for robot
grasping, scene understanding, action prediction [31, 49, 13, 12, 19, 28, 23,
47].

\begin{overpic}[width=398.9296pt]{figs/fig1.pdf} \par\end{overpic} Figure 1:
Observation. By observing the exocentric diverse interactions, the human
learns affordance knowledge determined by the object’s intrinsic properties
and transfer it to the egocentric view.
\begin{overpic}[width=381.5877pt]{figs/motivation.pdf} \put(21.6,-1.0){{(a)}}
\put(73.0,-1.0){{(b)}} \end{overpic} Figure 2: Motivation. (a) Exocentric
interactions can be decomposed into affordance-specific features $M$ and
differences in individual habits $E$. (b) There are co-relations between
affordances, e.g.“Cut with” inevitably accompanies “Hold” and is independent
of the object category (knife and scissors). Such co-relation is common
between objects. In this paper, we mainly consider extracting affordance-
specific cues $M$ from diverse interactions while preserving the affordance
co-relations to enhance the perceptual capability of the network.

As affordance is a dynamic property closely related to the interaction between
humans and environment [13], it is difficult to understand how to interact
with objects and establish an explicit link between the objects’ intrinsic
properties and affordances [29]. However, humans can easily perceive the
object’s affordance region by observing exocentric human-object interactions,
and give an egocentric definition. As shown in Fig. 1, although different
persons hold the racket in different positions due to their individual habits,
the human observer can perceive swingable regions determined by the intrinsic
properties (e.g., the long handle structure) of the racket from a group of
interacting images, despite the effect of individual differences, and transfer
the knowledge to the egocentric view, thereby constructing a bridge between
the object part and the affordance category.

To empower an agent with this ability to perceive the invariant egocentric
affordance from various exocentric interactions, this paper proposes a task of
affordance grounding from exocentric view, i.e., given exocentric human-object
interactions and egocentric object images, learning affordance knowledge and
transferring it to object images by only using affordance labels as
supervision. And in the testing stage, the output is the prediction of the
affordance region for a specific object with the input of an egocentric object
image and a particular affordance label.

To address this problem, we propose a cross-view knowledge transfer framework
to extract affordance-specific features from exocentric interactions and
transfer them to egocentric view. Specifically, we first devise an Affordance
Invariance Mining (AIM) module to decompose the exocentric human-object
interactions into the affordance representations determined by objects’
intrinsic properties and the differences originated from individual habits (as
shown in Fig. 2 (a)). We use low rank matrix decomposition [18, 22, 24, 10] to
minimize the intra-class differences caused by diverse interactions to obtain
affordance-specific cues. Furthermore, there is a correlation between the
object affordances (as shown in Fig. 2 (b)), which can be adopted to establish
the link between different affordances to reduce the uncertainty caused by
multiple affordances regions on the object. Therefore, we present a novel
Affordance Co-relation Preserving (ACP) strategy to perceive and localize the
affordance region by aligning the co-relation matrix of prediction results
from two views.

Despite the advances in affordance learning, the existing datasets [29, 43,
35, 32, 8] still bear limitations in terms of affordance/object category,
image quality, and scene complexity. To carry out a comprehensive study, this
paper proposes an affordance grounding dataset named AGD20K, consisting of
$20,061$ exocentric images and $3,755$ egocentric images from $36$ affordance
categories. The contrastive experiments against several representative methods
are performed on the AGD20K dataset. The results demonstrate the superiority
of our proposed method in capturing the intrinsic property of objects and
suppressing the interactive diversity of affordance.

Contributions: (1) We present a new affordance grounding from exocentric view
task and establish a large-scale AGD20K benchmark to facilitate the research
for empowering the agent to capture affordance knowledge from exocentric
human-object interactions. (2) We propose a novel cross-view knowledge
transfer framework for affordance grounding in which the affordance knowledge
is acquired from exocentric human-object interactions and transferred to
egocentric views while preserving the correlation between affordances, thereby
achieving better perception and localization of interactive affordance. (3)
Experiments on the AGD20K dataset demonstrate that our method outperforms
state-of-the-art methods and can serve as a strong baseline for future
research.

##  2 Related Works

###  2.1 Visual Affordance Grounding

The goal of affordance grounding is to locate the region of “action
possibilities” of an object. Numerous works [35, 7, 4, 8, 50, 20, 48, 27]
mainly build upon supervised approaches to establish mapping relations between
local regions of objects and affordance. Sawatzky et al. [43, 42] adopt an
Expectation-Maximization algorithm [6] to achieve weakly supervised affordance
detection using only a few key points. Nagarajan et al. [33] exploit only
affordance labels to ground the interactions from the videos. In contrast to
[33], our goal is to empower the agent to learn affordance knowledge from
exocentric human-object interactions. To this end, we propose an explicit
cross-view knowledge transfer framework that extracts affordance knowledge
determined by the intrinsic properties of objects from multiple exocentric
interactions and transfers it into egocentric images.

###  2.2 Visual Affordance Dataset

The emergence of the relevant datasets drives the development of affordance
grounding, as shown in Table 1. For example, Sawatzky et al. [43] select video
frames from CAD120 [19] to construct a weakly supervised affordance detection
dataset, using only cropped out object regions but in inferior image quality.
Other affordance-related datasets [32, 35, 4, 8, 41] face the problems of
small scale and low affordance/object category diversity and do not consider
human actions to reason about the affordance regions. PAD dataset [29]
considers the inference of human purpose from support images of human-object
interactions and transfers to a group of query images but does not provide
part-level affordance labels. In contrast to the above works, we explicitly
consider exocentric-to-egocentric viewpoint transformations and collect a much
larger scale of images, with richer affordance/object categories and part-
level annotations, which are more useful and applicable to real-world
application domains.

###  2.3 Learning View Transformations

The existing learning-view transformation works start from the theory of
mirror neurons [40], which adopts embedding learning to generate perspective
invariant representations from paired data, and leverage it for tasks such as
action recognition and video summarization under egocentric view [44, 45, 16,
39]. For example, Li et al. [25] extract key egocentric signals from the
exocentric view dataset during pre-training and distill them to the backbone
to guide feature learning in the egocentric video task. In contrast to the
above works, we aim to extract affordance knowledge from the diverse
exocentric human-object interactions and transfer it to the egocentric view,
which is challenging due to the uncertainty caused by various interactions and
the multiple affordance regions that objects contain.

##  3 Method

Our goal is to ground the object affordance regions in egocentric images.
During training, given a group of exocentric images
$\mathcal{I}_{exo}=\\{I_{1},...,I_{N}\\}$ ($N$ is the number of exocentric
images) and an egocentric object image $I_{ego}$, the network uses only
affordance labels as supervision, so as to learn affordance knowledge from
exocentric images and transfer it to egocentric images. During testing, only
given an egocentric image $I_{ego}$ and the affordance label $C_{a}$, the
network outputs the affordance region on the object.

Table 1: Statistics of related datasets and the proposed AGD20K dataset. Part: part-level annotation. HQ: high-quality annotation. $\sharp$Obj: number of object classes. $\sharp$Aff: number of affordance classes. $\sharp$Img: number of images. Dataset | Year | HQ | Part |  $\sharp$Obj. |  $\sharp$Aff. |  $\sharp$Img.
---|---|---|---|---|---|---
UMD [32] | 2015 | ✗ | ✓ | 17 | 7 | 30,000
[43] | 2017 | ✗ | ✓ | 17 | 7 | 3,090
IIT-AFF [35] | 2017 | ✗ | ✓ | 10 | 9 | 8,835
ADE-Aff [4] | 2018 | ✓ | ✓ | 150 | 7 | 10,000
PAD [29] | 2021 | ✓ | ✗ | 72 | 31 | 4,002
AGD20k (Ours) | 2021 | ✓ | ✓ | 50 | 36 | 23,816
|  |  |  |  |  |
\begin{overpic}[width=407.60385pt]{figs/method.pdf}
\put(26.0,58.1){\scriptsize{$\bm{1}$}} \put(25.9,42.6){\scriptsize{$\bm{2}$}}
\put(25.9,26.7){\scriptsize{$\bm{N}$}} \put(39.0,27.0){\scriptsize{$\bm{N}$}}
\put(49.2,27.0){\scriptsize{$\bm{N}$}}
\par\put(39.0,42.7){\scriptsize{$\bm{2}$}}
\put(49.3,42.7){\scriptsize{$\bm{2}$}}
\par\put(39.0,58.1){\scriptsize{$\bm{1}$}}
\put(49.3,58.1){\scriptsize{$\bm{1}$}} \par\put(32.45,63.7){\small{$\bm{W}$}}
\put(32.43,48.1){\small{$\bm{W}$}} \put(32.4,32.8){\small{$\bm{W}$}}
\par\put(38.8,32.7){\small{$\bm{H_{N}}$}}
\put(39.0,48.3){\small{$\bm{H_{2}}$}} \put(39.2,63.8){\small{$\bm{H_{1}}$}}
\par\put(26.6,68.1){\small{$\bm{M_{1}}$}}
\put(26.6,52.6){\small{$\bm{M_{2}}$}} \put(26.6,37.15){\small{$\bm{M_{N}}$}}
\par\put(30.0,70.4){\small{$\bm{Conv}$}} \put(30.0,54.9){\small{$\bm{Conv}$}}
\put(30.0,39.3){\small{$\bm{Conv}$}}
\par\put(39.0,5.8){\small{\cite[cite]{[\@@bibref{}{zhou2016learning}{}{}]}}}
\par\put(82.0,11.5){\small{\bm{$f_{ego}$}}}
\put(82.0,49.5){\small{\bm{$f_{exo}$}}} \put(19.25,64.0){\small{\bm{$Z_{1}$}}}
\put(19.25,48.2){\small{\bm{$Z_{2}$}}} \put(19.0,33.0){\small{\bm{$Z_{N}$}}}
\put(81.7,68.0){\scriptsize{\bm{$Q$}}} \put(87.7,54.9){\scriptsize{\bm{$P$}}}
\put(74.3,63.2){\small{\bm{$p$}}} \put(95.5,65.8){\small{\bm{$q$}}}
\put(82.0,64.5){\rotatebox{270.0}{{\scriptsize{(Eq. \ref{LACP})}}}}
\par\put(79.0,41.8){\rotatebox{270.0}{\small{\bm{$L_{KT}=||f_{exo}-f_{ego}||$}}}}
\put(89.5,38.0){\rotatebox{270.0}{{\small{(Eq. \ref{acp1}) $\sim$ (Eq.
\ref{LACP})}}}} \end{overpic} Figure 3: Overview of the proposed cross-view
knowledge transfer affordance grounding framework. It mainly consists of an
Affordance Invariance Mining (AIM) module and an Affordance Co-relation
Preservation (ACP) strategy. The AIM module (see in Sec. 3.1) aims to obtain
invariant affordance representations from diverse exocentric interactions. The
ACP strategy (see in Sec. 3.2) enhances the network’s affordance perception by
aligning the co-relation of the outputs of the two views.

Our proposed cross-view knowledge transfer framework for affordance grounding
is shown in Fig. 3. During training, we first use Resnet50 [14] to extract the
features of exocentric and egocentric images to obtain
$\mathcal{Z}_{exo}=\\{Z_{1},...,Z_{N}\\}$ and $Z_{ego}$, respectively. We then
present the Affordance Invariance Mining (AIM) module (see in Sec. 3.1) to
extract affordance-specific clues ($\mathcal{F}_{exo}$) from the exocentric
features. Meanwhile, we use two convolutional layers to map the egocentric
feature to the embedding space consistent with the exocentric view:
$F_{ego}=Conv(Z_{ego})$. Subsequently, the features of the two branches
($\mathcal{F}_{exo}$ and $F_{ego}$) are fed into the same convolution layer to
obtain features $\mathcal{D}_{exo}$ and $D_{ego}$ respectively. To ensure the
affordance knowledge can be transferred to the egocentric view, we average the
$\mathcal{D}_{exo}$ through the global average pooling (GAP) layer to obtain
the $f_{exo}$ and pass the $D_{ego}$ through the GAP layer to get the
$f_{ego}$, and align $f_{exo}$ and $f_{ego}$ using L2 loss $L_{KT}$. Then,
$f_{exo}$ and $f_{ego}$ are fed into the same fully connected layer to obtain
the affordance prediction. Finally, we propose an Affordance Co-relation
Preserving (ACP) strategy (see in Sec. 3.2) to enhance the network’s
perception of affordance by aligning the co-relation matrix of the outputs of
the two views. During testing, we feed the egocentric object images into the
network only through the egocentric branch, and then use the CAM [51]
technique to obtain the affordance regions of the object (see in Sec. 3.3).

\begin{overpic}[width=411.93767pt]{figs/dataset.pdf}
\put(0.0,72.0){\hbox{\pagecolor{black}{\color[rgb]{1,1,1} {(a)}}}}
\put(0.0,4.0){\hbox{\pagecolor{black}{\color[rgb]{1,1,1} {(b)}}}}
\put(57.0,52.5){\hbox{\pagecolor{black}{\color[rgb]{1,1,1} {(c)}}}}
\put(57.0,4.0){\hbox{\pagecolor{black}{\color[rgb]{1,1,1} {(d)}}}}
\par\put(0.0,64.5){\hbox{\pagecolor{white}{\color[rgb]{0,0,0}
\rotatebox{90.0}{\small{Train}}}}}
\put(0.0,56.5){\hbox{\pagecolor{white}{\color[rgb]{0,0,0}
\rotatebox{90.0}{\small{Test}}}}} \par\put(16.0,74.5){{\color[rgb]{0,0,0}
\small{Carry}}} \put(49.0,74.5){{\color[rgb]{0,0,0} \small{Sit on}}}
\put(79.5,74.5){{\color[rgb]{0,0,0} \small{Drink with}}}
\par\put(7.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Skis}}}
\put(15.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Snowboard}}}
\put(26.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Surfboard}}}
\put(39.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Bench}}}
\put(49.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Couch}}}
\put(58.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Skateboard}}}
\put(72.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Bottle}}}
\put(80.5,72.0){{\color[rgb]{0,0,0} \footnotesize{Wine glass}}}
\put(93.0,72.0){{\color[rgb]{0,0,0} \footnotesize{Cup}}} \par\end{overpic}
Figure 4: The properties of the AGD20K dataset. (a) Some examples from the
dataset. (b) The distribution of categories in AGD20K. (c) The word cloud
distribution of affordances in AGD20K. (d) Confusion matrix between the
affordance category and the object category in AGD20K, where the horizontal
axis denotes the object category and the vertical axis denotes the affordance
category.

###  3.1 Affordance Invariance Mining Module

As shown in Fig. 3, we decompose the interactions in exocentric images into
affordance-specific features $M$ and individual differences $E$. Inspired by
low-rank matrix decomposition [18, 22, 10], we represent the $M$ as the
multiplication of a dictionary matrix $W$ and a corresponding matrix $H$,
where the dictionary bases represent the sub-features of human-object
interaction, and minimize $E$ by iterative optimization to obtain a
reconstructed affordance representation $M$. Specifically, for the input
$Z_{i}$, we first reduce its dimensionality with a convolution layer and a
ReLU layer to ensure the non-negativity of the input, and then reshape them
into $X_{i}\in R^{c\times hw}$ ($c$, $h$ and $w$ are the channels, length, and
width of the feature maps respectively). We use non-negative matrix
factorization (NMF) [22] to update the dictionary and the coefficient
matrices. Consequently, $X_{i}$ is decomposed into two non-negative matrices
$W$ and $H_{i}$. Here $W\in R^{c\times r}$ is the dictionary matrix shared by
all exocentric features, while $H_{i}\in R^{r\times hw}$ is the coefficient
matrix of each exocentric feature, and $r$ is the rank of the low-rank matrix
$W$. To update $H_{i}$ and $W$ in parallel, we concatenate
$\mathcal{X}_{exo}=\\{X_{1},...,X_{N}\\}$ and
$\mathcal{H}=\\{H_{1},...,H_{N}\\}$ to obtain $X\in R^{c\times Nhw}$ and $H\in
R^{r\times Nhw}$. Mathematically, the optimization process can be formulated
as follows:

$\small\mathop{\min}\limits_{W,H}||X-WH||,\quad s.t.\ W_{ab}\geq 0,H_{bk}\geq
0.$ (1)

$W$ and $H$ are updated according to the following rules:

$\small H_{ab}\leftarrow
H_{ab}\frac{(W^{T}X)_{ab}}{(W^{T}WH)_{ab}},W_{ab}\leftarrow
W_{ab}\frac{(XH^{T})_{ab}}{(WHH^{T})_{ab}}.$ (2)

After several iterations, we get the output $M=WH$, and reshpae it to
$\mathcal{M}_{exo}=\\{M_{1},...,M_{N}\\},M_{i}\in R^{c\times h\times w}$.
Finally, we use a convolution layer to map it to the residual space and sum it
with the $\mathcal{Z}$ to get the final output $\mathcal{F}_{exo}$:

$\small F_{i}=Z_{i}+Conv(M_{i}),\quad i\in[1,N].$ (3)

In each batch of training, we update the initial dictionary matrix $W^{(0)}$
such that it can contain the statistical prior of the common subfeature of
human-object interaction, i.e.,

$\small W^{(0)}\leftarrow\alpha W^{(0)}+(1-\alpha)\bar{W},$ (4)

where $\bar{W}$ is the average over each mini-batch.

###  3.2 Affordance Co-relation Preserving Strategy

As shown in Fig. 3, we feed the feature representations of the two branches
($f_{exo}$ and $f_{ego}$) into the same fully connected layer respectively to
obtain the affordance category prediction scores $s$ and $g$:

$\small s=FC(f_{exo}),\quad g=FC(f_{ego}).$ (5)

Then, we align the affordance co-relation between the exocentric and
egocentric views by calculating the cross-entropy loss [15] $L_{ACP}$ of the
co-relation matrix of the prediction scores of the two branches:

$\small
p_{j}=\frac{exp(s_{j}/\bm{T})}{\sum_{k}^{N_{c}}exp(s_{k}/\bm{T})},\quad
q_{j}=\frac{exp(g_{j}/\bm{T})}{\sum_{k}^{N_{c}}exp(g_{k}/\bm{T})},$ (6)
$\small P=pp^{T},Q=qq^{T},$ (7) $\small
L_{ACP}=-\sum_{j}^{N_{c}}\sum_{k}^{N_{c}}P_{jk}log(Q_{jk}),$ (8)

where $\bm{T}$ is used to control the degree of attention paid to the
correlations between negative labels. $P_{jk}$ and $Q_{jk}$ denote the
correlation between classes $j$ and $k$ in the prediction results. Finally,
the total loss can be calculated as:

$\small L=\lambda_{1}L_{cls}+\lambda_{2}L_{ACP}+\lambda_{3}L_{KT},$ (9)

where $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$ are hyper-parameters to
balance the classification loss, ACP loss and $L_{KT}$ loss. $L_{cls}$ is the
sum of the cross-entropy losses of the classification results of the two
branches, and $L_{KT}$ is loss of cross-view affordance knowledge transfer:
$L_{KT}=||f_{exo}-f_{ego}||$.

Table 2: The results of different methods on AGD20k. The best results are in bold. “Seen” means that the training set and the test set contain the same object categories, while “Unseen” means that the object categories in the training set and the test set do not overlap. The ${\color[rgb]{0.91,0.33,0.5}\diamond}$ defines the relative improvement of our method over other methods. “Dark red”, “Orange” and “Purple” represent saliency detection, weakly supervised object localization and affordance grounding models, respectively. | Seen | Unseen
---|---|---
Methods | $\text{KLD}\downarrow$ | $\text{SIM}\uparrow$ | $\text{NSS}\uparrow$ | $\text{KLD}\downarrow$ | $\text{SIM}\uparrow$ | $\text{NSS}\uparrow$
Mlnet [5] | $5.197{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 70.4\%}$ | $0.280{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 19.3\%}$ | $0.596{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 55.5\%}$ | $5.012{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 64.3\%}$ | $0.263{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 8.4\%}$ | $0.595{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 39.3\%}$
DeepGazeII [21] | $1.858{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 17.2\%}$ | $0.280{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 19.3\%}$ | $0.623{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 48.8\%}$ | $1.990{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 10.2\%}$ | $0.256{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 11.3\%}$ | $0.597{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 38.9\%}$
EgoGaze [17] | $4.185{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 63.2\%}$ | $0.227{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 47.1\%}$ | $0.333{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 178.\%}$ | $4.285{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 58.3\%}$ | $0.211{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 35.1\%}$ | $0.350{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 137.\%}$
EIL [30] | $1.931{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 20.4\%}$ | $0.285{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 17.2\%}$ | $0.522{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 77.6\%}$ | $2.167{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 17.5\%}$ | $0.227{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 25.6\%}$ | $0.330{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 151.\%}$
SPA [36] | $5.528{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 72.2\%}$ | $0.221{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 51.1\%}$ | $0.357{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 160.\%}$ | $7.425{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 75.9\%}$ | $0.169{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 68.6\%}$ | $0.262{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 216.\%}$
TS-CAM [9] | $1.842{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 16.5\%}$ | $0.260{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 28.5\%}$ | $0.336{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 176.\%}$ | $2.104{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 15.1\%}$ | $0.201{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 41.8\%}$ | $0.151{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 449.\%}$
Hotspots [33] | $1.773{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 13.3\%}$ | $0.278{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 20.1\%}$ | $0.615{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 50.7\%}$ | $1.994{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 10.4\%}$ | $0.237{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 20.3\%}$ | $0.577{\color[rgb]{0.91,0.33,0.5}\scriptstyle~{}\diamond 43.7\%}$
Ours | $\bm{1.538}_{\pm 0.017}$ | $\bm{0.334}_{\pm 0.001}$ | $\bm{0.927}_{\pm 0.007}$ | $\bm{1.787}_{\pm 0.017}$ | $\bm{0.285}_{\pm 0.002}$ | $\bm{0.829}_{\pm 0.014}$
|  |  |  |  |  |

###  3.3 Inference

Our test procedure only requires an egocentric object image and an affordance
label as input to predict the affordance region. We utilize the class
activation mapping [51] by computing a weighted sum of the feature maps
$D^{i}$ of the last convolutional layer to obtain the affordance region
heatmap: $Y^{C_{a}}=\sum_{i}w^{C_{a}}_{i}D^{i}$, where $C_{a}$ is the
affordance class, $D^{i}$ is the $i$-th layer feature map, and $w^{C_{a}}_{i}$
is the weight corresponding to the $i$-th neuron under the $C_{a}$ category.

\begin{overpic}[width=403.26341pt]{figs/main_result.pdf}
\put(-2.0,41.7){\rotatebox{90.0}{\small{GT}}}
\put(-2.0,32.0){\rotatebox{90.0}{\small{Ours}}}
\put(-2.1,23.0){\rotatebox{90.0}{\small{\cite[cite]{[\@@bibref{}{nagarajan2019grounded}{}{}]}}}}
\put(-2.1,13.8){\rotatebox{90.0}{\small{\cite[cite]{[\@@bibref{}{mai2020erasing}{}{}]}}}}
\put(-2.1,4.4){\rotatebox{90.0}{\small{\cite[cite]{[\@@bibref{}{kummerer2016deepgaze}{}{}]}}}}
\put(21.3,-0.15){\hbox{\pagecolor{white}{\color[rgb]{0,0,0} {Seen}}}}
\put(71.3,-0.15){\hbox{\pagecolor{white}{\color[rgb]{0,0,0} {Unseen}}}}
\end{overpic} Figure 5: Visual affordance heatmaps on the AGD20K dataset. We
select the prediction results of representative methods of affordance
grounding (Hotspots [33]), weakly supervised object localization (EIL [30]),
and saliency detection (DeepGazeII [21]) for presentation.

##  4 Dataset

Dataset Collection.  The exocentric images are mainly obtained from HICO [3]
and COCO [26]. We select images from the HICO dataset according to the verb
category and the COCO dataset according to the object category. Then, we
manually remove images with ambiguous interactions. To enrich the diversity of
the dataset, we further collect $2,112$ exocentric images from free-license
websites. Meanwhile, We collect $3,755$ egocentric images from the Internet
with free use license according to object categories. Some examples are shown
in Fig. 4 (a).

\begin{overpic}[width=429.28616pt]{figs/T_seen.pdf} \end{overpic}

\begin{overpic}[width=429.28616pt]{figs/B_seen.pdf} \end{overpic}

\begin{overpic}[width=429.28616pt]{figs/K_seen.pdf} \end{overpic}

\begin{overpic}[width=429.28616pt]{figs/T_unseen.pdf}
\put(52.7,-4.8){\small$\bm{T}$} \end{overpic}

\begin{overpic}[width=429.28616pt]{figs/B_unseen.pdf}
\put(52.5,-4.8){\small{$\bm{r}$}} \end{overpic}

\begin{overpic}[width=429.28616pt]{figs/K_unseen.pdf}
\put(51.0,-4.8){\small$\bm{N}$} \end{overpic}

Figure 6: Hyper-parameter study. We investigate the influence of $\bm{T}$ in the ACP, the rank $r$ of the $W$ in the AIM, and the number of exocentrice images $N$, respectively. The top and bottom columns represent the “Seen” and ”Unseen” experimental settings, respectively. Table 3: Ablation study. We investigate the influence of the AIM module, ACP strategy and $L_{KT}$ on model performance. | AIM | ACP | $L_{KT}$ | $\text{KLD}\downarrow$ | $\text{SIM}\uparrow$ | $\text{NSS}\uparrow$
---|---|---|---|---|---|---
Seen |  |  |  | $1.985$ | $0.238$ | $0.302$
$\checkmark$ |  |  | $1.750$ | $0.280$ | $0.674$
| $\checkmark$ |  | $1.810$ | $0.257$ | $0.687$
|  | $\checkmark$ | $1.933$ | $0.241$ | $0.344$
$\checkmark$ | $\checkmark$ |  | $1.749$ | $0.286$ | $0.735$
$\checkmark$ |  | $\checkmark$ | $1.664$ | $0.309$ | $0.818$
| $\checkmark$ | $\checkmark$ | $1.741$ | $0.299$ | $0.679$
$\checkmark$ | $\checkmark$ | $\checkmark$ | $1.538$ | $0.334$ | $0.927$
Unseen |  |  |  | $2.059$ | $0.228$ | $0.445$
$\checkmark$ |  |  | $1.933$ | $0.261$ | $0.682$
| $\checkmark$ |  | $1.920$ | $0.250$ | $0.666$
|  | $\checkmark$ | $1.967$ | $0.265$ | $0.622$
$\checkmark$ | $\checkmark$ |  | $1.926$ | $0.269$ | $0.696$
$\checkmark$ |  | $\checkmark$ | $1.916$ | $0.272$ | $0.679$
| $\checkmark$ | $\checkmark$ | $1.922$ | $0.267$ | $0.640$
$\checkmark$ | $\checkmark$ | $\checkmark$ | $1.787$ | $0.285$ | $0.829$
|  |  |  |  |  |

Dataset Annotation.  We select $36$ affordance classes commonly used in real-
world application scenarios and assign labels to each image based on the
interaction between human and object in each exocentric image. Given the
object class contained in each affordance class, we assign affordance labels
based on the object class in the egocentric images. The testing process
requires pixel-level labels to calculate objective metrics. As the annotation
approach in [8], we take the form of points for regions of interaction, in
which the dense points are for regions of frequent interaction and vice versa.
Then, heatmaps of affordance regions can be obtained from the points as [8].
Some annotation examples are shown in Fig. 4 (a).

Statistic Analysis.  To obtain deeper insights into our AGD20K dataset, we
show its important features from the following aspects. The distribution of
categories in the dataset is shown in Fig. 4 (b), which shows that the dataset
contains a wide range of affordance/object categories in diverse scenarios.
The affordance word cloud is shown in Fig. 4 (c). The confusion matrix of
affordance and object categories is shown in Fig. 4 (d). It shows a multi-to-
multi relationship between affordance and object categories, posing a
significant challenge for the affordance grounding task. See supplementary
materials for more details.

##  5 Experiments

###  5.1 Benchmark Setting

To provide a comprehensive evaluation, we choose three commonly used metrics
Kullback-Leibler Divergence (KLD) [2], SIMilarity (SIM) [46] and Normalized
Scanpath Saliency (NSS) [37], see supplementary material for details of each
metric. Our model is implemented in PyTorch and trained with the SGD
optimizer. The input images are randomly clipped from $256\times 256$ to
$224\times 224$ with random horizontal flipping. We train the model for $35$
epochs on a single NVIDIA $1080$ti GPU with an initial learning rate of
$1e$-$3$. The hyper-parameters $\lambda_{1}$, $\lambda_{2}$ and $\lambda_{3}$
are set to $1$, $0.5$ and $0.5$ respectively. The hyper-parameter $\bm{T}$ in
the ACP is set to $1$. The rank $r$ of the dictionary matrix $W$ and the
number of iterations in the AIM are set to $64$ and $6$ respectively. The
number of exocentric images $N$ is set to $3$. Besides, three saliency
detection models (Mlnet [5], DeepGazeII [21], EgoGaze [17]), three weakly
supervised object localization models (EIL [30], SPA [36], TS-CAM [9]) and one
affordance grounding model (Hotspots [33]) are chosen for comparison. We
design two different settings: 1) “Seen”, i.e., the training set and the test
set contain the same object categories, and 2) “Unseen”, i.e., the object
categories in the training set and the test set do not overlap.

###  5.2 Quantitative and Qualitative Comparisons

The experimental results are shown in Table 2. Our method achieves the best
results in both “Seen” and “Unseen” settings. Taking KLD as the metric, our
method improves 17.2% compared to the best saliency model, 16.5% over the best
weakly supervised object localization (WSOL) model, and 13.3% over the
affordance grounding model in the “Seen” setting. Our method with the “Unseen”
setting improves 10.2% compared to the best saliency model, surpasses the best
WSOL model by 15.1%, and exceeds the affordance grounding model by 10.4%. It
indicates that our method can effectively transfer the affordance knowledge
from the exocentric view to the object in the egocentric view and has a good
generalization ability for unseen objects.

In addition, we visualize the affordance maps in “Seen” and “Unseen” settings,
as shown in Fig. 5. It shows that our method can obtain more accurate
prediction results for affordance grounding. While “Sit on” contains objects
with different appearances (“bed” and “chair”), our method can capture the
common features of the affordance region and obtain better prediction results.
Since “bottle” has two different affordances, “Open” and “Hold”, the network
predicts different affordance regions. In the “Unseen” setting, the “knife”
has two different affordances, “Hold” and “Cut with”. Our method can locate
different affordance regions based on the learned affordance knowledge,
demonstrating its superior generalization capability.

Table 4: Different classes. The KLD results of different methods on some representative affordance categories. Classes | Hold | Swing | Drink with | Lie on | Brush with
---|---|---|---|---|---
Mlnet [5] | $6.762$ | $9.248$ | $4.497$ | $4.767$ | $6.215$
DeepGazeII [21] | $2.071$ | $2.478$ | $2.067$ | $1.602$ | $2.385$
EgoGaze [17] | $4.671$ | $6.723$ | $4.268$ | $2.921$ | $5.135$
EIL [30] | $2.008$ | $2.486$ | $2.254$ | $1.377$ | $3.003$
SPA [36] | $3.006$ | $6.720$ | $7.683$ | $4.006$ | $8.043$
TS-CAM [9] | $1.628$ | $2.420$ | $2.300$ | $1.370$ | $2.642$
Hotspots [33] | $1.770$ | $2.178$ | $1.942$ | $1.566$ | $2.154$
Ours | $1.594$ | $2.161$ | $1.748$ | $1.039$ | $2.040$
|  |  |  |  |

\begin{overpic}[width=429.28616pt]{figs/scale_seen.pdf}
\put(46.0,-6.9){\small{Seen}} \end{overpic}

\begin{overpic}[width=429.28616pt]{figs/scale_unseen.pdf}
\put(29.0,-5.1){\small{Unseen}} \end{overpic}

Figure 7: Different scales. We split the test set into three subsets of “Big”,
“Middle” and “Small” according to the ratio of mask to the whole image, and
show the results of the NSS metrics.

###  5.3 Ablation Study

The ablation study results are shown in Table 3. It confirms the ability of
the AIM module to learn affordance-specific features from diverse exocentric
interactions, which play a significant role in improving network performance.
The ACP strategy improves more obviously than $L_{KT}$, indicating that the
preservation of affordance co-relation can more effectively improve the
network’s ability to perceive and locate affordance regions. In addition, we
investigate the influence of different hyper-parameter settings of $\bm{T}$ in
the ACP strategy (shown in Fig. 6 (left)), the rank $r$ of the dictionary
matrix $W$ in the AIM module (shown in Fig. 6 (middle)), and different
exocentric images $N$(see Fig. 6 (right)). It can be seen that the performance
of the model is more influenced by $\bm{T}$, while the rank $r$ does not have
a significant impact on the results. The number of exocentric images taken
from $2$ to $3$ has a larger impact on the model. For $N=1$, our model still
outperforms most contenders.

Table 5: Different sources. “Exo” means that training only uses exocentric images, “Exo&Ego” means that training uses both exocentric and egocentric images. | Method | Source | $\text{KLD}\downarrow$ | $\text{SIM}\uparrow$ | $\text{NSS}\uparrow$
---|---|---|---|---|---
Seen | EIL [30] | Exo | $1.931$ | $0.285$ | $0.522$
Exo&Ego | $2.156$ | $0.321$ | $0.747$
SPA [36] | Exo | $5.528$ | $0.221$ | $0.357$
Exo&Ego | $4.312$ | $0.252$ | $0.494$
TS-CAM [9] | Exo | $1.842$ | $0.260$ | $0.336$
Exo&Ego | $1.707$ | $0.290$ | $0.622$
Ours | Exo&Ego | $1.538$ | $0.334$ | $0.927$
Unseen | EIL [30] | Exo | $2.167$ | $0.227$ | $0.330$
Exo&Ego | $2.029$ | $0.256$ | $0.529$
SPA [36] | Exo | $7.425$ | $0.169$ | $0.262$
Exo&Ego | $6.174$ | $0.209$ | $0.433$
TS-CAM [9] | Exo | $2.104$ | $0.201$ | $0.151$
Exo&Ego | $2.002$ | $0.228$ | $0.305$
Ours | Exo&Ego | $\bm{1.787}$ | $\bm{0.285}$ | $\bm{0.829}$
|  |  |  |  |

###  5.4 Performance Analysis

Different Classes.  The KLD metrics on some representative categories are
shown in Table 4. “Hold” and “Swing” both contain diverse object categories
with different appearances. “Drink with” and “Hold” contain overlapped object
categories but have completely different affordance regions. Objects of “Lie
on” are generally labeled with a larger region, while those of “Brush with”
are generally smaller. Our model exceeds others regarding different aspects of
the challenge, which confirms its robustness. See supplementary material for
the KLD metrics for each category.

Different Scales.  We divide the test set into “big”, “middle” and “small”
splits according to the proportion of mask to the whole image (see
supplementary material for details). The test results are shown in Fig. 7. Our
model outperforms all other methods in all splits on both settings, showing
its ability to capture the intrinsic affordance properties of objects, even in
more challenging cases. The performance of the experimental results on all
metrics are shown in the supplementary material.

Different Sources.  The results for different sources are shown in Table 5. It
shows that using both exocentric and egocentric images improves most methods,
but the improvement is limited. Our method still surpasses all models, showing
that the knowledge transfer from explicitly cross-views is effective in
learning from exocentric diverse interactions to egocentric invariant
affordance representation. The performance of the experimental results on all
metrics are shown in the supplementary material.

Limitations.  Our method still has limitations, e.g., the predicted affordance
maps may contain intermediate background regions when multiple objects appear
and irrelevant background regions may be activated for slender object. In the
future, we will refer to [36] to refine the generated results to obtain more
accurate results.

##  6 Conclusion

In this paper, we make an attempt to address a new challenging task named
affordance grounding from exocentric view. Specifically, we propose a novel
cross-view knowledge transfer framework that can extract invariant affordance
from diverse exocentric interactions and transfer it to egocentric view. We
establish a large affordance grounding dataset named AGD20K, which contains
20K well-annotated images, serving as a pioneer testbed for the task. Our
model outperforms representative models from related areas and can serve as a
strong baseline for future research.

Broader Impacts.  The research on affordance grounding from exocentric view
will advance the realization of embodied intelligence. However, harmful human
demonstrations (risky behaviors) may lead to negative guidance for the agent,
which should be prohibited by strict legislation.

Acknowledgments.This work was supported by National Key R&D Program of China
under Grant 2020AAA0105701, National Natural Science Foundation of China
(NSFC) under Grants 61872327 and ARC FL-170100117.

## References

  * [1] Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme.  Interactive perception: Leveraging action in perception and perception in action.  IEEE Transactions on Robotics, 33(6):1273–1291, 2017.
  * [2] Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand.  What do different evaluation metrics tell us about saliency models?  IEEE transactions on pattern analysis and machine intelligence, 41(3):740–757, 2018.
  * [3] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng.  Learning to detect human-object interactions.  In 2018 IEEE winter conference on applications of computer vision (WACV), pages 381–389. IEEE, 2018.
  * [4] Ching-Yao Chuang, Jiaman Li, Antonio Torralba, and Sanja Fidler.  Learning to act properly: Predicting and explaining affordances from images.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 975–983, 2018.
  * [5] Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara.  A deep multi-level network for saliency prediction.  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 3488–3493. IEEE, 2016.
  * [6] Arthur P Dempster, Nan M Laird, and Donald B Rubin.  Maximum likelihood from incomplete data via the em algorithm.  Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22, 1977.
  * [7] Thanh-Toan Do, Anh Nguyen, and Ian Reid.  Affordancenet: An end-to-end deep learning approach for object affordance detection.  In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 5882–5889. IEEE, 2018.
  * [8] Kuan Fang, Te-Lin Wu, Daniel Yang, Silvio Savarese, and Lim J. Joseph.  Demo2vec: Reasoning object affordances from online videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [9] Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, and Qixiang Ye.  Ts-cam: Token semantic coupled attention map for weakly supervised object localization.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2886–2895, October 2021.
  * [10] Zhengyang Geng, Meng-Hao Guo, Hongxu Chen, Xia Li, Ke Wei, and Zhouchen Lin.  Is attention better than matrix decomposition?  arXiv preprint arXiv:2109.04553, 2021.
  * [11] James J Gibson.  The theory of affordances.  Hilldale, 1977.
  * [12] Helmut Grabner, Juergen Gall, and Luc Van Gool.  What makes a chair a chair?  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1529–1536. IEEE, 2011.
  * [13] Mohammed Hassanin, Salman Khan, and Murat Tahtali.  Visual affordance and function understanding: A survey.  ACM Computing Surveys (CSUR), 54(3):1–35, 2021.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [16] Hsuan-I Ho, Wei-Chen Chiu, and Yu-Chiang Frank Wang.  Summarizing first-person videos from third persons’ points of view.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 70–85, 2018.
  * [17] Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato.  Predicting gaze in egocentric video by learning task-dependent attention transition.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 754–769, 2018.
  * [18] Tamara G Kolda and Brett W Bader.  Tensor decompositions and applications.  SIAM review, 51(3):455–500, 2009.
  * [19] Hema Swetha Koppula, Rudhir Gupta, and Ashutosh Saxena.  Learning human activities and object affordances from rgb-d videos.  The International Journal of Robotics Research, 32(8):951–970, 2013\.
  * [20] Hema S Koppula and Ashutosh Saxena.  Physically grounded spatio-temporal object affordances.  In European Conference on Computer Vision (ECCV), pages 831–847. Springer, 2014.
  * [21] Matthias Kümmerer, Thomas SA Wallis, and Matthias Bethge.  Deepgaze ii: Reading fixations from deep features trained on object recognition.  arXiv preprint arXiv:1610.01563, 2016.
  * [22] Daniel D. Lee and H. Sebastian Seung.  Algorithms for non-negative matrix factorization.  In NIPS, 2000.
  * [23] Xingming Li, Wei Zhai, and Yang Cao.  A tri-attention enhanced graph convolutional network for skeleton-based action recognition.  IET Computer Vision, 15(2):110–121, 2021.
  * [24] Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu.  Expectation-maximization attention networks for semantic segmentation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 9167–9176, 2019.
  * [25] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman.  Ego-exo: Transferring visual representations from third-person to first-person videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6943–6953, 2021.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 740–755. Springer, 2014.
  * [27] Liangsheng Lu, Wei Zhai, Hongchen Luo, Yu Kang, and Yang Cao.  Phrase-based affordance detection via cyclic bilateral interaction.  arXiv preprint arXiv:2202.12076, 2022.
  * [28] Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao.  Learning visual affordance grounding from demonstration videos.  arXiv preprint arXiv:2108.05675, 2021.
  * [29] Hongchen Luo, Wei Zhai, Jing Zhang, Yang Cao, and Dacheng Tao.  One-shot object affordance detection.  arXiv preprint arXiv:2108.03658, 2021.
  * [30] Jinjie Mai, Meng Yang, and Wenfeng Luo.  Erasing integrated learning: A simple yet effective approach for weakly supervised object localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8766–8775, 2020.
  * [31] Priyanka Mandikal and Kristen Grauman.  Learning dexterous grasping with object-centric visual affordances.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6169–6176. IEEE, 2021.
  * [32] Austin Myers, Ching L Teo, Cornelia Fermüller, and Yiannis Aloimonos.  Affordance detection of tool parts from geometric features.  In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 1374–1381. IEEE, 2015.
  * [33] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman.  Grounded human-object interaction hotspots from video.  In Proceedings of the IEEE International Conference on Computer Vision (CVPR), pages 8688–8697, 2019.
  * [34] Tushar Nagarajan and Kristen Grauman.  Learning affordance landscapes for interaction exploration in 3d environments.  arXiv preprint arXiv:2008.09241, 2020.
  * [35] Anh Nguyen, Dimitrios Kanoulas, Darwin G Caldwell, and Nikos G Tsagarakis.  Object-based affordances detection with convolutional neural networks and dense conditional random fields.  In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5908–5915. IEEE, 2017.
  * [36] Xingjia Pan, Yingguo Gao, Zhiwen Lin, Fan Tang, Weiming Dong, Haolei Yuan, Feiyue Huang, and Changsheng Xu.  Unveiling the potential of structure preserving for weakly supervised object localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11642–11651, 2021.
  * [37] Robert J Peters, Asha Iyer, Laurent Itti, and Christof Koch.  Components of bottom-up gaze allocation in natural images.  Vision research, 45(18):2397–2416, 2005.
  * [38] Santhosh K Ramakrishnan, Dinesh Jayaraman, and Kristen Grauman.  An exploration of embodied visual exploration.  International Journal of Computer Vision, 129(5):1616–1649, 2021\.
  * [39] Krishna Regmi and Mubarak Shah.  Bridging the domain gap for ground-to-aerial image matching.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 470–479, 2019.
  * [40] Giacomo Rizzolatti and Laila Craighero.  The mirror-neuron system.  Annu. Rev. Neurosci., 27:169–192, 2004.
  * [41] Anirban Roy and Sinisa Todorovic.  A multi-scale cnn for affordance segmentation in rgb images.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 186–201. Springer, 2016.
  * [42] Johann Sawatzky and Jurgen Gall.  Adaptive binarization for weakly supervised affordance segmentation.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 1383–1391, 2017.
  * [43] Johann Sawatzky, Abhilash Srikantha, and Juergen Gall.  Weakly supervised affordance detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [44] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari.  Actor and observer: Joint modeling of first and third-person videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7396–7404, 2018.
  * [45] Bilge Soran, Ali Farhadi, and Linda Shapiro.  Action recognition in the presence of one egocentric and multiple static cameras.  In Asian Conference on Computer Vision, pages 178–193. Springer, 2014.
  * [46] Michael J Swain and Dana H Ballard.  Color indexing.  International Journal of Computer Vision (IJCV), 7(1):11–32, 1991\.
  * [47] Yuxiang Yang, Zhihao Ni, Mingyu Gao, Jing Zhang, and Dacheng Tao.  Collaborative pushing and grasping of tightly stacked objects via deep reinforcement learning.  IEEE/CAA Journal of Automatica Sinica, 9(1):135–145, 2021.
  * [48] Wei Zhai, Hongchen Luo, Jing Zhang, Yang Cao, and Dacheng Tao.  One-shot object affordance detection in the wild.  arXiv preprint arXiv:2108.03658, 2021.
  * [49] Jing Zhang and Dacheng Tao.  Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things.  IEEE Internet of Things Journal, 8(10):7789–7817, 2020.
  * [50] Xue Zhao, Yang Cao, and Yu Kang.  Object affordance detection with relationship-aware network.  Neural Computing and Applications, 32(18):14321–14333, 2020.
  * [51] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2921–2929, 2016.


#  MentalHealthAI: Utilizing Personal Health Device Data to Optimize
Psychiatry Treatment

Manan Shukla and Oshani Seneviratne  PhD

## Abstract

Mental health disorders remain a significant challenge in modern healthcare,
with diagnosis and treatment often relying on subjective patient descriptions
and past medical history. To address this issue, we propose a personalized
mental health tracking and mood prediction system that utilizes patient
physiological data collected through personal health devices. Our system
leverages a decentralized learning mechanism that combines transfer and
federated machine learning concepts using smart contracts, allowing data to
remain on users’ devices and enabling effective tracking of mental health
conditions for psychiatric treatment and management in a privacy-aware and
accountable manner. We evaluate our model using a popular mental health
dataset that demonstrates promising results. By utilizing connected health
systems and machine learning models, our approach offers a novel solution to
the challenge of providing psychiatrists with further insight into their
patients’ mental health outside of traditional office visits.

## Introduction

Mental health conditions such as depression and anxiety are some of the most
challenging medical problems to diagnose and treat. Current treatment
guidelines for these disorders primarily utilize subjective assessments,
relying on patient self-report or clinician evaluation to inform clinical
decisions. As such, the lack of objective markers for clinical outcomes
presents a significant bottleneck in psychiatry. Furthermore, a patient’s mood
or emotions may change over time, but clinicians only have access to a
patient’s data at the time of the visit, leading to a potentially biased
sampling of the patient’s mental state. To address this, collecting data from
the patient over a long period would be ideal for effective diagnosis and
treatment. However, collecting such data is also challenging due to privacy
concerns. Connected health applications enable data to be generated and stored
in a decentralized manner, where the data may reside cross-device. A common
challenge in health informatics in federated and decentralized settings is
that training and test data are not independently and identically distributed
(non-IID), which is especially true in scenarios that apply to predict the
mental health of individuals using a combination of medical and environmental
signals. Because health data is typically not identically distributed, the
generalization performance tends to be worse, and lower accuracy can result
from overlooking the distribution shift in the training and testing data21.
More importantly, since non-IID data in healthcare applications comes from
different clients, protecting data privacy is crucial in decentralized
learning settings16.

Furthermore, applying connected health technologies in a mental health
population poses multiple problems9. First is the concern about data security
and privacy. Studies have shown that mental health populations typically
consider their data sensitive and vary in sharing this information due to
perceived mental health stigmas. Surveys have shown that 65% of patients with
mental health disorders are unlikely to share patient data with their
psychiatrists8. If the psychiatrists aim to rely on patients’ history,
studies17 have shown patient histories only to be 62% accurate, leading to
psychiatric misdiagnoses as high as 65.9% for major depressive disorders and
85.8% for panic disorders. Therefore, a technological solution is necessary to
provide psychiatrists with health insights without collecting raw data from
the patient’s smart health devices. Second, current models do not account for
the granularity of mental health disorders. As explained in the American
Psychiatric Association’s Clinical Practice Guidelines2, patient emotions are
subject to rapid changes within the span of a day or a week, and elements such
as sleep or diet can lead to quick changes in mood. While many have utilized
information from Electronic Health Records (EHR) to predict mental health
crises7, these models overlook granular patient changes. Therefore, they
cannot generate a patient baseline (in fact, getting data through facial
expressions or EHR systems can lead to biased results). Understanding the
immediate effects of medication, such as antidepressants, is crucial for
psychiatrists and requires granular patient data that cannot be retrieved
otherwise. Currently, the most feasible way to collect this granular patient
data is through a smartphone and a patient’s health devices. This method,
however, has the issue of unequal data streams. Different patients have
different personal health devices. For example, while one patient may have
five devices, another may only have one. While training a model on the patient
with five devices may lead to better results, the input from a patient
population to this model will decrease (as not many patients have so many
personal health devices). Therefore, there is a need to obtain insights even
with the feature types being unequal from patient to patient. We present a
decentralized federated learning algorithm called _MentalHealthAI_ to
alleviate these challenges. First, _MentalHealthAI_ uses on-device machine
learning to prevent data from leaving the patient’s smartphone. However, smart
contracts are utilized in this framework to elect an aggregator, thereby
creating a decentralized aggregator instead of a traditional centralized
server. A self-executing piece of code, called a smart contract, can encode
rules that will be executed in a decentralized manner on a blockchain22. The
data remains on the patient’s device in each epoch, and the model parameters
are transferred from that device to the aggregator. Second, as each smartphone
may collect a different set of patient features, _MentalHealthAI_ utilizes a
decision tree based methodology to derive model insights even when features
and labels are not necessarily uniform.

## System Design and Implementation

Figure 1: System Architecture

At its core, the current system is a decentralized-learning infrastructure
that utilizes physiological data to predict patient moods and therefore
provides mental insights to a patient’s psychiatrist without the requirements
of a uniform set of features (as is necessary for typical machine learning
algorithms). The overall architecture can be found in Figure 1, and its
specific features are described below. Clients X, Y, and Z each represent
different patients. Each patient owns several IoT devices, indicated by the
different data streams A, B, and C. Each data stream has the same dimensions
but different content (A may be heart rate data, B may be blood pressure, and
C may be skin temperature). The final model is created by adding the union of
models trained from different combinations of data streams to the random
forest decision tree classification system. During the evaluation, we use the
features in the POPANE dataset3 as individual data streams to simulate this
concept.

Personal health data do not contain uniform features from patient to patient.
The common problem is that the devices used by different individuals are
different. Traditional machine learning is limited in cases where one patient
has data collected from many disparate data streams, such as heart rate, blood
pressure, electroencephalogram (EEG), and electrocardiogram (ECG), while
another patient has only one (for example, just the heart rate). This
limitation exists because only the intersection of feature types is considered
rather than every feature type present. For each patient, we assume a
smartphone acts as the gateway between the patient’s IoT devices, as depicted
in Algorithm 1.

Algorithm 1 Smartphone (Client) Federation of IoT Devices

1:$DeviceAddresses=[DevAddr1,DevAddr2...]$ $\triangleright$ IoT device
addresses

2:$MentalHealthAIdf=dataframe()$

3:for $addr\in\\{1,\dots,DeviceAddresses\\}$ do $\triangleright$ Collect data
from each IoT device within private mobile environment

4: $data\leftarrow GetData(addr)$

5: $MentalHealthAIdf.AddColumn(data)$

6:end for

We utilized the POPANE dataset3 for the simulation, where we divided the
patient population based on the type and number of data streams each patient
has. For example, we place patients with six of the same data streams (set
$A$) in a different cohort than patients with only three data streams (set
$B$). Here, _data streams_ can refer to heart rate and blood pressure data.
However, if set B is a subset of set A, $A\cap B$ can be added to cohort B’s
training set (as the data streams used in B and $A\cap B$ are the same). Now,
consider a patient population where the number of data streams a patient has
varies from 1 to 6. For any patient with >1 data streams, a power set
excluding the empty set is generated as shown in Algorithm 2. For example,
given a patient with data streams d = {A, B, C}, P(d) = [{}, {A}, {B}, {A, B},
{C}, {A, C}, {B, C}, {A, B, C}], excluding {}, we divert each element in this
power set (representing a set of data streams) into separate cohorts. This
process maximizes the utility of patients with multiple data streams, as it
maximizes the amount of data found in each cohort (in comparison to simply
dividing the patient population based on the number of data streams present).
We then train multiple machine learning models on these cohorts, where one
model is trained from data from one cohort. Note that the labels are
unaltered, regardless of the feature subset. When combined with
_MentalHealthAI_ ’s decentralized AI architecture, it is also important to
note that multiple smartphones will be selected as aggregators but will be
training different models with different training subsets.

Algorithm 2 Smartphone (Client) Generation of Data Stream Combinations

1:$n=MentalHealthAIdf.headers.length()$

2:$subsets=[]$

3:$n=0$

4:while $n\neq 1$ do $\triangleright$ Generate all subsets of MentalHealthAIdf
features with sizes $\\{n-1,\dots,1\\}$

5: $SubsetList=GenerateSubsets(MentalHealthAIdf,SubsetSize=n)$

6: $subsets.append(SubsetList)$

7: $subsets.flatten()$

8: $n=n-1$ $\triangleright$ "subsets” is now a one-dimensional list with every
possible subset from size $\\{1,\dots,n\\}$

9:end while

Based on the patient population and available data streams at a given time,
certain models will be more accurate than others, and this relationship can
change frequently. Furthermore, every patient’s mood with different baseline
emotion levels may differ. While models can predict a large portion of the
population successfully, they may not be accurate enough for a specific
patient. Algorithm 3 shows the model generation process for each client based
on the available datasets.

Algorithm 3 Smartphone (Client) Model Generation

1:$models=[]$ $\triangleright$ For models trained on each subset in subsets

2:for $subset\in\\{subsets\\}$ do

3: $model=GenerateModel(subset)$ $\triangleright$ Generate a model with the
subset as the features and the original labels as the label. See Figure 5.
Note: Model Generation/Training can happen in parallel.

4: $models.append(model)$

5:end for

Using a smart contract deployed on a blockchain as the “secure model
aggregator,” the client interacts with it by emitting events to indicate
learning has finished. The corresponding smart contract code is depicted in
Algorithm 4. The smart contract employs a voting process to elect the next
“leader” to perform model aggregation. As each model has been trained on
different feature subsets, decentralized aggregation occurs independently for
each model. For example, if we have three models trained on features [A],
[A,B], and [A,B,C], each of these models will be aggregated with other models
trained on the same set of features from a different patient on a different
client. If the patient’s smartphone is not elected as an aggregator, the smart
contract will send the model parameters from the patient’s smartphone to the
smartphone elected as the model aggregator.

Algorithm 4 Smart Contract Leader/Aggregator Election

1:$AggregatorStorage=\\{\\}$

2:$AggreagatorStorage=InitializeAggregatorStorage(models,DataStreams)$

3:for $(model,DataStream)\in\\{AggregatorStorage\\}$ do

4: $VotingResultAddress=SelectAggregator(DataStream)$

5: $EmitEvent(VotingResultAddress,DataStream)$

6:end for

7:$TransactData(AggregatorStorage,SmartphoneAddress)$

The clients will interact with the smart contract as shown in Algorithm 5.
Once the clients have finished training, they will notify the smart contract
and be considered for the next “leader” election. The client will also monitor
events emitted from the smart contract to see if it is elected as an
aggregator. If it is, then it will receive models from other smartphones.

Algorithm 5 Smartphone (Client) Interaction with the Smart Contract for Model
Aggregation

1:$Transact(SetLearningFinished(True),SmartContractAddress)$ $\triangleright$
Indicate learning finished.

2:$Transact(GetModelParams(models,subsets),SmartContractAddress)$
$\triangleright$ Transmit Model Parameters.

3:$AssignedDataStreams=[]$

4:if $MonitorAggregatorEvent(SmartContractAddress,SmartphoneAddress)$ then

5: $AssignedDataStreams=Transact(GetAssignedDatastream,SmartContractAddress)$

6: $models=CollectNodeModels()$

7:end if

Once the model parameters have been received, utilizing a decision tree, the
“leader” client select the best prediction model for the patient as shown in
Algorithm 6. It collects mapping from the smart contract with data stream as
_key_ and aggregator smart contract address as _value_. For example, assume a
patient has three devices/data streams. This patient’s models include every
model trained on the following data stream combinations: [{A}, {B}, {A, B},
{C}, {A, C}, {B, C}, {A, B, C}]. Then, a calibration period is set for a
certain period to collect new patient emotional features/labels (in Algorithm
6, it is set to 7 days). Each set of features in our simulation contributes to
the models generated daily. The random forest decision tree (such as the one
shown in Figure 2) will then use these models to predict the patient’s
emotional labels. The decision tree is run on the patient’s smartphone after
data stream based models have been generated and distributed back to the
individual nodes from the aggregator.

Algorithm 6 Smartphone (Client) Local Model Aggregation Using Decision Tree

1:$ModelStorage=\\{\\}$

2:$AggregatorStorage=GetAggregatorStorage(SmartContractAddress)$

3:for $subset\in\\{subsets\\}$ do

4: $model=GetModel(subset,AggregatorStorage[subset])$

5: $ModelStorage[subset]=model$

6:end for

7:$CurrentData=dataframe()$

8:for $7DaysTime...$ do:

9: $CurrentData.AddRow(PhysiologicalData,EmotionalState)$

10:end for

11:$DecisionTreeData=dataframe()$

12:for $Feature\in\\{PhysiologicalData\\}$ do

13: $DecisionTreeData.AddCol(features)$

14:end for

15:$DecisionTreeData.AddCol("EmotionalLabel")$

16:for $model\in\\{ModelStorage\\}$ do

17: $DecisionTreeData.AddCol(model.ModelName)$

18: for $PhysiologicalData\in\\{CurrentData\\}$ do

19:
$DecisionTreeData.AddRow(PhysiologicalData,EmotionalLabel,ModelPrediction)$

20: end for

21:end for

22:$DecisionTreeModel=TrainDecisionTree(DecisionTreeData)$

Figure 2: An Example Decision Tree. Each branch title is in the following
format: data streams utilized in curly braces, the local model accuracy, and
the criteria to split the branch. Below, the multi-colored branches represent
the label distribution in the model, and the number in parentheses represents
the total feature/labels utilized. Please see Figure 5 to cross-reference the
acronyms used in this diagram.

## Evaluation and Results

Figure 3: Learning Results After Leader Election and Model Aggregation.
_Nodes_ refer to the other smartphones contributing to the combined model.

We evaluated our system using a mental health dataset named POPANE3. The
POPANE dataset contains a set of 142 patients whose ECG, Electrodermal
Activity (EDA), Skin Temperature (ST), Respiration (Resp), Systolic Blood
Pressure (SBP), and Diastolic Blood Pressure (DBP) have been measured and
labeled with positive and negative affect, which is rated from a scale of
0-10, with 0 indicating negative affect, and 10 indicating positive affect. We
chose this dataset primarily because it closely matches our use case. A
personal health device can measure each of the physiological parameters given
above, and training on such a dataset can provide insight into the utility of
such a system in a much larger population. Secondly, the data provided is
collected on a second-to-second basis, similar to the collection rates found
in many current IoT devices, such as smartwatches that measure heart rate or
ECGs on the patient’s skin. Finally, a major advantage of utilizing the POPANE
dataset is its non-IID distributed data, as seen in Figure 4. The figure
clearly shows that the affect is not equally distributed throughout the
dataset (and is not likely to represent the standard population), which is
more akin to what may be present in real-world situations, where random
samples proportionate to the overall population are unlikely. Thus, through
this dataset, we aim to investigate the resilience of _MentalHealthAI_ in non-
IID settings.

Figure 4: Frequency of Various Affects in the POPANE Dataset3

First, we assessed the training results from a model run on a centralized
server. The model was a simple Artificial Neural Network (ANN) with three
dense layers with softmax activation, as shown in Figure 5. We decided upon
the activation function based on favorable learning results. As mentioned, we
used six physiological features to assess the patient’s affect, ranked from
0-10 to serve as the output. Each physiological feature is considered a
separate data stream for this evaluation, containing data from different IoT
devices. We set the data into a train-test split of 70-30% and used a sparse
categorical cross-entropy loss function due to the nature of the output
categorical labels. Multiple checkpoints were implemented, such as early
stopping (which will stop training if accuracy does not improve after multiple
epochs in a row) and learning adjustment (which will lower the learning rate
by a factor of ten if the accuracy does not improve). We ran the model for 107
epochs and stopped the learning process because there was no change in
training accuracy. After multiple trials, this epoch value led to the best
learning result in our model. The overall accuracy was approximately 86%.

Figure 5: A Simplified View of the Neural Network Model Architecture

Based on these results, we can conclude that there is a link between
physiological parameters and a patient’s emotional state. We chose accuracy as
our primary evaluation metric to ensure the model is clinically viable. We
then evaluated the decentralized learning aspects of this system. Since we
could not acquire physical devices to test the model’s performance in the real
world, we evaluated the decentralized learning components through simulation.
In this simulation, we assume a consortium of 142 patients modeled using the
POPANE dataset, each with data collected through IoT devices. A global model
updates itself based on data from each patient to form the final trained
model. We trained the models in the same fashion as the ANN discussed above.
Figure 3 shows the test-set accuracy after training on each node. As shown in
Figure 3, successful learning can happen in a discontinuous situation. While
the nodes had an initial training accuracy of 51%, this increased immediately
to 86% after training from two additional nodes, confirming that
_MentalHealthAI_ can obtain high accuracy even in distributed settings.
However, note that such results may not be obtainable in real-world
conditions. Primarily, data collected in the POPANE dataset has been obtained
in a controlled environment rather than during regular day-to-day activities.
Therefore, if truly deployed in a community, there is a greater chance of
false positives, false negatives, and inaccurate readings from IoT devices.
However, given accurate input data, we assert that _MentalHealthAI_ can be
deployed in such a community setting. We then evaluated the decision tree
aspect of the system to determine the model accuracy in non-ideal settings. We
divided the 142 patients in the POPANE dataset into four patient cohorts. Each
cohort represented patients with a certain number of IoT devices (1 device, 2
devices, 3 devices, or 4 devices). Similar to what we explained in the methods
section, we extracted data streams from each cohort based on the power set of
their features. For example, in the cohort with 2 data streams (A and B),
three sets of data were created: A, B, (A, B), each with the same label. We
repeated this process for each patient cohort. Models were trained on the
following data streams: ST, ECG, ST, ECG, EDA, ST, EDA, ECG, EDA, ST, ECG,
EDA, Resp, ST, Resp, ECG, Resp, ST, ECG, Resp, EDA, Resp, ST, EDA, Resp, ECG,
EDA, Resp, ST, ECG, EDA, Resp. Next, we simulated the “calibration” period,
where each model generated emotion predictions based on new data to which the
models were not exposed. This data then served as the input to the random
forest model, which then provided the emotional predictions based on the
predictions of the previously trained models. Figure 2 depicts the decision
tree for a single client (i.e., a smartphone belonging to a patient).

We simulated a standard baseline solution, _MentalHealthAI-Baseline_ , to the
above problem as a means of comparison. As a typical machine learning model
cannot utilize different feature sets, the most optimized results will likely
only come from the cohort with 4 data streams (35 patients). We trained
standard ANN with the same hyperparameters as above on this data set, with an
overall accuracy of 86%. In comparison, _MentalHealthAI-Fed_ had an overall
accuracy of 80%, a substantial improvement. By utilizing unequal feature sets
through multiple model combinations and a random forest model, one can improve
learning results compared to a model that requires uniform features. While
this accuracy level is lower than the original baseline model, it is important
to acknowledge the differences in data. The baseline ANN model simulated an
ideal world where 142 willing patients with access to 6 separate IoT devices.
However, finding 142 patients with more than three personal health devices in
the real world is intuitively infeasible for many reasons, such as cost and
access. However, through this unique _MentalHealthAI_ framework, we
demonstrate that high accuracy is achievable even in less-than-realistic
settings. We believe that this occurs due to multiple reasons. First,
_MentalHealthAI_ utilizes models without noise and irrelevant features, making
them less susceptible to their effects. Second, models trained on less number
of features can succeed by having access to a greater number of patients.
Third, a random forest model can select the best model for the patient, a
choice that can change over time. Finally, _MentalHealthAI_ was compared to
current state-of-the-art emotion prediction systems and machine learning
methods in adjacent domains. As shown in Table 1, it is clear that compared to
other past AI models, _MentalHealthAI_ can produce greater accuracy with both
the baseline ANN model and the decentralized decision tree architecture. Note
that due to the novelty of the POPANE dataset at the time we developed our
model, we were unable to compare our results to similar models that may have
been trained on the same data.

Table 1: Comparison of Mental Health Prediction Systems
Legend:  Diastolic Blood Pressure (DBP), Electrocardiogram (ECG), Electrodermal Activity (EDA) Electroencephalogram (EEG), Environmental Temperature (ET), Fingertip Blood Oxygen Saturation (OXY), Galvanic Skin Response (GSR), Heart Rate (HR), Heart Rate Variability (HRV), Respirations (Resp), Skin Conductivity (SC), Skin Temperature (ST), Systolic Blood Pressure (SBP),  Algorithm/Model | Data Used | Accuracy
---|---|---
Lan Z. et al11 | EEG | 49.63%
Cheng et. al4 |  | ECG, HRV
---
79.51%
Peter et. al14 |  | ST, SC, ET, HR
---
75%
Wen et. al19 |  | GSR, OXY, HR
---
74%
MentalHealthAI-Baseline |  | ECG, EDA, ST, Resp, SBP, DBP
---
86%
MentalHealthAI-Fed |  | ECG, EDA, ST, Resp, SBP, DBP
---
80%

## Related Work

Federated learning, introduced by McMahan et al.13, enables learning from
decentralized data sources, where clients volunteer to participate in
federated learning, i.e., they can join or leave the systems whenever they
want. Simply put, federated learning enables learning from decentralized data
sources10. A variant of federated learning in blockchain settings is swarm
learning18, where a smart contract would elect a node to perform model updates
at each epoch instead of a central aggregator. This selected node aggregates
and broadcasts the model parameters to all other nodes. We drew inspiration
from this methodology in the work presented in this paper. However, swarm
learning nodes are essentially large and powerful hospital servers utilized in
applications such as leukemia and tuberculosis prediction18. Our work involves
learning in a much more decentralized setting that leverages IoT devices and
smartphones with much smaller memory and performance. At the same time, input
features are all uniform in the original swarm learning implementation18,
which we believe is an assumption that may not hold in other decentralized
settings. We have embraced the non-IID assumption in our implementation.

Pfitzner et al.15 conducted a systematic literature review on the concept of
and research into federated learning and its applicability for confidential
healthcare datasets. In particular, Lee and Shin12 conducted an experiment
using the Modified National Institute of Standards and Technology (MNIST),
Medical Information Mart for Intensive Care-III (MIMIC-III), and ECG datasets
to evaluate the performance of a federated learning system compared to the
state-of-the-art method for in-hospital mortality using imbalanced data.
Additionally, a small but growing number of works have focused on the
application of federated learning in mental health applications. FedMood20
uses mobile phone and IoT data in a “multi-view” federated learning setting to
detect the emotions of individuals. However, a central aggregator is still
necessary for federated learning to be possible, which is risky, especially
for patients in vulnerable populations. Chhikara et al.5 describe a federated
learning framework that uses images to detect human emotion. While they
achieved successful learning results, such a model is infeasible for granular
changes in emotion/mood. While physiological data can provide hour-by-hour
changes in a patient’s emotion (such as changes after taking an
antidepressant), obtaining patient pictures every hour is infeasible. Garriga
et al.7’s work is similar to the previously described work but utilizes EHR to
predict mental health crises. While this is a valuable model for predicting
specific mental health events, the model cannot assess granular emotional
changes. Instead, crises are only extrapolated based on patient visits to the
clinic or the hospital (which often occurs when the patient is sick). Through
our work, we aim to see both granular and longitudinal changes in emotions,
i.e., how mood changes throughout the day (especially with medication
interventions) and how moods change over a week. A novel variant of federated
learning is personalized federated learning6. The generated model is adapted
to better fit a local dataset (for example, data belonging to a single
patient). Such a model adaptation can lead to a more personalized model for
the specific patient. In other words, each client’s model does not need to be
the same. While this strategy can prove useful for this application, we have
not yet employed it due to problems with categorical overfitting (where the
model chooses one category with any given input). Such overfitting will likely
occur in this setting as emotion/mood may remain the same for many hours.
Subjecting this to a model may lead the model to assume that the patient’s
emotions always remain the same. A more generalized model can expose a greater
variation of training data. Architecturally, _MentalHealthAI_ provides many
learning advantages that other AI strategies in this domain do not provide.
Compared to traditional machine learning, _MentalHealthAI_ introduces privacy-
sensitive strategies to address a previously stigmatized population. We
specifically allow learning on decentralized edge nodes, i.e., smartphones and
only require transferring model parameters to make _MentalHealthAI_ less
susceptible to noise and irrelevant features. However, _MentalHealthAI_ takes
this further by introducing decentralized aggregators, preventing attacks on a
centralized aggregator. The unique contribution of _MentalHealthAI_ lies in
its ability to utilize the available data, regardless of variations in the
feature set.

## Conclusion and Future Work

Advances in AI techniques and IoT devices have transformed how chronic
illnesses are treated today, such as asthma, hypertension, and diabetes.
However, an area of medicine where connected health has remained relatively
untapped has been mental health. In most situations, patient history, which is
inaccurate and imperfect, has predominantly been used to treat this disorder.
At the same time, psychiatry poses many unique challenges to connected health
adoption. First, a sensitive patient population may not support releasing
personal data, i.e., information potentially harmful if leaked. Secondly,
significant variations exist in the number of data streams a patient has, thus
potentially limiting learning. Finally, qualitative elements such as mood or
emotions can change rapidly throughout the day and the week. For example,
simple changes in diet, medications, or even sleep can lead to different
emotions. Therefore, monitoring granular emotional changes is key to
successfully monitoring mental health. While many have focused on using facial
expressions or EHR records to predict crises, these models overlook the small
changes that can lead to mental health issues. Therefore, to solve these
problems, we utilized a unique combination of various AI and blockchain
techniques to enhance data privacy and ownership in a system called
_MentalHealthAI_. It is an innovative combination of smart contracts and
decentralized learning to create models useful for psychiatrists but in a way
that protects the patient’s privacy. IoT devices provide a second-by-second
change in the patient’s outward physiological signs, allowing for a granular
understanding of the patient’s health. It allows for successful learning even
when data is stored in a patient’s smartphone. As part of the evaluation, we
used a novel mental health dataset and divided the patient population into
cohorts based on the data streams available for each patient. We demonstrated
that we could predict emotions/moods from physiological data in a
decentralized and privacy-preserving manner.

Our methodology for predicting mental health disorders has several benefits.
First, it increases accessibility. For example, if 20% of the patient
population has only one IoT device, a traditional machine learning algorithm
would be trained on only this limited population. In comparison,
_MentalHealthAI_ can utilize the entire patient population for model training
in a decentralized and privacy-preserving manner, which can provide greater
model utility for patients who do not have access to physiological data
generators (i.e., IoT Devices). Secondly, it can increase model accuracy,
especially in fields that have yet to be studied extensively due to non (or
limited) data availability. Therefore, certain data streams may contribute to
the model’s accuracy, and different combinations of data stream features
enable the better establishment of links between features and labels. Finally,
this method can better adapt to non-IID settings. Intuitively, patient
populations are unique, as most patients are more likely to have between 1-3
IoT devices. Therefore, a model trained on more patients but with fewer
features can have greater accuracy than one trained on more features but with
a smaller patient cohort. This relationship can change from community to
community and region to region. We address this issue by having different
population cohorts to provide accurate results while being resilient to
changes in the patient population composition.

There are various limitations to this work. We are yet to evaluate our work
with real smartphones in a decentralized setting in real life. Therefore, an
initial user study is necessary to determine the effectiveness and impact of
the prediction accuracy. Secondly, understanding physiological changes during
emotions such as surprise, fear, or agitation would be valuable in addition to
detecting moods and emotions in patients as a baseline. Therefore, the current
model may need to be retrained on a separate dataset, and hyperparameters
tuned appropriately to recognize these emotions. Another important challenges
are model approximation and optimization, i.e., is there a model that performs
well on all clients? And how to find such a model? By continuing to work on
these limitations, we can deploy such infrastructure in the mental health
patient population and provide utility to psychiatrists needing an objective
metric to assess their patients.

## References

  * 1
  * Association 2023 American Psychiatric Association. 2023\.  _Clinical Practice Guidelines of American Psychiatric Association_.  psychiatry.org.  https://www.psychiatry.org/psychiatrists/practice/clinical-practice-guidelines
  * Behnke et al. 2022 Maciej Behnke, Mikołaj Buchwald, Adam Bykowski, Szymon Kupiński, and Lukasz D Kaczmarek. 2022\.  Psychophysiology of positive and negative emotions, dataset of 1157 cases and 8 biosignals.  _Scientific Data_ 9, 1 (2022), 1–15.
  * Cheng et al. 2017 Zi Cheng, Lin Shu, Jinyan Xie, and C. L. Philip Chen. 2017\.  A novel ECG-based real-time detection method of negative emotions in wearable applications. In _2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC)_. IEEE, 296–301.  https://doi.org/10.1109/SPAC.2017.8304293
  * Chhikara et al. 2020 Prateek Chhikara, Prabhjot Singh, Rajkumar Tekchandani, Neeraj Kumar, and Mohsen Guizani. 2020\.  Federated learning meets human emotions: A decentralized framework for human–computer interaction for iot applications.  _IEEE Internet of Things Journal_ 8, 8 (2020), 6949–6962.
  * Fallah et al. 2020 Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. 2020\.  Personalized federated learning: A meta-learning approach.  _arXiv preprint arXiv:2002.07948_ (2020).
  * Garriga et al. 2022 Roger Garriga, Javier Mas, Semhar Abraha, Jon Nolan, Oliver Harrison, George Tadros, and Aleksandar Matic. 2022.  Machine learning model to predict mental health crises from electronic health records.  _Nature medicine_ (2022), 1–9.
  * Grando et al. 2020 Adela Grando, Julia Ivanova, Megan Hiestand, Hiral Soni, Anita Murcko, Michael Saks, David Kaufman, Mary Jo Whitfield, Christy Dye, Darwyn Chern, et al. 2020\.  Mental health professional perspectives on health data sharing: Mixed methods study.  _Health informatics journal_ 26, 3 (2020), 2067–2082.
  * Ivanova et al. 2020 Julia Ivanova, Adela Grando, Anita Murcko, Michael Saks, Mary Jo Whitfield, Christy Dye, and Darwyn Chern. 2020.  Mental health professionals’ perceptions on patients control of data sharing.  _Health informatics journal_ 26, 3 (2020), 2011–2029.
  * Kairouz et al. 2021 Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. 2021\.  Advances and open problems in federated learning.  _Foundations and Trends® in Machine Learning_ 14, 1–2 (2021), 1–210.
  * Lan et al. 2016 Zirui Lan, Olga Sourina, Lipo Wang, and Yisi Liu. 2016\.  Real-time EEG-based emotion monitoring using stable features.  _The Visual Computer_ 32 (2016), 347–358.
  * Lee and Shin 2020 Geun Hyeong Lee and Soo-Yong Shin. 2020.  Federated learning on clinical benchmark data: performance assessment.  _Journal of medical Internet research_ 22, 10 (2020), e20891.
  * McMahan et al. 2017 Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017\.  Communication-efficient learning of deep networks from decentralized data. In _Artificial intelligence and statistics_. PMLR, 1273–1282.
  * Peter et al. 2005 Christian Peter, Eric Ebert, and Helmut Beikirch. 2005\.  A wearable multi-sensor system for mobile acquisition of emotion-related physiological data. In _Affective Computing and Intelligent Interaction: First International Conference, ACII 2005, Beijing, China, October 22-24, 2005\. Proceedings 1_. Springer, 691–698.
  * Pfitzner et al. 2021 Bjarne Pfitzner, Nico Steckhan, and Bert Arnrich. 2021\.  Federated learning in a medical context: a systematic literature review.  _ACM Transactions on Internet Technology (TOIT)_ 21, 2 (2021), 1–31.
  * Rieke et al. 2020 Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al. 2020\.  The future of digital health with federated learning.  _NPJ digital medicine_ 3, 1 (2020), 1–7.
  * Vermani et al. 2011 Monica Vermani, Madalyn Marcus, and Martin A Katzman. 2011\.  Rates of detection of mood and anxiety disorders in primary care: a descriptive, cross-sectional study.  _The primary care companion for CNS disorders_ 13, 2 (2011), 27211\.
  * Warnat-Herresthal et al. 2021 Stefanie Warnat-Herresthal, Hartmut Schultze, Krishnaprasad Lingadahalli Shastry, Sathyanarayanan Manamohan, Saikat Mukherjee, Vishesh Garg, Ravi Sarveswara, Kristian Händler, Peter Pickkers, N Ahmad Aziz, et al. 2021\.  Swarm learning for decentralized and confidential clinical machine learning.  _Nature_ 594, 7862 (2021), 265–270.
  * Wen et al. 2014 Wanhui Wen, Guangyuan Liu, Nanpu Cheng, Jie Wei, Pengchao Shangguan, and Wenjin Huang. 2014\.  Emotion Recognition Based on Multi-Variant Correlation of Physiological Signals.  _IEEE Transactions on Affective Computing_ 5 (2014), 126–140.
  * Xu et al. 2021 Xiaohang Xu, Hao Peng, Lichao Sun, Md Zakirul Alam Bhuiyan, Lianzhong Liu, and Lifang He. 2021\.  Fedmood: Federated learning on mobile health data for mood detection.  _arXiv preprint arXiv:2102.09342_ (2021).
  * Zhao et al. 2018 Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. 2018\.  Federated learning with non-iid data.  _arXiv preprint arXiv:1806.00582_ (2018).
  * Zheng et al. 2020 Zibin Zheng, Shaoan Xie, Hong-Ning Dai, Weili Chen, Xiangping Chen, Jian Weng, and Muhammad Imran. 2020.  An overview on smart contracts: Challenges, advances and platforms.  _Future Generation Computer Systems_ 105 (2020), 475–491.