# ST-HOI: A Spatial-Temporal Baseline for
Human-Object Interaction Detection in Videos

Meng-Jiun Chiou  National University of SingaporeSingaporeSingapore
<EMAIL_ADDRESS>,  Chun-Yu Liao  ASUS Intelligent Cloud
ServicesTaipeiTaiwan<EMAIL_ADDRESS>,  Li-Wei Wang  ASUS Intelligent Cloud
ServicesTaipeiTaiwan<EMAIL_ADDRESS>,  Roger Zimmermann  National
University of SingaporeSingaporeSingapore<EMAIL_ADDRESS>and  Jiashi
Feng  National University of SingaporeSingaporeSingapore<EMAIL_ADDRESS>
(2021)

###### Abstract.

Detecting human-object interactions (HOI) is an important step toward a
comprehensive visual understanding of machines. While detecting non-temporal
HOIs (_e.g._ , sitting on a chair) from static images is feasible, it is
unlikely even for humans to guess temporal-related HOIs (_e.g._ ,
opening/closing a door) from a single video frame, where the neighboring
frames play an essential role. However, conventional HOI methods operating on
only static images have been used to predict temporal-related interactions,
which is essentially guessing without temporal contexts and may lead to sub-
optimal performance. In this paper, we bridge this gap by detecting video-
based HOIs with explicit temporal information. We first show that a naive
temporal-aware variant of a common action detection baseline does not work on
video-based HOIs due to a feature-inconsistency issue. We then propose a
simple yet effective architecture named Spatial-Temporal HOI Detection (ST-
HOI) utilizing temporal information such as human and object trajectories,
correctly-localized visual features, and spatial-temporal masking pose
features. We construct a new video HOI benchmark dubbed VidHOI111The dataset
and source code are available at https://github.com/coldmanck/VidHOI where our
proposed approach serves as a solid baseline.

Human-Object Interaction, Action Detection, Video Understanding

††copyright: rightsretained††journalyear: 2021††conference: Proceedings of the
2021 Workshop on Intelligent Cross-Data Analysis and Retrieval; August 21–24,
2021; Taipei, Taiwan††booktitle: Proceedings of the 2021 Workshop on
Intelligent Cross-Data Analysis and Retrieval (ICDAR ’21), August 21–24, 2021,
Taipei, Taiwan††doi: 10.1145/3463944.3469097††isbn:
978-1-4503-8529-9/21/08††ccs: Human-centered computing††ccs: Computing
methodologies Activity recognition and understanding

##  1\. Introduction

Figure 1. An illustrative comparison between conventional HOI methods and our
ST-HOI when inferencing on videos. (a) Traditional HOI approaches (_e.g._ ,
the baseline in (Wan et al., 2019)) take in only the target frame and predict
HOIs based on ROI-pooled visual features. These models are unable to
differentiate between push, pull or lean on in this example due to the lack of
temporal context. (b) ST-HOI takes in not only the target frame but
neighboring frames and exploits temporal context based on trajectories. ST-HOI
can thus differentiate temporal-related interactions and prefers push to other
interactions in this example.  Figure 2. (a) Relative performance change (in
percentage), on different video tasks by replacing 2D-CNN backbones with 3D
ones (blue bars) (Tran et al., 2015; Carreira and Zisserman, 2017;
Feichtenhofer et al., 2019), and on VideoHOI by adding trajectory feature
(tangerine bar). VideoHOI (in triplet mAP) is to detect HOI in videos and was
performed ourselves on our VidHOI benchmark. STAD (Gu et al., 2018) (in
triplet mAP) means Spatial-Temporal Action Detection and was performed on AVA
dataset (Gu et al., 2018). STSGG (Ji et al., 2020) (PredCls mode; in
Recall@20) stands for Spatial-Temporal Scene Graph Generation and was
performed on Action Genome (Ji et al., 2020). (b) An illustration of temporal-
RoI pooling in 3D baselines (_e.g._ (Feichtenhofer et al., 2019)). Temporal
pooling is usually applied to the output of the penultimate layer of a 3D-CNN
(shape of $d\times T\times H\times W$) which average-pools along the time axis
into shape of $d\times 1\times H\times W$, followed by RoI Pooling to obtain
feature maps of shape $d\times 1\times h\times w$. This temporal-RoI pooling,
however, is equivalent to pooling the instance-of-interest feature at the same
location in the keyframe throughout the video segment, which is erroneous for
moving humans and objects.

Thanks to the rapid development of deep learning (Goodfellow et al., 2016; He
et al., 2016), machines are already surpassing or approaching human level
performance in language tasks (Wu et al., 2016), acoustic tasks (Yin et al.,
2019), and vision tasks (_e.g.,_ image classification (He et al., 2015) and
visual place recognition (Chiou et al., 2020)). Researchers thus started to
focus on how to replicate these successes to other semantically higher-level
vision tasks (_e.g._ , visual relationship detection (Lu et al., 2016; Chiou
et al., 2021)) and vision-language tasks (_e.g._ , image captioning (Vinyals
et al., 2015) and visual question answering (Antol et al., 2015)) so that
machines learn not just to recognize the objects but to understand their
relationships and the contexts. Especially, human-object interaction (HOI)
(Smith et al., 2013; Chao et al., 2018; Gao et al., 2018; Qi et al., 2018; Li
et al., 2019; Wang et al., 2019; Wan et al., 2019; Gupta et al., 2019; Li et
al., 2020; Wang et al., 2020; Liao et al., 2020; Ulutan et al., 2020; Zhong et
al., 2020) aiming to detect actions and spatial relations among humans and
salient objects in images/videos has attracted increasing attention, as we
sometimes task machines to understand human behaviors, _e.g.,_ pedestrian
detection (Zhang et al., 2017) and unmanned store systems (Lo and Wang, 2019).

Although there are abundant studies that have achieved success in detecting
HOI in static images, the fact that few of them (Jain et al., 2016; Qi et al.,
2018; Sunkesula et al., 2020) consider temporal information (_i.e._ ,
neighboring frames before/after the target frame) when performed on video data
means they are actually “guessing” temporal-related HOIs with only naive co-
occurrence statistics. While conventional image-based HOI methods (_e.g._ ,
the baseline model in (Wan et al., 2019)) can be used for inference on videos,
they treat input frames as independent and identically distributed (i.i.d.)
data and make independent predictions for neighboring frames. However, video
data are sequential and structured by nature and thus are not i.i.d. What is
worse is that, without temporal context these methods are unable to
differentiate (especially, opposite) temporal interactions, such as push
versus pull a human and open versus close a door. As shown in Figure 1(a),
given a video segment, traditional HOI models operate on a single frame at a
time and make predictions based on 2D-CNN (_e.g._ , (He et al., 2016)) visual
features. These models by nature could not distinguish interactions between
two people such as push, pull, lean on and chase, which are visually similar
in static images. A possible reason causing video-based HOI underexplored is
the lack of a suitable video-based benchmark and a feasible setting. To bridge
this gap, we first construct a video HOI benchmark from VidOR (Shang et al.,
2019), dubbed VidHOI , where we follow the common protocol in video and HOI
tasks to use a keyframe-centered strategy. With VidHOI, we urge the use of
video data and propose VideoHOI as – in both training and inference –
performing HOI detection with videos.

Spatial-Temporal Action Detection (STAD) is another task bearing a resemblance
to VideoHOI by requiring to localize the human and detect the actions being
performed in videos. Note that STAD does not consider the objects that a human
is interacting with. STAD is usually tackled by first using a 3D-CNN (Tran et
al., 2015; Carreira and Zisserman, 2017) as the backbone to encode temporal
information into feature maps. This is followed by RoI pooling with object
proposals to obtain actor features, which are then classified by linear
layers. Essentially, this approach is similar to a common HOI baseline
illustrated in Figure 1(a) and differs only in the use of 3D backbones and the
absence of interacting objects. Based on conventional HOI and STAD methods, a
naive yet intuitive idea arises: can we enjoy the best of both worlds, by
replacing 2D backbones with 3D ones and exploiting visual features of
interacting objects? This idea, however, did not work straightforwardly in our
preliminary experiment, where we replaced the backbone in the 2D baseline (Wan
et al., 2019) with the 3D one (_e.g._ , SlowFast (Feichtenhofer et al., 2019))
to perform VideoHOI. The relative change of performance after replacing the
backbone is presented in the left most entry in Figure 2(a) with a blue bar.
In VideoHOI experiment, the 3D baseline provides only a limited relative
improvement ($\sim$2%), which is far from satisfactory considering the
additional temporal context. In fact, this phenomenon has also been observed
in two existing works under similar settings (Gu et al., 2018; Ji et al.,
2020), where both experiments in STAD and another video task Spatial-Temporal
Scene Graph Generation (STSGG) present an even worse, counter-intuitive
result: replacing the backbone is actually harmful (also presented as blue
bars in Figure 2(a)). We probed the underlying reason by analyzing the
architecture of these 3D baselines and found that, surprisingly, temporal
pooling together with RoI pooling does not work reasonably. As illustrated in
Figure 2(b), temporal pooling followed by RoI pooling, which is a common
practice in conventional STAD methods, is equivalent to cropping features of
the same region across the whole video segment without considering the way
objects move. It is not unusual for moving humans and objects in neighboring
frames to be absent from its location in the target keyframe. Temporal-and-
RoI-pooling features at the same location could be getting erroneous features
such as other humans/objects or meaningless background. Dealing with this
inconsistency, we propose to recover the missing spatial-temporal information
in VideoHOI by considering human and object trajectories. The performance
change of this temporal-augmented 3D baseline on VideoHOI is represented by
the tangerine bar in Figure 2(a), where it achieves $\sim$23% improvement, in
sharp contrast to $\sim$2% of the original 3D baseline. This experiment
reveals the importance of incorporating the ”correctly-localized” temporal
information.

Keeping the aforementioned ideas in mind, in this paper we propose Spatial-
Temporal baseline for Human-Object Interaction detection in videos, or ST-HOI,
which makes accurate HOI prediction with instance-wise spatial-temporal
features based on trajectories. As illustrated in Figure 1(b), three kinds of
such features are exploited in ST-HOI: (a) trajectory features (moving
bounding boxes; shown as the red arrow), (b) correctly-localized visual
features (shown as the yellow arrow), and (c) spatial-temporal actor poses
(shown as the green arrow).

The contribution of our work is three-fold. First, we are among the first to
identify the feature inconsistency issue existing in the naive 3D models which
we address with simple yet “correct” spatial-temporal feature pooling. Second,
we propose a spatial-temporal model which utilizes correctly-localized visual
features, per-frame box coordinates and a novel, temporal-aware masking pose
module to effectively detect video-based HOIs. Third, we establish the
keyframe-based VidHOI benchmark to motivate research in detecting spatial-
temporal aware interactions and hopefully inspire VideoHOI approaches
utilizing the multi-modality data, _i.e._ , video frames, texts (semantic
object/relation labels) and audios.

##  2\. Related Work

###  2.1. Human-Object Interaction (HOI)

HOI Detection aims to reason over interactions between humans (actors) and
target objects. HOI is closely related to visual relationship detection (Lu et
al., 2016; Chiou et al., 2021) and scene graph generation (Xu et al., 2017),
in which the subject in (subject-predicate-object) are not restricted to a
human. HOI in static images has been intensively studied recently (Smith et
al., 2013; Chao et al., 2018; Gao et al., 2018; Qi et al., 2018; Li et al.,
2019; Wang et al., 2019; Wan et al., 2019; Gupta et al., 2019; Li et al.,
2020; Wang et al., 2020; Liao et al., 2020; Ulutan et al., 2020; Zhong et al.,
2020). Most of the existing methods can be divided into two categories by the
order of human-object pair proposal and interaction classification. The first
group (Gao et al., 2018; Chao et al., 2018; Li et al., 2019; Wang et al.,
2019; Wan et al., 2019; Gupta et al., 2019; Ulutan et al., 2020; Li et al.,
2020) performs human-object pair generation followed by interaction
classification, while the second group (Gkioxari et al., 2018; Qi et al.,
2018; Wang et al., 2020; Liao et al., 2020) first predicts the most probable
interactions performed by a person followed by associating them with the most-
likely objects. Our ST-HOI belongs to the first group as we establish a
temporal model based on trajectories (continuous object proposals).

In contrast to the popularity of image-based HOI, there are only a few of
studies in VideoHOI (Koppula and Saxena, 2015; Jain et al., 2016; Qi et al.,
2018; Sunkesula et al., 2020) and, to the best of our knowledge, all of which
conducted experiments on CAD-120 (Koppula et al., 2013) dataset. In CAD-120,
the interactions are defined by merely 10 high-level activities (_e.g._ ,
making cereal or microwaving food) in 120 RGB-D videos. This setting is not
favorable to real-life scenarios where machines may be asked to understand
more fine-grained actions. Moreover, previous methods (Koppula and Saxena,
2015; Jain et al., 2016; Qi et al., 2018) adopted pre-computed hand-crafted
features such as SIFT (Lowe, 2004) which have been outperformed by deep neural
networks, and ground truth features including 3D poses and depth information
from RGB-D videos which are unlikely to be available in real life scenarios.
While (Sunkesula et al., 2020) adopted a ResNet (He et al., 2016) as their
backbone, their method is inefficient by requiring $M\times N$ computation for
extracting $M$ humans’ and $N$ objects’ features. Different from these
existing methods, we evaluate on a larger and more diversified video HOI
benchmark dubbed VidHOI, which includes annotations of 50 predicates on
thousands of videos. We then propose a spatial-temporal HOI baseline that
operates on RGB videos and does not utilize any additional information.

###  2.2. Spatial-Temporal Action Detection (STAD)

STAD aims to localize actors and detect the associated actions (without
considering interacting objects). One of the most popular benchmark for STAD
is AVA (Gu et al., 2018), where the annotation is done at a sampling frequency
of 1 Hz and the performance is measured by framewise mean AP. We followed this
annotation and evaluation style when constructing VidHOI, where we converted
the original labels into the same format.

As explained in section 1, a standard approach to STAD (Tran et al., 2015;
Carreira and Zisserman, 2017) is extracting spatial-temporal feature maps with
a 3D-CNN followed by RoI pooling to crop human features, which are then
classified by linear layers. As shown in Figure 2(a), a naive modification
that incorporates RoI-pooled human/object features does not work for VideoHOI.
In contrast, our ST-HOI tackles VideoHOI by incorporating multiple temporal
features including trajectories, correctly-localized visual features and
spatial-temporal masking pose features.

Figure 3. An illustration of the two proposed spatial-temporal features. (a)
In contrast to performing RoI pooling followed by temporal pooling like (Wu et
al., 2019; Feichtenhofer et al., 2019), we adopt a reverse approach to first
frame-wise RoI-pool instance feature maps using trajectories, which are then
averaged pool along the time axis to get correctly-localized visual features.
(b) With $N$ object trajectories (including $M$ human), for each frame we
utilize a trained human pose prediction model (_e.g._ , (Fang et al., 2017))
to generate 2D actor pose feature and extract a dual spatial mask for all
$M\times(N-1)$ valid pair. The pose feature and the mask are concatenated and
down-sampled, followed by two 3D convolution layers and spatial-temporal
pooling to generate the masking pose features.

###  2.3. Spatial-Temporal Scene Graph Generation

Spatial-Temporal Scene Graph Generation (STSGG) (Ji et al., 2020) aims to
generate symbolic graphs representing pairwise visual relationships in video
frames. A new benchmark, Action Genome, is also proposed in (Ji et al., 2020)
to facilitate researches in STSGG. Ji et al. (Ji et al., 2020) dealt with
STSGG by combining off-the-shelf scene graph generation models with long-term
feature bank (Wu et al., 2019) on top of a 2D- or 3D-CNN, where they found
that the 3D-CNN actually undermines the performance. While observing similar
results in VidHOI (Figure 2(a)), we go one step further to find out the
underlying reason is that RoI features across frames were erroneously pooled.
We correct this by utilizing object trajectories and applying Tube-of-Interest
(ToI) pooling on generated trajectories to obtain correctly-localized position
information and feature maps throughout video segments.

##  3\. Methodology

###  3.1. Overview

We follow STAD approaches (Tran et al., 2015; Carreira and Zisserman, 2017;
Feichtenhofer et al., 2019) to detect VideoHOI in a keyframe-centric strategy.
Denote $V$ as a video which has $T$ keyframes with sampling frequency of 1 Hz
as $\\{I_{t}\\},t=\\{1,...,T\\}$, and denote $C$ as the number of pre-defined
interaction classes. Given $N$ instance trajectories including $M$ human
trajectories ($M\leq N$) in a video segment centered at the target frame, for
human $m\in\\{1,...,M\\}$ and object $n\in\\{1,...,N\\}$ in keyframe $I_{t}$,
we aim to detect pairwise human-object interactions $r_{t}=\\{0,1\\}^{C}$,
where each entry $r_{t,c},c\in\\{1,...,C\\}$ means whether the interaction $c$
exists or not.

Refer to Figure 1(b) for an illustration of our ST-HOI. Our model takes in a
video segment (sequence of $T$ frames) centered at $I_{t}$ and utilizes a
3D-CNN as the backbone to extract spatial-temporal feature maps of the whole
segment. To rectify the mismatch caused by temporal-RoI pooling, based on $N$
object (including human) trajectories
$\\{j_{i}\\},i=\\{1,..,N\\},j_{i}\in\mathbb{R}^{T\times 4}$ we generate
temporal-aware features including correctly-localized features and spatial-
temporal masking pose features. These features together with trajectories are
concatenated and classified by linear layers. Note that we aim at a simple but
effective temporal-aware baseline to VideoHOI so that we do not utilize tricks
in STAD such as non-local block (Wang et al., 2018) or long-term feature bank
(Wu et al., 2019), and in image-based HOI like interactiveness (Li et al.,
2019), though we note that these may be used to boost the performance.

###  3.2. Correctly-localized Visual Features

We have discussed in previous sections on inappropriately pooled RoI features.
We propose to tackle this issue by reversing the order of temporal pooling and
RoI-pooling. This approach has recently been proposed in (Hou et al., 2017)
and named as tube-of-interest pooling (ToIPool). Refer to Figure 3(a) for an
illustration. Denote $v\in\mathbb{R}^{d\times T\times H\times W}$ as the
output of the penultimate layer of our 3D-CNN backbone, and denote
$v_{t}\in\mathbb{R}^{d\times H\times W}$ as the $t$-th feature map along the
time axis. Recall that we have $N$ trajectories centered at a keyframe.
Following the conventional way, we also exploit visual context when predicting
an interaction, which is done by utilizing the union bounding box feature of a
human and an object. For example, the sky between human and kite could help to
infer the correct interaction fly. Recall that $j_{i}$ represents the
trajectory of object $i$, where we further denote $j_{i,t}$ as the 2D bounding
box at time $t$. The spatial-temporal instance features $\\{\bar{v}_{i}\\}$
are then obtained using ToIPool with RoIAlign (He et al., 2017) by

(1) $\bar{v}_{i}=\frac{1}{T}\sum_{t=1}^{T}\text{RoIAlign}(v_{t},j_{i,t}),$

where $\bar{v}_{i}\in\mathbb{R}^{d\times h\times w}$ and $h$ and $w$ means
height and width of the pooled feature maps, respectively. $\bar{v}_{i}$ is
flattened before concatenating with other features.

###  3.3. Spatial-Temporal Masking Pose Features

Human poses have been widely utilized in image-based HOI methods (Li et al.,
2019; Gupta et al., 2019; Wan et al., 2019) to exploit characteristic actor
pose to infer some special actions. In addition, some existing works (Wang et
al., 2019; Wan et al., 2019) found that spatial information can be used to
identify interactions. For instance, for human-ride-horse one can imagine the
actor’s skeleton as legs widely open (on horse sides), and the bounding box
center of human is usually on top of that of horse. However, none of the
existing works consider this mechanism in temporal domain: when riding a horse
the human should be moving with horse as a whole. We argue that this
temporality is an important property and should be utilized as well.

The spatial-temporal masking pose module is presented at Figure 3(b). Given
$M$ human trajectories, we first generate $M$ spatial-temporal pose features
with a trained human pose prediction model. On frame $t$, the predicted human
pose $h_{i,t}\in\mathbb{R}^{17\times 2},i=\\{1,..,M\\},t=\\{1,..,T\\}$ is
defined as 17 joint points mapped to the original image. We transform
$h_{i,t}$ into a skeleton on a binary mask with
$f_{h}:\\{h_{i,t}\\}\in\mathbb{R}^{17\times
2}\to\\{\bar{h}_{i,t}\\}\in\mathbb{R}^{1\times H\times W}$, by connecting the
joints using lines, where each line has a distinct value $x\in[0,1]$. This
helps the model to recognize and differentiate different poses.

For each of $M\times(N-1)$ valid human-object pairs on frame $t$, we also
generate two spatial masks $s_{i,t}\in\mathbb{R}^{2\times H\times
W},i=\\{1,...,M\times(N-1)\\}$ corresponding to human and object respectively,
where the values inside of each bounding box are ones and outsides are zeroed-
out. These masks enable our model to predict HOI with reference to important
spatial information.

For each pair, we concatenate the skeleton mask $\bar{h}_{i,t}$ and spatial
masks $s_{i,t}$ along the first dimension to get the initial spatial masking
pose feature $p_{i,t}\in\mathbb{R}^{3\times H\times W}$:

(2) $p_{i,t}=[s_{i,t};\bar{h}_{i,t}].$

We then down-sample $\\{p_{i,t}\\}$, feed into two 3D convolutional layers
with spatial and temporal pooling, and flatten to obtain the final spatial-
temporal masking pose feature $\\{\bar{p}_{i,t}\\}$.

Table 1. A comparison of our benchmark VidHOI with existing STAD (AVA (Gu et
al., 2018)), image-based (HICO-DET (Chao et al., 2018) and V-COCO (Gupta and
Malik, 2015)) and video-based (CAD-120 (Koppula et al., 2013) and Action
Genome (Ji et al., 2020)) HOI datasets. VidHOI is the only dataset that
provides temporal information from video clips and complete multi-person and
interacting-object annotations. VidHOI also provides the most annotated
keyframes and defines the most HOI categories in the existing video datasets.
$\dagger$Two less categories as we combine adult, child and baby into a single
category, person.

Dataset | Video | Localized | Video | # Videos | # Annotated | # Objects | # Predicate | # HOI | # HOI
---|---|---|---|---|---|---|---|---|---
dataset? | object? | hours |  | images/frames | categories | categories | categories | Instances
HICO-DET (Chao et al., 2018) | ✗ | ✓ | - | - | 47K | 80 | 117 | 600 | 150K
V-COCO (Gupta and Malik, 2015) | ✗ | ✓ | - | - | 10K | 80 | 25 | 259 | 16K
AVA (Gu et al., 2018) | ✓ | ✗ | 108 | 437 | 3.7M | - | 49 | 80 | 1.6M
CAD-120 (Koppula et al., 2013) | ✓ | ✓ | 0.57 | 0.5K | 61K | 13 | 6 | 10 | 32K
Action Genome (Ji et al., 2020) | ✓ | $\bigtriangleup$ | 82 | 10K | 234K | 35 | 25 | 157 | 1.7M
VidHOI | ✓ | ✓ | 70 | 7122 | 7.3M | 78$\dagger$ | 50 | 557 | 755K

###  3.4. Prediction

We fuse the aforementioned features, including correctly-localized visual
features $\bar{v}$, spatial-temporal masking pose features $p$, and instance
trajectories $j$ by concatenating them along the last axis

(3)
$v_{\text{so}}=[\bar{v}_{s};\bar{v}_{u};\bar{v}_{o};j_{s};j_{o};\bar{p}_{so}],\\\
$

where we slightly abuse the notation to denote the subscriptions $s$ as the
subject, $o$ as the object and $u$ as their union region. $v_{\text{so}}$ is
then fed into two linear layers with the final output size being the number of
interaction classes in the dataset. Since VideoHOI is essentially a multi-
label learning task, we train the model with per-class binary cross entropy
loss.

During inference, we follow the heuristics in image-based HOI (Chao et al.,
2018) to sort all the possible pairs by their softmax scores and evaluate on
only top 100 predictions.

##  4\. Experiments

###  4.1. Dataset and Performance Metric

While we have discussed in section 2.1 about the problem of lacking a suitable
VideoHOI dataset by analyzing CAD-120 (Koppula et al., 2013), we further
explain why Action Genome (Ji et al., 2020) is also not a feasible choice
here. First, the authors acknowledged that the dataset is still incomplete and
contains incorrect labels (Ji, 2020). Second, Action Genome is produced by
annotating Charades (Sigurdsson et al., 2016), which is originally designed
for activity classification where each clip contains only one ”actor”
performing predefined tasks; should any other people show up, there are
neither any bounding box nor interaction label about them. Finally, the videos
are purposedly-generated by volunteers, which are rather unnatural. In
contrast, VidHOI are based on VidOR (Shang et al., 2019) which is densely
annotated with all humans and predefined objects showing up in each frame.
VidOR is also more challenging as the videos are non-volunteering user-
generated and thus jittery at times. A comparison of VidHOI and the existing
STAD and HOI datasets is presented in Table 1.

VidOR is originally collected for video visual relationship detection where
the evaluation is trajectory-based. The volumetric Interaction Over Union
(vIOU) between a trajectory and a ground truth needs to be over 0.5 before
considering its relationship prediction; however, how to obtain accurate
trajectories with correct start- and end-timestamp remains challenging (Sun et
al., 2019; Shang et al., 2017). We notice that some image-based HOI datasets
(_e.g._ , HICO-DET (Chao et al., 2018) and V-COCO (Gupta and Malik, 2015)) as
well as STAD datasets (_e.g._ , AVA (Gu et al., 2018)) are using a keyframe-
centered evaluation strategy, which bypasses the aforementioned issue. We thus
adopt the same and follow AVA to sample keyframes at a 1 FPS frequency, where
the annotations on the keyframe at timestamp $t$ are assumed to be fixed for
$t\pm 0.5$s. In detail, we first filter out those keyframes without presenting
at least one valid human-object pair, followed by transforming the labels from
video clip-based to keyframe-based to align with common HOI metrics (_i.e._ ,
frame mAP). We follow the original VidOR split in (Shang et al., 2019) to
divide VidHOI into a training set comprising 193,911 keyframes in 6,366 videos
and a validation set222The VidOR testing set is not available publicly. with
22,808 keyframes in 756 videos. As shown in Figure 4, there are 50 relation
classes including actions (_e.g._ , push, pull, lift, etc.) and spatial
relations (_e.g._ , next to, behind, etc.). While half (25) of the predicate
classes are temporal-related, they account for merely $\sim$5% of the dataset.

Figure 4. Predicate distribution of the VidHOI benchmark shows that most of
the predicates are non-temporal-related. Figure 5. Performance comparison in
predicate-wise AP (pAP). The performance boost after adding trajectory
features is observed for most of the predicates. Interestingly, both spatial
(_e.g._ , next to, behind) and temporal (_e.g._ , towards, away) predicates
benefit from the temporal-aware features. Predicates sorted by the number of
occurrence. Models in Oracle mode.

Following the evaluation metric in HICO-DET, we adopt mean Average Precision
(mAP), where a true positive HOI needs to meet three below criteria: (a) both
the predicted human and object bounding boxes have to overlap with the ground
truth boxes with IOU over 0.5, (b) the predicted target category need to be
matched and (c) the predicted interaction is correct. Over 50 predicates, we
follow HICO-DET to define HOI categories as 557 triplets on which we compute
mean AP. By defining HOI categories with triplets we can bypass the polysemy
problem (Zhong et al., 2020), _i.e._ , the same predicate word can represent
very different meaning when pairing with distinct objects, _e.g._ , person-
fly-kite and person-fly-airplane. We report the mean AP over three categories:
(a) Full: all 557 categories are evaluated, (b) Rare: 315 categories with less
than 25 instances in the dataset, and (c) Non-rare: 242 categories with more
than or equal to 25 instances in the dataset. We also examine the models in
two evaluation modes: Oracle models are trained and tested with ground truth
trajectories, while models in Detection mode are tested with predicted
trajectories.

###  4.2. Implementation Details

We adopt Resnet-50 (He et al., 2016) as our 2D backbone for the preliminary
experiments, and utilize Resnet-50-based SlowFast (Feichtenhofer et al., 2019)
as our 3D backbone for all the other experiments. SlowFast contains the Slow
and Fast pathways, which correspond to the texture details and the temporal
information, respectively, by sampling video frames in different frequencies.
For a 64-frame segment centered at the keyframe, $T=32$ frames are alternately
sampled to feed into the Slow pathway; only $T/\alpha$ frames are fed into the
Fast pathway, where $\alpha=8$ in our experiments. We use FastPose (Fang et
al., 2017) to predict human poses and adopt the predicted trajectories
generated by a cascaded model of video object detection, temporal NMS and
tracking algorithm (Sun et al., 2019). Like object detection is to 2D HOI
detection, trajectory generation is an essential module but not a main focus
of this work. If a bounding box is not available in neighboring frames (_i.e._
, the trajectory is shorter than $T$ or not continuous throughout the
segment), we fill it with the whole-image as a box. We train all models from
scratch for 20 epochs with the initial learning rate $1\times 10^{-2}$, where
we use step decay learning rate to reduce the learning rate by $10\times$ at
the $10^{\text{th}}$ and $15^{\text{th}}$ epoch. We optimize our models using
synchronized SGD with momentum of 0.9 and weight decay of $10^{-7}$. We train
each 3D video model with eight NVIDIA Tesla V100 GPUs with batch size being
128 (_i.e._ , 16 examples per GPU), except for the full model where we set
batch size as 112 due to the memory restriction. We train the 2D model with a
single V100 with batch size being 128.

During training, following the strategy in SlowFast we randomly scale the
shorter side of the video to a value in $[256,320]$ pixels, followed by random
horizontal flipping and random cropping into $224\times 224$ pixels. During
inference, we only resize the shorter side of the video segment to 224 pixels.

###  4.3. Quantitative Results

Table 2. Results of the baselines and our ST-HOI on VidHOI validation set (numbers in mAP). There are two evaluation modes: Detection and Oracle, which differ only in the use of predicted or ground truth trajectories during inference. T: Trajectory features. V: Correctly-localized visual features. P: Spatial-temporal masking pose features. ”%” means the full mAP change compared to the 2D model. | Model | Full | Non-rare | Rare | %
---|---|---|---|---|---
Oracle | 2D model (Wan et al., 2019) | 14.1 | 22.9 | 11.3 | -
3D model | 14.4 | 23.0 | 12.6 | 2.1
Ours-T | 17.3 | 26.9 | 16.8 | 22.7
Ours-T+V | 17.3 | 26.9 | 16.3 | 22.7
Ours-T+P | 17.4 | 27.1 | 16.4 | 23.4
Ours-T+V+P | 17.6 | 27.2 | 17.3 | 24.8
Detection | 2D model (Wan et al., 2019) | 2.6 | 4.7 | 1.7 | -
3D model | 2.6 | 4.9 | 1.9 | 0.0
Ours-T | 3.0 | 5.5 | 2.0 | 15.4
Ours-T+V | 3.1 | 5.8 | 2.0 | 19.2
Ours-T+P | 3.2 | 6.1 | 2.0 | 23.1
Ours-T+V+P | 3.1 | 5.9 | 2.1 | 19.2

Since we aim to deal with a) the lack of temporal-aware features in 2D HOI
methods, b) the feature inconsistency issue in common 3D HOI methods and c)
the lack of a VideoHOI benchmark, we mainly compare with the 2D model (Wan et
al., 2019) and its naive 3D variant on VidHOI to understand if our ST-HOI
addresses these issues effectively.

The performance comparison between our full ST-HOI model (Ours-T+V+P) and
baselines (2D model, 3D model) are presented in Table LABEL:tab:results, in
which we also present ablation studies on our different features (modules)
inlcuding trajectory features (T), correctly-localized visual features (V) and
spatial-temporal masking pose features (P). Table LABEL:tab:results shows that
3D model only has a marginal improvement compared to 2D model (overall
$\sim$2%) under all settings in both evaluation modes. In contrast, adding
trajectory features (Ours-T) leads to a much larger 23% improvement in Oracle
mode or 15% in Detection mode, showing the importance of correct spatial-
temporal information. We also find that by adding additional temporal-aware
features (_i.e.,_ V and P) increasingly higher mAPs are attained, and our full
model (Ours-T+V+P) reports the best mAPs in Oracle mode, achieving the highest
$\sim$25% relative improvement. We notice that the performance of Ours-T+V is
close to that of Ours-T under Oracle setting, which is possibly because the
ground truth trajectories (T) have provided enough “correctly-localized”
information so that the correct features do not help much. We also note that
the performance of Ours-T+P is slightly higher than that of Ours-T+V+P under
Detection mode, which is assumably due to the same, aforementioned reason and
the inferior performance resulting from the predicted trajectories. The
overall performance gap between Detection and Oracle models is significant,
indicating the room for improvement in trajectory generation. Another
interesting observation is that Full mAPs are very close to Rare mAPs,
especially under Oracle mode, showing that the long-tail effect over HOIs is
strong (but common and natural).

To understand the effect of temporal features on individual predicates, we
compare with predicate-wise AP (pAP) shown in Figure 5. We observe that,
again, under most of circumstances naively replacing 2D backbones with 3D ones
does not help video HOI detection. Both temporal predicates (_e.g._ , towards,
away, pull) and spatial (_e.g._ , next_to, behind, beneath) predicates benefit
from the additional temporal-aware features in ST-HOI. These findings verify
our main idea about the essential use of trajectories and trajectory-based
features. In addition, each additional features do not seem to contribute
equally for different predicates. For instance, we see that while Ours-T+V+P
performs the best on some predicates (_e.g._ , behind and beneath), our sub-
models achieve the highest mAP on other predicates (_e.g._ , watch and ride).
This is assumedly because predicate-wise performance is heavily subject to the
number of examples, where major predicates have 10-10000 times more examples
than minor ones (as shown in Figure 4).

Figure 6. Results (in predicate-wise AP) of the baselines and our full model w.r.t. top frequent temporal predicates. Table 3. Results of temporal-related and spatial (non-temporal) related triplet mAP. T%/S% means relative temporal/spatial mAP change compared to 2D model (Wan et al., 2019). |  | Temporal | T% | Spatial | S%
---|---|---|---|---|---
Oracle | 2D model (Wan et al., 2019) | 8.3 | - | 18.6 | -
3D model | 7.7 | -7.2 | 20.9 | 12.3
Ours-T | 14.4 | 73.5 | 24.7 | 32.8
Ours-T+V | 13.6 | 63.9 | 24.6 | 32.3
Ours-T+P | 12.9 | 55.4 | 25.0 | 34.4
Ours-T+V+P | 14.4 | 73.5 | 25.0 | 34.4
Detection | 2D model (Wan et al., 2019) | 1.5 | - | 2.7 | -
3D model | 1.6 | 6.7 | 2.9 | 7.4
Ours-T | 1.8 | 20.0 | 3.3 | 23.6
Ours-T+V | 1.8 | 20.0 | 3.3 | 23.6
Ours-T+P | 1.8 | 20.0 | 3.3 | 23.6
Ours-T+V+P | 1.9 | 26.7 | 3.3 | 23.6

Since the majority of HOI examples are spatial-related ($\sim$95%, as shown in
Figure 4), the results above might not be suitable for demonstrating the
temporal modeling ability of our proposed model. We thus focus on the
performance on only temporal-related predicates in Figure 6, which
demonstrates that Ours-T+V+P greatly outperforms the baselines regarding the
top frequent temporal predicates. Table LABEL:tab:temporal_spatial_results
presents the triplet mAPs of spatial- or temporal-only predicates, showing
Ours-T significantly improves the 2D model on temporal-only mAP by relative
+73.9%, in sharp contrast to -7.1% of the 3D model in Oracle mode. Similar to
our observation with Table LABEL:tab:results, Ours-T performs on par with
Ours-T+V+P for temporal-only predicates; however, it falls short of spatial-
only predicates, showing that spatial/pose information is still essential for
detecting spatial predicates. Overall, these results demonstrate the
outstanding spatial-temporal modeling ability of our approach.

Figure 7.  Performance comparison (in AP) of some temporal-related HOIs in
VidHOI validation set. Compared to 2D model, 3D model only shows limited
improvement for the presented examples, while our ST-HOI variants provide huge
performance boost. Models are in Oracle mode.  Figure 8. Examples of video
HOIs predicted by the 2D model (Wan et al., 2019) and our ST-HOI, both in
Oracle mode. Each consists of five consecutive keyframes sampled in 1 Hz,
where an entry in tables denotes whether a predicate between the subject
(human; a green box) and the object (also human in both cases; a red box) is
detected correctly (True Positive) or not (False Positive or False Negative).
Compared to the 2D baseline, our model predicts more accurate temporal HOIs
(_e.g._ , hold_hand_of in $T_{4}$ and $T_{5}$ of the upper example and lift in
$T_{1}$ of the lower example). ST-HOI also produces less false positives in
both examples.

We also compare the performance with respect to some HOI triplets in Figure 7.
Similar to the results on predicate-wise mAP, we also observe the large gap
between naive 2D/3D models and our models with the temporal features. ST-HOI
variants are more accurate in predicting especially temporal-aware HOIs
(hug/lean_on-person and push/pull-baby_walker). We also see in some examples
that Ours-T+V+P does not perform the best among all the variants, _e.g._ ,
lean_on-person), which is similar to the phenomenon we observed in Figure 5.

###  4.4. Qualitative Results

To understand the effectiveness of our proposed method, we visualize two video
HOI examples of VidHOI predicted by the 2D model (Wan et al., 2019) and
Ours-T+V+P (both in Oracle mode) in Figure 8. Each (upper and lower) example
is a 5-second video segment (_i.e._ , five keyframes) with a HOI prediction
table where each entry means either True Positive (TP), False Positive (FP),
False Negative (FN) or True Negative (TN) for both models. The upper example
shows that, compared to the 2D model, Ours-T+V+P makes more accurate HOI
detection by successfully predicting hold_hand_of at $T_{4}$ and $T_{5}$.
Moreover, Ours-T+V+P is able to predict interactions that requires temporal
information, such as lift at $T_{1}$ in the lower example. However, we can see
that there is still room for improvement for Ours-T+V+P in the same example,
where lift is not detected in the following $T_{2}$ to $T_{4}$ frames.
Overall, our model produces less false positives throughout the dataset, which
in turn contributes to its higher mAP and pAP.

##  5\. Conclusion

In this paper, we addressed the inability of conventional HOI approaches to
recognize temporal-aware interactions by re-focusing on neighboring video
frames. We discussed the lack of a suitable setting and dataset for studying
video-based HOI detection. We also identified a feature-inconsistency problem
in a common video action detection baseline which arises from its improper
order of RoI feature pooling and temporal pooling. To deal with the first
issue, we established a new video HOI benchmark dubbed VidHOI and introduced a
keyframe-centered detection strategy. We then proposed a spatial-temporal
baseline ST-HOI which exploits trajectory-based temporal features including
correctly-localized visual features, spatial-temporal masking pose features
and trajectory features, solving the second problem. With quantitative and
qualitative experiments on VidHOI, we showed that our model provides a huge
performance boost compared to both the 2D and 3D baselines and is effective in
differentiating temporal-related interactions. We expect that the proposed
baseline and the dataset would serve as a solid starting point for the
relatively underexplored VideoHOI task. Based on our baseline, we also hope to
motivate further VideoHOI works to design advanced models with the multi-modal
data including video frames, semantic object/relation labels and audios.

## Acknowledgment

This research is supported by Singapore Ministry of Education Academic
Research Fund Tier 1 under MOE’s official grant number T1 251RES2029.

## References

  * (1)
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering. In _Proceedings of the IEEE international conference on computer vision_. 2425–2433.
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo vadis, action recognition? a new model and the kinetics dataset. In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6299–6308.
  * Chao et al. (2018) Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. 2018.  Learning to detect human-object interactions. In _2018 ieee winter conference on applications of computer vision (wacv)_. IEEE, 381–389.
  * Chiou et al. (2020) Meng-Jiun Chiou, Zhenguang Liu, Yifang Yin, An-An Liu, and Roger Zimmermann. 2020.  Zero-Shot Multi-View Indoor Localization via Graph Location Networks. In _Proceedings of the 28th ACM International Conference on Multimedia_. 3431–3440.
  * Chiou et al. (2021) Meng-Jiun Chiou, Roger Zimmermann, and Jiashi Feng. 2021\.  Visual Relationship Detection With Visual-Linguistic Knowledge From Multimodal Representations.  _IEEE Access_ 9 (2021), 50441–50451.
  * Fang et al. (2017) Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017\.  RMPE: Regional Multi-person Pose Estimation. In _ICCV_.
  * Feichtenhofer et al. (2019) Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. 2019.  Slowfast networks for video recognition. In _Proceedings of the IEEE international conference on computer vision_. 6202–6211.
  * Gao et al. (2018) Chen Gao, Yuliang Zou, and Jia-Bin Huang. 2018.  iCAN: Instance-centric attention network for human-object interaction detection.  _arXiv preprint arXiv:1808.10437_ (2018).
  * Gkioxari et al. (2018) Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. 2018.  Detecting and recognizing human-object interactions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8359–8367.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016\.  _Deep Learning_.  MIT Press.  http://www.deeplearningbook.org.
  * Gu et al. (2018) Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. 2018\.  Ava: A video dataset of spatio-temporally localized atomic visual actions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6047–6056.
  * Gupta and Malik (2015) Saurabh Gupta and Jitendra Malik. 2015.  Visual Semantic Role Labeling.  _arXiv preprint arXiv:1505.04474_ (2015).
  * Gupta et al. (2019) Tanmay Gupta, Alexander Schwing, and Derek Hoiem. 2019\.  No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In _Proceedings of the IEEE International Conference on Computer Vision_. 9677–9685.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017.  Mask r-cnn. In _Proceedings of the IEEE international conference on computer vision_. 2961–2969.
  * He et al. (2015) K. He, X. Zhang, S. Ren, and J. Sun. 2015\.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In _2015 IEEE International Conference on Computer Vision (ICCV)_. 1026–1034.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hou et al. (2017) Rui Hou, Chen Chen, and Mubarak Shah. 2017.  Tube convolutional neural network (T-CNN) for action detection in videos. In _Proceedings of the IEEE international conference on computer vision_. 5822–5831.
  * Jain et al. (2016) Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. 2016.  Structural-rnn: Deep learning on spatio-temporal graphs. In _Proceedings of the ieee conference on computer vision and pattern recognition_. 5308–5317.
  * Ji (2020) Jingwei Ji. 2020\.  Question about the annotations · Issue #2 · JingweiJ/ActionGenome.  https://github.com/JingweiJ/ActionGenome/issues/2
  * Ji et al. (2020) Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. 2020.  Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 10236–10247.
  * Koppula et al. (2013) Hema Swetha Koppula, Rudhir Gupta, and Ashutosh Saxena. 2013\.  Learning human activities and object affordances from rgb-d videos.  _The International Journal of Robotics Research_ 32, 8 (2013), 951–970.
  * Koppula and Saxena (2015) Hema S Koppula and Ashutosh Saxena. 2015.  Anticipating human activities using object affordances for reactive robotic response.  _IEEE transactions on pattern analysis and machine intelligence_ 38, 1 (2015), 14–29.
  * Li et al. (2020) Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang, Ze Ma, Mingyang Chen, and Cewu Lu. 2020\.  PaStaNet: Toward Human Activity Knowledge Engine. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 382–391.
  * Li et al. (2019) Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. 2019\.  Transferable interactiveness knowledge for human-object interaction detection. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3585–3594.
  * Liao et al. (2020) Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. 2020\.  Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 482–490.
  * Lo and Wang (2019) Chi-Hung Lo and Yi-Wen Wang. 2019.  Constructing an Evaluation Model for User Experience in an Unmanned Store.  _Sustainability_ 11, 18 (2019).
  * Lowe (2004) David G Lowe. 2004\.  Distinctive image features from scale-invariant keypoints.  _International journal of computer vision_ 60, 2 (2004), 91–110.
  * Lu et al. (2016) Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016\.  Visual relationship detection with language priors. In _European conference on computer vision_. Springer, 852–869.
  * Qi et al. (2018) Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. 2018.  Learning human-object interactions by graph parsing neural networks. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 401–417.
  * Shang et al. (2019) Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. 2019\.  Annotating objects and relations in user-generated videos. In _Proceedings of the 2019 on International Conference on Multimedia Retrieval_. 279–287.
  * Shang et al. (2017) Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. 2017.  Video visual relation detection. In _Proceedings of the 25th ACM international conference on Multimedia_. 1300–1308.
  * Sigurdsson et al. (2016) Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016.  Hollywood in homes: Crowdsourcing data collection for activity understanding. In _European Conference on Computer Vision_. Springer, 510–526.
  * Smith et al. (2013) Brian A Smith, Qi Yin, Steven K Feiner, and Shree K Nayar. 2013\.  Gaze locking: passive eye contact detection for human-object interaction. In _Proceedings of the 26th annual ACM symposium on User interface software and technology_. 271–280.
  * Sun et al. (2019) Xu Sun, Tongwei Ren, Yuan Zi, and Gangshan Wu. 2019\.  Video visual relation detection via multi-modal feature fusion. In _Proceedings of the 27th ACM International Conference on Multimedia_. 2657–2661.
  * Sunkesula et al. (2020) Sai Praneeth Reddy Sunkesula, Rishabh Dabral, and Ganesh Ramakrishnan. 2020.  LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos. In _Proceedings of the 28th ACM International Conference on Multimedia_. 691–699.
  * Tran et al. (2015) Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015.  Learning spatiotemporal features with 3d convolutional networks. In _Proceedings of the IEEE international conference on computer vision_. 4489–4497.
  * Ulutan et al. (2020) Oytun Ulutan, ASM Iftekhar, and Bangalore S Manjunath. 2020\.  VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13617–13626.
  * Vinyals et al. (2015) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015.  Show and tell: A neural image caption generator. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3156–3164.
  * Wan et al. (2019) Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. 2019.  Pose-aware multi-level feature network for human object interaction detection. In _Proceedings of the IEEE International Conference on Computer Vision_. 9469–9478.
  * Wang et al. (2019) Tiancai Wang, Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang, Ling Shao, and Jorma Laaksonen. 2019.  Deep contextual attention for human-object interaction detection. In _Proceedings of the IEEE International Conference on Computer Vision_. 5694–5702.
  * Wang et al. (2020) Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. 2020\.  Learning Human-Object Interaction Detection using Interaction Points. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 4116–4125.
  * Wang et al. (2018) Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018.  Non-local neural networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 7794–7803.
  * Wu et al. (2019) Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. 2019\.  Long-term feature banks for detailed video understanding. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 284–293.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_ (2016).
  * Xu et al. (2017) Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. 2017\.  Scene graph generation by iterative message passing. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 5410–5419.
  * Yin et al. (2019) Yifang Yin, Meng-Jiun Chiou, Zhenguang Liu, Harsh Shrivastava, Rajiv Ratn Shah, and Roger Zimmermann. 2019\.  Multi-level fusion based class-aware attention model for weakly labeled audio tagging. In _Proceedings of the 27th ACM International Conference on Multimedia_. 1304–1312.
  * Zhang et al. (2017) Shanshan Zhang, Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. 2017.  Towards reaching human performance in pedestrian detection.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 4 (2017), 973–986.
  * Zhong et al. (2020) Xubin Zhong, Changxing Ding, Xian Qu, and Dacheng Tao. 2020\.  Polysemy deciphering network for human-object interaction detection. In _Proc. Eur. Conf. Comput. Vis_.


11institutetext: Sano Centre for Computational Medicine, Cracow, Poland
11email<EMAIL_ADDRESS>22institutetext: Informatics Institute,
University of Amsterdam, Amsterdam, The Netherlands 33institutetext: Amsterdam
University Medical Center, Amsterdam, The Netherlands 44institutetext: The
Medical Centre of Postgraduate Education, Warsaw, Poland 55institutetext:
Medical University of Warsaw, Warsaw, Poland 66institutetext: Warsaw
University of Technology, Warsaw, Poland 77institutetext: IDEAS NCBR, Warsaw,
Poland 88institutetext: Tooploox, Wroclaw, Poland 99institutetext:
Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA

# TabAttention: Learning Attention Conditionally on Tabular Data

Michal K. Grzeszczyk 11 Szymon Płotka 11 2 2 3 3 Beata Rebizant 44 Katarzyna
Kosińska-Kaczyńska 44 Michał Lipa 55 Robert Brawura-Biskupski-Samaha 44
Przemysław Korzeniowski 11 Tomasz Trzciński 66 7 7 8 8 Arkadiusz Sitek 99

###### Abstract

Medical data analysis often combines both imaging and tabular data processing
using machine learning algorithms. While previous studies have investigated
the impact of attention mechanisms on deep learning models, few have explored
integrating attention modules and tabular data. In this paper, we introduce
TabAttention, a novel module that enhances the performance of Convolutional
Neural Networks (CNNs) with an attention mechanism that is trained
conditionally on tabular data. Specifically, we extend the Convolutional Block
Attention Module to 3D by adding a Temporal Attention Module that uses multi-
head self-attention to learn attention maps. Furthermore, we enhance all
attention modules by integrating tabular data embeddings. Our approach is
demonstrated on the fetal birth weight (FBW) estimation task, using 92 fetal
abdominal ultrasound video scans and fetal biometry measurements. Our results
indicate that TabAttention outperforms clinicians and existing methods that
rely on tabular and/or imaging data for FBW prediction. This novel approach
has the potential to improve computer-aided diagnosis in various clinical
workflows where imaging and tabular data are combined. We provide a source
code for integrating TabAttention in CNNs at
https://github.com/SanoScience/Tab-Attention.

###### Keywords:

Attention Fetal Ultrasound Tabular Data

##  1 Introduction

Many clinical procedures involve collecting data samples in the form of
imaging and tabular data. New deep learning (DL) architectures fusing image
and non-image data are being developed to extract knowledge from both sources
of information and improve predictive capabilities [9]. While concatenation of
tabular and imaging features in final layers is widely used [8, 11], this
approach limits the interaction between them. To facilitate better knowledge
transfer between these modalities more advanced techniques have been proposed.
Duanmu et al. [5] presented the Interactive network in which tabular features
are passed through a separate branch and channel-wise multiplied with imaging
features at different stages of Convolutional Neural Network (CNN). Pölsterl
et al. [16] proposed a Dynamic Affine Feature Map Transform (DAFT) to shift
and scale feature maps conditionally on tabular data. In [6], Guan et al.
presented a method for transforming tabular data and processing them together
with 3D feature maps via VisText self-attention module. The importance of the
attention mechanism on DL models’ performance has been extensively studied
[23]. Convolutional Block Attention Module (CBAM) [25] has been shown to
improve the performance of DL models on high dimensional data [24, 26].
Despite these advances, few studies have explored the potential of
incorporating attention maps with imaging and tabular data simultaneously.

We develop such a solution and as an example of application, we use fetal
birth weight (FBW) prediction from ultrasound (US) data. It is a challenging
task requiring clinicians to collect US videos of fetal body parts and fetal
biometry measurements. Currently, abdominal circumference (AC), head
circumference (HC), biparietal diameter (BPD), and femur length (FL) are used
to estimate FBW with heuristic formulae [7]. The predicted weight is the
indicator of perinatal health prognosis or complications in pregnancy and has
an impact on the method of delivery (vaginal or Cesarean) [17]. Unfortunately,
the current approach to FBW estimation is often imprecise and can lead to a
mean absolute percentage error (MAPE) of 10%, even if performed by experienced
sonographers [20]. An ensemble of Machine Learning algorithms was proposed by
Lu et al. [12] for solving this task. CNNs are applied for fetal biometry
measurements estimation from US standard planes [1] or US videos [15]. Tao et
al. [21] approach this problem with a recurrent network utilizing temporal
features of fetal weight changes over weeks concatenated with fetal
parameters. Płotka et al. [14] developed BabyNet, a hybrid CNN with
Transformer layers to estimate FBW directly from US videos. Recent studies
show that there is a strong correlation between the image features of the
abdominal plane and the estimated fetal weight, indicating that it can serve
as a dependable indicator for evaluating fetal growth [3]. We utilize the US
videos of the abdomen (imaging data) and biometry measurements with other
numerical values (tabular data) during our experiments.

In this work, we introduce TabAttention, a novel module designed to enhance
the performance of CNNs by incorporating tabular data. TabAttention extends
the CBAM to the temporal dimension by adding a Temporal Attention Module (TAM)
that leverages Multi-Head Self-Attention (MHSA) [23]. Our method utilizes
pooled information from imaging feature maps and tabular data (represented as
tabular embeddings) to generate attention maps through Channel Attention
Module (CAM), Spatial Attention Module (SAM), and TAM. By incorporating
tabular data, TabAttention enables the network to better identify what, where,
and when to focus on, thereby improving performance. We evaluate our method on
the task of estimating FBW from abdominal US videos and demonstrate that
TabAttention is at least on par with existing methods, including those based
on tabular and/or imaging data, as well as clinicians. The main contributions
of our work are: 1) the introduction of TabAttention, a module for conditional
attention learning with tabular data, 2) the extension of CBAM to the temporal
dimension via the TAM module, and 3) the validation of our method on the FBW
estimation task, where we demonstrate that it is competitive with state-of-
the-art methods.

##  2 Method

Figure 1: The overview of proposed TabAttention module inspired by CBAM [25].
We add a Temporal Attention Module to the existing architecture to extend the
method to 3D data processing. In our setting, channel, spatial and temporal
attention maps are trained conditionally on tabular data. $\otimes$ indicates
element-wise multiplication.

In this section, we introduce the fundamental components of the TabAttention
module. We detail the development of CBAM augmented with a Temporal Attention
Module. Then, we elaborate on how TabAttention leverages tabular embeddings to
modulate the creation of attention maps and outline how the module can be
seamlessly incorporated into the residual block of ResNet.

Fig. 1 presents the overview of the TabAttention module. Given US video
sequence $S\in\mathbb{R}^{T_{0}\times 1\times H_{0}\times W_{0}}$ of height
$H_{0}$, width $W_{0}$ and frame number $T_{0}$ as the input, 3D CNN produces
intermediate temporal feature maps $S^{\prime}\in\mathbb{R}^{T\times C\times
H\times W}$ where $C$ is the number of channels. In our setting, the CBAM
block generates $T$ 1D channel attention maps $M_{c}\in\mathbb{R}^{T\times
C\times 1\times 1}$ and $T$ 2D spatial attention maps
$M_{s}\in\mathbb{R}^{T\times 1\times H\times W}$. We create attention maps
separately for every temporal feature map as the information of what is
meaningful and where it is important to focus on might change along the
temporal dimension. To account for the temporal changes and focus on when is
the informative part we add TAM which infers temporal attention map
$M_{t}\in\mathbb{R}^{T\times 1\times 1\times 1}$. Intermediate temporal
feature maps $S^{\prime}$ are refined with attention maps in the following
way:

$S^{\prime\prime}=M_{c}(S^{\prime})\otimes S^{\prime}\quad\quad\quad
S^{\prime\prime\prime}=M_{s}(S^{\prime\prime})\otimes
S^{\prime\prime}\quad\quad\quad\boldsymbol{O^{\prime}}=M_{t}(S^{\prime\prime\prime})\otimes
S^{\prime\prime\prime}$ (1)

Here $\boldsymbol{O^{\prime}}$ denotes the output of the module and $\otimes$
is an element-wise multiplication during which attention maps are broadcasted
along all unitary dimensions.

Figure 2: Details of TabAttention components: CAM (a), SAM (b) and TAM (c)
with tabular embeddings. In TAM, only one attention head is visualised.
TabAttention is integrated with residual block as presented in (d). $\otimes$
indicates matrix multiplication.

In general, attention maps are computed based on information aggregated by
average- and max-pooling along specified dimensions which are then passed
through shared layers for refinement (Fig. 2). Then, these refined descriptors
are passed through the sigmoid function to create final attention maps. To
account for the tabular information during attention maps computing, we embed
the input tabular data $Tab\in\mathbb{R}^{D}$, where $D$ is the number of
numerical features, with two linear layers and Rectified Linear Unit (ReLU)
activation in between. The tabular data is embedded to the size of pooled
feature maps. The embedding is passed through shared layers in the same way as
pooled feature maps. Therefore, the attention maps are computed conditionally
on tabular data. Thus, the output of TabAttention
$\boldsymbol{O^{\prime}_{t}}$ is computed as follows:

$S^{\prime\prime}_{t}=M_{c}(S^{\prime},Tab)\otimes S^{\prime}\quad\quad
S^{\prime\prime\prime}_{t}=M_{s}(S^{\prime\prime}_{t},Tab)\otimes
S^{\prime\prime}_{t}\quad\quad\boldsymbol{O^{\prime}_{t}}=M_{t}(S^{\prime\prime\prime}_{t},Tab)\otimes
S^{\prime\prime\prime}_{t}$ (2)

Channel Attention Module. We follow the design of the original CBAM [25]. We
split temporal feature maps into $T$ feature maps $F_{i}$ where $i\in 1,...,T$
so that each of them is passed through CAM separately. To compute the channel
attention ($M_{c}$), we aggregate the spatial information through average- and
max-pooling to produce descriptors ($F^{c}_{avg_{i}}$,
$F^{c}_{max_{i}}\in\mathbb{R}^{C\times 1\times 1}$). We pass the tabular data
through a multi-layer perceptron ($MLP_{emb_{c}}$) with one hidden layer (of
size $\mathbb{R}^{\frac{C}{z}}$, where $z$ is the reduction ratio set to 16)
and ReLU activation to embed it into the same dimension as spatial
descriptors. Then, both descriptors, with tabular embedding are passed through
the shared network which is MLP with a hidden activation size of
$\mathbb{R}^{\frac{C}{z}}$ and one ReLU activation. After the MLP is applied,
the output vectors are element-wise summed to produce the attention map. We
concatenate attention maps of all feature maps to produce $M_{c}$:

$\displaystyle M_{c}(S^{\prime},Tab)=[M^{f}_{c}(F_{i},Tab)]_{i=1,...,T}$ (3)
$\displaystyle
M^{f}_{c}(F_{i},Tab)=\sigma(MLP(F^{c}_{max_{i}})+MLP(F^{c}_{avg_{i}})+MLP(MLP_{emb_{c}}(Tab)))$
(4)

Spatial Attention Module. After splitting the temporal feature maps, we
average- and max-pool them along channel dimension to produce feature
descriptors ($F^{s}_{avg_{i}}$, $F^{s}_{max_{i}}\in\mathbb{R}^{1\times H\times
W}$). We pass the tabular data through $MLP_{emb_{s}}$ with one hidden layer
of size $\mathbb{R}^{\frac{H\times W}{2}}$ and ReLU activation to embed it
into the same dimension as spatial descriptors. We reshape this embedding to
the size of feature descriptors and concatenate it with them. We pass the
following representation through a 2D convolution layer and the sigmoid
activation:

$\displaystyle M_{s}(S^{\prime\prime},Tab)=[M^{f}_{s}(F_{i},Tab)]_{i=1,...,T}$
(5) $\displaystyle
M^{f}_{s}(F_{i},Tab)=\sigma(Conv([F^{s}_{max_{i}},F^{s}_{avg_{i}},Reshape(MLP_{emb_{s}}(Tab))]))$
(6)

Temporal Attention Module. We create temporal descriptors by average- and max-
pooling temporal feature maps along all non-temporal dimensions
($F^{t}_{avg_{i}}$, $F^{t}_{max_{i}}\in\mathbb{R}^{T\times 1\times 1\times
1}$). We embed tabular data with $MLP_{emb_{t}}$ with one hidden layer of size
$\mathbb{R}^{\frac{T}{2}}$ into the same dimension. We concatenate created
vectors and treat them as the embedding of the US sequence which we pass to
the MHSA layer (with 2 heads). We create the query (Q), key (K) and value (V)
with linear layers and an output size of $d$ (4). We add relative positional
encodings [19] $r$ to K. After passing through MHSA, we squash the refined
representation with one MLP layer and sigmoid function to create a temporal
attention map $M_{t}$:

$\displaystyle
MHSA(S_{emb})=MLP\left(\left[softmax\left(\frac{Q_{j}(K_{j}+r)^{T}}{\sqrt{d}}\right)V_{j}\right]_{j=1,2}\right)$
(7) $\displaystyle
M_{t}(S^{\prime\prime\prime},Tab)=\sigma(MHSA([F^{t}_{max},F^{t}_{avg},MLP_{emb_{t}}(Tab)]))$
(8)

TabAttention can be integrated within any 3D CNN (or 2D CNN in case TAM is
omitted). As illustrated in Fig. 2, we add TabAttention between the first ReLU
and the second convolution in the residual block to integrate our module with
3D ResNet-18.

##  3 Experiments and Results

This section describes the dataset used and provides implementation details of
our proposed method. We benchmark the performance of TabAttention against
several state-of-the-art methods and compare them to results obtained by
clinicians. Additionally, we conduct an ablation study to demonstrate the
significance of each key component utilized in our approach.

Figure 3: Exemplary input to our method (a) of tabular data and frames of
abdominal scans. Fetal US scans of the head (b), abdomen (c), and femur (d)
were used to obtain AC, HC, BPD and FL utilized in tabular data.

Dataset. This study was approved by the Ethics Committee of the Medical
University of Warsaw (Reference KB.195/2021) and informed consent was obtained
for all subjects. The multi-site dataset was acquired using international
standards approved by [18]. The dataset consists of 92 2D fetal US video scans
captured in the standard abdominal plane view. These scans were collected from
92 pregnant women (31.89 $\pm$ 4.76 years), across three medical centers, and
obtained as part of routine US examination done less than 24 hours before
delivery. This allowed us to obtain the real ground truth which was baby
weight soon after birth. Five experienced sonographers (14.2 $\pm$ 4.02 years
of experience) acquired the data using a single manufacturer device (General
Electric) and several models (GE Voluson E6, S8, P8, E10, and S10). The
abdominal fetal US videos (5-10 seconds, 13-37 frames per second) were saved
in the DICOM file format. We resized the pixel spacing to 0.2 mm $\times$ 0.2
mm for all video clips. As tabular data, we used six numerical features: AC
(34.51 $\pm$ 2.35 cm), HC (33.56 $\pm$ 1.41 cm), BPD (9.40 $\pm$ 0.46 cm), FL
(7.28 $\pm$ 0.33 cm), GA (38.29 $\pm$ 1.47 weeks), and mother’s age. The
examples of how the measurements were obtained are presented in Fig. 3. The
actual birth weight of the fetus obtained right post-delivery (3495 $\pm$ 507
grams) was used as the target of the prediction.

Implementation details. We use 3D ResNet-18 as our base model. We implement
all experiments with PyTorch and train networks using NVIDIA A100 80GB GPU for
250 epochs with a batch size of 16 and an initial learning rate chosen with
grid search from the set of $\\{1\times 10^{-2},1\times 10^{-3},1\times
10^{-4}\\}$. To minimize the Mean Squared Error loss function, we employ the
Adam [10] optimizer with L2 regularization of $1\times 10^{-4}$ and cosine
annealing learning rate scheduler. To evaluate the reliability of the
regression algorithm, we conduct five-fold cross-validation (CV) and ensure
that each patient’s data is present in only one fold. To ensure similar birth
weight distribution in all folds, we stratify them based on the assignment of
data samples into three bins: $<$ 3000 g, $>$ 4000 g, and in-between. The
input frames are of size 128 $\times$ 128 pixels. We follow the approach
presented in [14], we set the number of input frames to 16 and average per-
patient predictions of all 16 frame segments from the single video. Throughout
the training process, we employ various data augmentation techniques such as
rotation, random adjustments to brightness and contrast, the addition of
Gaussian noise, horizontal flipping, image compression, and motion blurring
for every batch. We standardize all numerical features to a mean of 0 and a
standard deviation of 1. We use Root Mean Square Error (RMSE), Mean Absolute
Error (MAE), and MAPE to evaluate the regression performance.

Comparison with state-of-the-art methods. We compare TabAttention with several
methods utilizing tabular data only (Linear Regression [13], XGBoost [4]),
imaging data only (3D ResNet-18 [22], BabyNet [14]), both types of data
(Interactive [5], DAFT [16]), and Clinicians. The predictions of Clinicians
were achieved using Hadlock III [7] formula and AC, HC, BPD, FL measurements.
The comparison of results from the five-fold CV is presented in Table 1.
TabAttention achieves the lowest MAE, RMSE and MAPE (170 $\pm$ 26, 225 $\pm$
37, 5.0 $\pm$ 0.8 respectively) among all tested methods. Our approach
outperforms clinically utilized heuristic formulae, machine learning, and
image-only DL methods (two-tailed paired t-test p-value $<$ 0.05). Results of
TabAttention are also best compared with all DL models utilizing tabular and
imaging modalities, however, the difference does not reach statistical
significance with a p-value around 0.11.

Table 1: Five-fold cross-validation results of state-of-the-art methods utilizing imaging (Img.) and/or tabular (Tab.) data. The mean of MAE, RMSE, and MAPE across all folds are presented. The best results are bolded. Method | Img. | Tab. | mMAE [g] | mRMSE [g] | mMAPE [%]
---|---|---|---|---|---
BabyNet [14] | ✓ | ✗ | $294\pm 30$ | $386\pm 56$ | $8.5\pm 1.0$
3D ResNet-18 [22] | ✓ | ✗ | $289\pm 38$ | $373\pm 43$ | $8.5\pm 1.1$
XGBoost | ✗ | ✓ | $259\pm 23$ | $328\pm 26$ | $7.6\pm 0.1$
Linear Regression | ✗ | ✓ | $207\pm 18$ | $260\pm 19$ | $6.0\pm 0.1$
Clinicians | ✗ | ✓ | $205\pm 14$ | $253\pm 13$ | $5.9\pm 0.0$
DAFT [16] | ✓ | ✓ | $175\pm 30$ | $244\pm 42$ | $5.3\pm 1.0$
Interactive [5] | ✓ | ✓ | $172\pm 27$ | $230\pm 44$ | $5.2\pm 0.9$
TabAttention (ours) | ✓ | ✓ | $\mathbf{170\pm 26}$ | $\mathbf{225\pm 37}$ | $\mathbf{5.0\pm 0.8}$

Ablation study. We conduct ablation experiments to validate the effectiveness
of key components of our proposed method (Table 2). We employ 3D ResNet-18 as
the baseline model. The integration of TAM or CBAM with attention maps learned
conditionally on tabular data into the 3D ResNet-18 architecture improves the
predictive performance of the network. Subsequently, the incorporation of full
TabAttention further enhances its capabilities.

Table 2: Five-fold cross-validation results of ablation study with key components of TabAttention. The first row is the result of the baseline method. The next rows refer to modules of TabAttention with or without tabular embeddings (Tab.) and the last one is full TabAttention. Method | Img. | Tab. | mMAE [g] | mRMSE [g] | mMAPE [%]
---|---|---|---|---|---
3D ResNet-18 [22] | ✓ | ✗ | $289\pm 38$ | $373\pm 43$ | $8.5\pm 1.1$
\+ TAM | ✓ | ✗ | $288\pm 43$ | $389\pm 65$ | $8.4\pm 1.2$
\+ CBAM + Tab. | ✓ | ✓ | $271\pm 51$ | $371\pm 99$ | $7.7\pm 1.3$
\+ TAM + Tab. | ✓ | ✓ | $180\pm 32$ | $237\pm 45$ | $5.5\pm 0.1$
\+ TabAttention (ours) | ✓ | ✓ | $\mathbf{170\pm 26}$ | $\mathbf{225\pm 37}$ | $\mathbf{5.0\pm 0.8}$

##  4 Discussion and Conclusions

In this work, we present a novel method, TabAttention, that can effectively
compete with current state-of-the-art image and/or tabular-based approaches in
estimating FBW. We found that it outperformed Clinicians achieving mMAPE of
5.0% vs. 5.9% (p-value $<$ 0.05). A key advantage of our approach is that it
does not require any additional effort from clinicians since the necessary
data is already collected as part of standard procedures. This makes
TabAttention an alternative to the heuristic formulas that are currently used
in clinical practice. We should note that while TabAttention achieved the
lowest metrics among the DL models we evaluated, the differences between our
approach and other DL methods using tabular data were not statistically
significant, partly due to the small performance change. This small difference
in the performance is likely caused by the fact that the tabular features used
in TabAttention are mainly derived from the same modality (i.e. US scans), so
they do not carry additional information, but instead can be considered as
refined features already present in the scans. To develop TabAttention, we
used tabular data as a hint for the network to learn attention maps and gain
additional knowledge about essential aspects presented in the scans. This
approach significantly improved the performance of baseline methods and
demonstrated its practical applicability.

Accurate estimation of FBW is crucial in determining the appropriate delivery
method, whether vaginal or Cesarean. Low birth weight (less than 2500 g) is a
major risk factor for neonatal death, while macrosomia (greater than 4000 g)
can lead to delivery traumas and maternal complications, such as birth canal
injuries, as reported by Benacerraf et al. [2]. Thus, precise prediction of
FBW is vital for very low and high weights. Notably, in this respect, our
method is robust to outliers with high or low FBW since there is no
correlation between true FBW and absolute prediction error (Pearson
correlation coefficient of -0.029).

This study has limitations. Firstly, a relatively small study cohort was used,
which may affect the accuracy and generalization of the results. To address
this, future work will include a larger sample size by using additional
datasets. Secondly, our dataset is limited to only Caucasian women and may not
be representative of other ethnicities. It is important to investigate the
performance of our method with datasets from different ethnic groups and US
devices to obtain more robust and generalizable results. Lastly, our method
relies on fetal biometry measurements that are subject to inter- and intra-
observer variabilities. This variability could potentially affect the
network’s performance and influence the measurements’ quality. Future studies
should consider strategies to reduce measurement variabilities, such as
standardized protocols or automated measurements, to improve the accuracy of
the method.

To summarize, we have introduced TabAttention, a new module that enables the
conditional learning of attention on tabular data and can be integrated with
any CNN. Our method has many potential applications, including serving as a
computer-aided diagnosis tool for various clinical workflows. We have
demonstrated the effectiveness of TabAttention on the FBW prediction task,
utilizing both US and tabular data, and have shown that it outperforms other
methods, including clinically used ones. In the future, we plan to test the
method in different clinical applications where imaging and tabular data are
used together.

## Acknowledgements

This work is supported by the European Union’s Horizon 2020 research and
innovation programme under grant agreement Sano No 857533 and the
International Research Agendas programme of the Foundation for Polish Science,
co-financed by the European Union under the European Regional Development
Fund.

## References

  * [1] Bano, S., Dromey, B., Vasconcelos, F., Napolitano, R., David, A.L., Peebles, D.M., Stoyanov, D.: Autofb: Automating fetal biometry estimation from standard ultrasound planes. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 228–238. Springer (2021)
  * [2] Benacerraf, B.R., Gelman, R., Frigoletto Jr, F.D.: Sonographically estimated fetal weights: accuracy and limitation. American journal of obstetrics and gynecology 159(5), 1118–1121 (1988)
  * [3] Campbell, S., Wilkin, D.: Ultrasonic measurement of fetal abdomen circumference in the estimation of fetal weight. BJOG: An International Journal of Obstetrics & Gynaecology 82(9), 689–697 (1975)
  * [4] Chen, T., Guestrin, C.: XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 785–794. KDD ’16, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2939672.2939785
  * [5] Duanmu, H., Huang, P.B., Brahmavar, S., Lin, S., Ren, T., Kong, J., Wang, F., Duong, T.Q.: Prediction of pathological complete response to neoadjuvant chemotherapy in breast cancer using deep learning with integrative imaging, molecular and demographic data. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part II 23. pp. 242–252. Springer (2020)
  * [6] Guan, Y., Cui, H., Xu, Y., Jin, Q., Feng, T., Tu, H., Xuan, P., Li, W., Wang, L., Duh, B.L.: Predicting esophageal fistula risks using a multimodal self-attention network. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24. pp. 721–730. Springer (2021)
  * [7] Hadlock, F.P., Harrist, R., Sharman, R.S., Deter, R.L., Park, S.K.: Estimation of fetal weight with the use of head, body, and femur measurements—a prospective study. American journal of obstetrics and gynecology 151(3), 333–337 (1985)
  * [8] Holste, G., Partridge, S.C., Rahbar, H., Biswas, D., Lee, C.I., Alessio, A.M.: End-to-end learning of fused image and non-image features for improved breast cancer classification from mri. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3294–3303 (2021)
  * [9] Huang, S.C., Pareek, A., Seyyedi, S., Banerjee, I., Lungren, M.P.: Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. NPJ digital medicine 3(1), 136 (2020)
  * [10] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
  * [11] Liu, M., Zhang, J., Adeli, E., Shen, D.: Joint classification and regression via deep multi-task multi-channel learning for alzheimer’s disease diagnosis. IEEE Transactions on Biomedical Engineering 66(5), 1195–1206 (2018)
  * [12] Lu, Y., Zhang, X., Fu, X., Chen, F., Wong, K.K.: Ensemble machine learning for estimating fetal weight at varying gestational age. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 9522–9527 (2019)
  * [13] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)
  * [14] Płotka, S., Grzeszczyk, M.K., Brawura-Biskupski-Samaha, R., Gutaj, P., Lipa, M., Trzciński, T., Sitek, A.: Babynet: Residual transformer module for birth weight prediction on fetal ultrasound video. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part IV. pp. 350–359. Springer (2022)
  * [15] Płotka, S., Klasa, A., Lisowska, A., Seliga-Siwecka, J., Lipa, M., Trzciński, T., Sitek, A.: Deep learning fetal ultrasound video model match human observers in biometric measurements. Physics in Medicine & Biology 67(4), 045013 (2022)
  * [16] Pölsterl, S., Wolf, T.N., Wachinger, C.: Combining 3d image and tabular data via the dynamic affine feature map transform. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24. pp. 688–698. Springer (2021)
  * [17] Pressman, E.K., Bienstock, J.L., Blakemore, K.J., Martin, S.A., Callan, N.A.: Prediction of birth weight by ultrasound in the third trimester. Obstetrics & Gynecology 95(4), 502–506 (2000)
  * [18] Salomon, L., Alfirevic, Z., Da Silva Costa, F., Deter, R., Figueras, F., Ghi, T.a., Glanc, P., Khalil, A., Lee, W., Napolitano, R., et al.: Isuog practice guidelines: ultrasound assessment of fetal biometry and growth. Ultrasound in obstetrics & gynecology 53(6), 715–723 (2019)
  * [19] Shaw, P., Uszkoreit, J., Vaswani, A.: Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 (2018)
  * [20] Sherman, D.J., Arieli, S., Tovbin, J., Siegel, G., Caspi, E., Bukovsky, I.: A comparison of clinical and ultrasonic estimation of fetal weight. Obstetrics & Gynecology 91(2), 212–217 (1998)
  * [21] Tao, J., Yuan, Z., Sun, L., Yu, K., Zhang, Z.: Fetal birthweight prediction with measured data by a temporal machine learning method. BMC Medical Informatics and Decision Making 21(1), 1–10 (2021)
  * [22] Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6450–6459 (2018)
  * [23] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [24] Wang, X., Liu, D., Zhang, Y., Li, Y., Wu, S.: A spatiotemporal multi-stream learning framework based on attention mechanism for automatic modulation recognition. Digital Signal Processing 130, 103703 (2022)
  * [25] Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
  * [26] Yadav, S., Rai, A.: Frequency and temporal convolutional attention for text-independent speaker recognition. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6794–6798. IEEE (2020)