(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version
11institutetext: University of Bristol, UK
https://sid2697.github.io/hoi-ref

# HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision

Siddhant Bansal Michael Wray Dima Damen

###### Abstract

Large Vision Language Models (VLMs) are now the de facto state-of-the-art for
a number of tasks including visual question answering, recognising objects,
and spatial referral. In this work, we propose the HOI-Ref task for egocentric
images that aims to understand interactions between hands and objects using
VLMs. To enable HOI-Ref, we curate the HOI-QA dataset that consists of $3.9$M
question-answer pairs for training and evaluating VLMs. HOI-QA includes
questions relating to locating hands, objects, and critically their
interactions (_e.g_. referring to the object being manipulated by the hand).
We train the first VLM for HOI-Ref on this dataset and call it VLM4HOI. Our
results demonstrate that VLMs trained for referral on third person images fail
to recognise and refer hands and objects in egocentric images. When fine-tuned
on our egocentric HOI-QA dataset, performance improves by $27.9\%$ for
referring hands and objects, and by $26.7\%$ for referring interactions.

###### Keywords:

Vision Language Models Egocentric Vision Referral Hand Object Interactions

##  1 Introduction

Understanding hand-object interactions from images [35, 36] and videos [11,
46], including egocentric videos [18, 21, 44], offers a fine-grained
perception beyond recognising objects in isolation. The capability of
referring hands as well as objects they are interacting with opens up a range
of possibilities for robotics and augmented-reality (AR) applications. For
example, while cooking, current high-end AR glasses are only capable of adding
timers over the cookware [25]. To make these devices truly assistive, we need
frameworks that are capable of understanding hand-object interactions from an
egocentric perspective.

Motivated by the need to assess and improve hand-object interaction
understanding, in this work, we introduce HOI-Ref, Hand-Object Interaction
Referral task for egocentric images. Referral111The terms referral, grounding,
and localisation have been used interchangeably in prior work. In this paper,
we consistently use the term referral for this task. is the task of locating
an object in an image—typically using a bounding box, or to name an object
given its bounding box [9, 8, 38]. As shown in Fig. 1, given an image from an
egocentric video, the goal is to: (a) refer the hands and objects being
interacted with and (b) understand the interaction between them. For example,
the left hand is holding the jar and the right hand is holding the lid. We
solve this by creating a unified model capable of understanding spatial
coordinates, hands, objects, and hand-object interactions. To this end, we
explore Vision Language Models (VLMs) for hand-object interaction referral in
egocentric images.

Thanks to the availability of open-source Large Language Models (LLMs) that
are trained on huge amounts of textual data [12, 58, 57], training VLMs has
recently flourished. Training regimes (such as fine-tuning a projection layer
[32]) have enabled LLMs to generalise well to a range of input queries for
solving visual tasks [3, 15, 32, 2]. A few works have looked into referential
dialogue [9] and object referral [8], primarily using large-scale datasets
[38, 32]. We build upon these works that utilise LLMs for referring to various
objects present in the scene and modify the prompt to perform multiple tasks.
Therefore, we propose our VLM4HOI model, which enables HOI-Ref on egocentric
images using VLMs.

Figure 1: Hand-Object Interaction Referral. Given an image from an egocentric
video, the goal here is to refer the hands and the objects being interacted
with. For example, here we wish to refer the left and right hand along with
the two objects (jar and lid) that the hands are interacting with.

To train and evaluate VLM4HOI, we curate a new egocentric dataset (HOI-QA) of
3.9M question-answer pairs.

We construct HOI-QA by creating a general pipeline that converts available
annotations from egocentric datasets to referral question-answer pairs for
both training and evaluation. For HOI-QA, we utilise the annotations from [17,
21, 18], where masks, bounding boxes or hand-object interactions have been
annotated. To the best of our knowledge, HOI-QA is the largest effort to
enable hand-object interaction referral for egocentric vision.

Finally, we evaluate VLM4HOI on two aspects of HOI-Ref, its ability to refer
hands and objects and its capability to understand the interaction between the
hand and the interacted object. To summarise, our contributions are:

  1. 1.

We propose the HOI-Ref task for Egocentric Vision to enable hand-object
interaction referral;

  2. 2.

We collect the HOI-QA dataset consisting of $3.9$M question-answer pairs to
train and evaluate models for the HOI-Ref task;

  3. 3.

We train VLM4HOI on HOI-QA as the first VLM for HOI-Ref, showcasing its
ability to better understand hand-object interactions compared to baselines.

Our HOI-QA dataset, VLM4HOI model, and code are released on project’s webpage
to encourage research on HOI-Ref task for egocentric vision. Next, we review
recent literature in Sec. 2; before presenting details of our model VLM4HOI
and dataset collection details of HOI-QA in Sec. 3; evaluate and discuss our
findings in Sec. 4; and provide conclusions and future work in Sec. 5.

##  2 Related Works

We first review Large Language Models, before introducing how they have been
combined into Vision Tasks. Next, we present how Vision Language Models have
tackled referral before reviewing their usage for Egocentric Vision.

##### Large Language Models (LLMs)

have seen a huge surge of popularity [6, 13, 43, 59] due to their generative,
reasoning, and general knowledge abilities. These methods have been expanded
via instruction-tuning [37, 53, 52], creating instruction-tuned versions of
previous LLMs [1, 12, 37, 57], allowing for better generalisation for
downstream and zero-shot tasks including summarisation, text generation, or
question answering.

##### Vision Language Models (VLMs)

have tackled a variety of downstream tasks such as retrieval, captioning, and
visual question answering. Current VLMs combine LLMs with powerful vision-
language representations [3, 7, 15, 19, 30, 33, 47, 56, 61, 54, 55, 58] via
prompt-engineering [7], prompt-tuning [30, 33, 54, 55], and instruction tuning
[3, 15, 32, 56, 58, 61].

Commonly, models combine a VIT encoder trained from CLIP [7, 42] or BLIP [29]
with an LLM such as Vicuna [12] or LLAMA [58]. These models are later
instruction-tuned using multi-modal data to further improve their
effectiveness for downstream tasks. We similarly use instruction tuning to
improve the performance of our model. However, instead of using an LLM to
create the instructions [61], we use rich information such as bounding boxes,
dense narrations, and object interactions to generate high quality
instructions for the HOI-QA dataset.

##### Referral in VLMs

goes beyond captioning and question answering to referring tasks [4, 8, 9, 32,
34, 38, 22]. These models similarly combine vision encoders with LLMs using
LoRA [24] or a projection layer to reduce computation required for training.
Instructions are augmented to include bounding boxes to refer answers within
the image or name an object given a bounding box. Going beyond including tags
for bounding boxes in the prompt, MiniGPT-v2 [8] proposes using unique
identifiers for question types which helps disambiguate the tasks and improves
performance on downstream tasks.

We similarly generate instructions in the form of question-answer pairs,
focusing on the unique hand-object interactions present within egocentric
images. We adopt the question identifiers proposed in [8], to disambiguate
task types for the HOI-QA dataset. Moreover, we use annotations within
egocentric datasets [17, 16, 21] to design instructions so that VLM4HOI can
learn to refer not only objects but also interactions in egocentric images.

##### VLMs for egocentric vision

have at their onset [10, 14, 40, 41, 60] focused on video-text retrieval [17],
natural language queries [21], and zero-shot action recognition tasks [17, 27,
31, 49, 50]. These models are trained on the massive-scale Ego4D dataset [21]
using crowd-sourced narrations to learn a video-text embedding. EgoVLP [41]
proposed a multi-choice question benchmark set to verify the performance of
the VLM during training. Follow up works have improved performance via the use
of LLMs to narrate the videos as an additional supervision [60] and a cross-
modal fusion layer within the backbone [40].

More recently, models have been trained for downstream tasks including object
tracking via referring expressions [28] and task verification [23]. RefEgo
[28], proposes a new dataset for referring object expression which requires
models to refer an object from a description across frames of a video along
with a new model, solely focused on this task. Their model is based upon
M-DETR [26] modified to cope with objects disappearing from view. We instead
target hand-object interaction referral in frames, going beyond finding
objects. Additionally, our benchmark, and subsequent model, allow for bounding
boxes to be used as input as opposed to solely as output.

Up to our knowledge, we are the first to introduce the HOI-Ref task, propose
the VLM4HOI model to tackle it along with the HOI-QA dataset for training and
evaluation.

##  3 Method: Hand-Object Interaction Referral

Our objective is to enable Vision Language Models (VLMs) to refer hands,
objects, and their interaction—the ability to refer the object in contact with
the hand, or the hand in contact with the object. To this end, our work builds
on VLMs that take in visual input, in the form of an image along with a text
query, and can then generate the appropriate response to refer objects and
interactions.

Figure 2: (a) VLM4HOI for hand-object interaction referral in egocentric
images. The VLM4HOI model takes in an image ($I$), passes it through a vision
encoder ($g$) and a projection layer ($W_{\phi}$) to obtain embeddings
($E_{p}$) in language model’s ($f_{\theta}$) embedding space. This is
concatenated with the tokenised text ($E_{L}$) and passed through $f_{\theta}$
to generate a language response ($E_{a}$). We show two examples where based on
the task instruction template, the model generates an output. (b), the model
identifies a bounding box input as the right hand. (c), the model takes in the
image and a question to refer the object being held in the right hand and
outputs a bounding box.

###  3.1 VLM4HOI: VLM for Referral

As in prior works [8, 9, 32], we rely on the abilities of a pre-trained LLM,
and integrate the visual tokens into the embedding space of the language
model. More specifically, we encode an image $I$ using a visual encoder $g$, a
linear projection layer $W_{\phi}$ and a large language model $f_{\theta}$. We
explore two types of instructions, one where the instructions include a
bounding box to be described, and another where the bounding box is generated
in the response.

In detail, we pass the image $I$ through a frozen vision encoder $g$ to obtain
patch embeddings $g(I)$. We use an additional trainable projection layer
$W_{\phi}$ that projects $g(I)$ to a language model’s $f_{\theta}$’s embedding
space:

$E_{p}=W_{\phi}\cdot g(I)$ (1)

Once we have the visual tokens ($E_{p}$) in $f_{\theta}$’s embedding space,
the instruction ($T$) is tokenised ($E_{L}$) and passed along with $E_{p}$
through $f_{\theta}(\cdot)$ to generate a response $E_{a}$:

$E_{a}=f_{\theta}(E_{p}\oplus E_{L})$ (2)

where $\oplus$ represents concatenation, and $E_{a}$ is further decoded using
the tokeniser to obtain the language response.

To train the model using language instructions, following [9, 8], we encode
the bounding box using the format:
${\\{<X_{\text{left}}><Y_{\text{top}}><X_{\text{right}}><Y_{\text{bottom}}>\\}}$,
for example, {<68><22><90><96>} in Fig. 2. This enables the model to perform
referral without utilising a specialised template. As shown in Fig 2, we fine-
tune the parameters $\theta$ and $\phi$ on a large number of referral
examples.

Having presented our model, we now introduce the training data for the tasks
of direct referral—that is the referral of hands and/or objects as well as
interaction referral—the referral of an object in contact with the hand, and
the hand in contact with the object. Once trained, our model performs referral
for hands, objects, and interactions. We thus refer to our model as VLM4HOI.

###  3.2 Framework to Generate Question-Answer Pairs

To enable hand-object referral and understand the interaction between hands
and objects using VLMs, question-answer pairs with hand-object referral
information are crucial. In this section, we propose a framework that
identifies publicly available annotations that can be converted to question-
answer pairs for enabling hand-object interaction referral.

####  3.2.1 Annotation Types.

We use commonly provided annotations in egocentric videos including:
Narrations to encode the general activity happening in the egocentric image;
Object Detection GT bounding boxes with corresponding semantics, in the form
of closed vocabulary (classes) or open vocabulary (captions). We use these
detections to enhance the referral capability of VLMs and to understand hand-
object interaction; Hand Detection GT bounding boxes with corresponding hand
sides (right/left) enable referral and distinction between the interactions of
the two hand sides; Hand-object contact GT which highlight whether the hand-
side is in contact with an object, and if so which object. We utilise these
annotations for interaction referral specifically. For example this enables
asking questions similar to, what is in the left hand of the person? Answering
such a question not only requires the model to correctly refer the hand, but
also identify the object in that hand.

Any egocentric dataset consisting of one or more of the annotation types
mentioned above can be utilised for generating question-answer pairs to train
VLMs for hand-object interaction understanding. Fig. 3 shows an example of
each annotation.

####  3.2.2 Question-Answer Pairs.

Here we discuss how the various annotation types are converted to question-
answer pairs. Note that {object/hand bounding box} in the following discussion
corresponds to the bounding box representation shown in Fig. 3 and discussed
in Sec. 3.1.

  1. •

Narrations provide us with verb and noun pairs describing the action happening
in the egocentric image. We convert the verb form to its present tense and
formulate the sentence as, “The person {verb} the {noun}”. This sentence is
then used as a description of the event within image. For example, Question:
This is a photo from an ongoing video of hand object interactions. What is
happening in the photo? Answer: The person turns on the tap. For a visual
example, refer to Question $2$ in Fig. 3. Such QA pairs allow the VLM to have
a global understanding of the ongoing action in the image.

  2. •

Object/Hand Detection Annotations. We use the bounding boxes of hands and
objects to create the following QA pairs:  (a) Question: Where is the
{object/hand}? Answer: {Object’s/hand’s bounding box};  (b) Question:
{Object’s/Hand’s bounding box}. Answer: {Object’s/Hand’s name};  (c) Question:
{Object’s name}. Answer:
{Object’s name}
 {Object’s bounding box};  (d)
Question: This is a photo from an ongoing video of hand object interactions.
Describe the ongoing action in this image. Answer: The person opens the

tap
 {tap’s bounding box}.  For example, Questions $1$, $6$, $7$, and
$8$ in Fig. 3. Such QA pairs open up the possibility to train VLMs for
recognising hands and objects and referring them spatially in the image.

  3. •

Hand Object Contact information. We combine bounding box annotations with the
knowledge about what object the hand is holding to create the following QA
pairs:  (a) Question: What is the person holding in their {left/right} hand?
Answer: {Object’s name};  (b) Question: Describe the ongoing action in this
image in detail. Answer: The person is working in a kitchen. They are holding
a
{object’s name}
 {object’s bounding box} in their
left hand

{left hand’s bounding box} and a
{object’s name}
 {object’s bounding
box} in their
right hand
 {right hand’s bounding box}. (c) Question: Is
the {left/right} hand holding or touching an object? Answer: The object in the
{left/right} hand is a {object’s name} OR {Left/Right} hand is not holding
anything. (d) Question: Where is the object being manipulated by the hand?
Answer: {object’s bounding box in left/right hand}.  For example, Questions
$3$, $4$, and $5$ in Fig. 3. Such QA pairs not only train VLMs to spatially
refer the hands and objects, but also enable them to understand what hand side
is manipulating which object.

Figure 3: Question-Answer pairs generation for training VLMs to understand
hand-object interaction. We use multiple annotation types to create the
question-answer pairs. Top shows the annotations utilised and Bottom shows the
types of question-answer pairs generated from these annotations. As shown, we
convert the segments to bounding boxes to generate various referral questions
and utilise contact information to understand interaction between hands and
objects. Right shows the distribution of questions in the proposed HOI-QA
dataset (Sec. 3.4).

###  3.3 HOI-Ref: Hand-Object Interaction Referral

In this section, we dive deeper into the formulation of the Hand-Object
Interaction Referral (HOI-Ref) task for egocentric images. HOI-Ref aims to
create a framework to train and evaluate VLMs on two aspects of hand-object
interaction referral:  (a) ability to spatially refer and recognise hands and
objects and  (b) capability to understand the interaction between hand and
object.  To achieve this, the question-answer pairs created in Sec. 3.2 are
strategically divided into two parts (Fig. 4):  (A) HO-Ref: Hand and Object
Referral, and  (B) I-Ref: Hand-Object Interaction Referral.

Figure 4: HOI-Ref task to train and evaluate VLMs for hand-object interaction
referral. HOI-Ref focuses on the following two aspects: ability to spatially
refer and recognise hands and objects and the capability to understand hand-
object interaction. Columns (1) and (2) evaluate spatially referring hands and
objects whereas, columns (3) and (4) aim at object and hand side recognition.
Moving across rows (A) and (B) shows HOI-Ref’s ability to evaluate for direct
referral vs interaction referral. For example, in A-1, referring a bottle is
simply asking where is the bottle however, for B-1, it involves knowing which
hand is holding the bottle.

####  3.3.1 HO-Ref: Hand and Object Referral.

The questions in HO-Ref are further divided into four general categories
depending on the input $\rightarrow$ output types:

  1. 1.

Object’s Name $\rightarrow$ Object’s bounding box. Given the object’s name,
the goal is to predict its spatial coordinates. The question in this category
is: Where is the {object’s name}? For example, question A-1 in Fig. 4.

  2. 2.

Hand side $\rightarrow$ Hand’s bounding box. Given the hand’s side, the goal
is to predict its spatial coordinates. The questions in this category are:
Where is the {left/right} hand of the person? Or Where are the hands of the
person? For example, question A-2 in Fig. 4.

  3. 3.

Object’s bounding box $\rightarrow$ Object’s Name. Given the object’s bounding
box, the goal is to predict its name. The question in this category is:
{object’s bounding box}. For example, question A-3 in Fig. 4.

  4. 4.

Hand’s bounding box $\rightarrow$ Hand side. Given the hand’s bounding box,
the goal is to predict the hand’s side. The question in this category is:
{left/right hand’s bounding box}. For example, question A-4 in Fig. 4.

####  3.3.2 I-Ref: Hand-Object Interaction Referral.

Similar to above, we have four corresponding categories. However, answering
questions for I-Ref requires not only identifying the object/hand but the
interaction between this and another hand/object, hence the term Interaction
Referral. The four categories are:

  1. 1.

Hand’s name $\rightarrow$ Object’s bounding box. Given the hand side’s name,
the goal is to predict spatial coordinates of the object in that hand. The
question in this category is: Locate the object being manipulated by
{right/left} hand. For example, question B-1 in Fig. 4.

  2. 2.

Object’s Name $\rightarrow$ Hand’s bounding box. Given an object’s name, the
goal is to predict spatial coordinates of the hand holding the object. The
question in this category is: Which hand has the {object’s name}? For example,
question B-2 in Fig. 4.

  3. 3.

Hand’s name $\rightarrow$ Object’s name. Opposing the previous category, given
a hand side, the goal is to generate the object’s name in that hand. The
question in this category is: What is the object in {left/right} hand? For
example, question B-3 in Fig. 4.

  4. 4.

Object’s name $\rightarrow$ Hand’s name. Given an object’s name, the goal is
to predict hand side that is holding the object. The questions in this
category are: What hand is holding the {object’s name}? Or What {object} is in
the hands of the person? For example, question B-4 in Fig. 4.

Task Tags Until now the question-answer pairs generated do not have a marker
on them that guides the VLM on what to expect in terms of the style of
input/output. For example, it might be challenging for a VLM to know what to
do with Question A-3 in Fig. 4. To alleviate this issue, following the lead of
[8], we adopt the multi-task instruction template by adding various task tags
to the questions generated from the process described earlier. Similar to [8],
we add the task tag at the beginning of the question. We use [refer] for
questions where hand or object is referred, these are A-1, A-2, B-1 and B-2 in
Fig. 4. We use [identify] for questions where task is to generate referred
hand or object’s name. These are A-3 and A-4 in Fig. 4. We use [vqa] for
questions without bounding boxes, these are B-3 and B-4 in Fig. 4.

###  3.4 HOI-QA: Dataset for Hand-Object Interaction Referral

Having established the framework to generate question-answer pairs in Sec.
3.2, and the targeted HO-Ref and I-Ref from Sec. 3.3, we now detail our
collection of QA pairs using publicly available egocentric datasets.

Out of various available datasets for egocentric videos [17, 21, 5, 44, 31,
16, 45], we select EPIC-Kitchens [17, 16, 18] and Ego4D [21] as our data
source for generating the QA pairs. These are the largest datasets in
egocentric vision and provide crucial annotations including narrations and
hand-object interaction information. This is in contrast with other datasets
that do not have bounding box annotations [48, 5] or hand-object interaction
information [48, 5, 31]. Other datasets that have the hand-object information
are noticeably smaller [44, 45].

EPIC-Kitchens contains all the annotation types mentioned in Sec. 3.2, _i.e_.,
narrations [16, 17], object bounding boxes [16], and hand-object contact
information [18]. Ego4D contains a large number of narrations and hand and
object bounding box annotation types, particularly within the forecasting
hands and objects (FHO) benchmark which we select as a subset.

Table 1: Comparison of referral-based question-answering datasets. Our HOI-QA
dataset has multiple orders of magnitude more questions than the current
largest third-person dataset

Dataset # QA Pairs Automatic Generation? Task Tags? Egocentric? Shikra-RD [9]
$5,922$ ✗ ✗ ✗ Flicker30k [8, 39] $2.5$K ✓ ✗ ✗ Multi-task Conv. [8] $60.9$K ✓ ✓
✗ HOI-QA (ours) $\mathbf{3.9}$M ✓ ✓ ✓

We create 3.9M question-answer pairs for the HOI-QA dataset. Out of 3.9M QA
pairs, 1.8M pairs are from EPIC-Kitchen’s and 2.1M pairs are from Ego4D’s FHO
benchmark. Based on the train/val splits of EPIC-Kitchens and Ego4D, HOI-QA is
divided into $3.2$M QA pairs for train and $748$K for test. Out of $748$K QA
pairs in the test set, approximately $20\%$ are from EPIC-Kitchens and $80\%$
are from Ego4D. Note that the split is on the video level, i.e. all QA pairs
from the same video are in the same split (train/test).

The scale of the proposed HOI-QA is suitable for training large models. In
Tab. 1, we compare HOI-QA with existing referential-based question-answering
datasets. HOI-QA contains the highest number of question-answer pairs; is the
only dataset for egocentric images; and does not require prompting an LLM [9]
for its creation, resulting in a cleaner set of annotations. Figure 3 (right)
shows the distribution of questions within HOI-QA.

##  4 Experiments

In this section, we first provide details on implementation (Sec. 4.1);
evaluation protocol (Sec. 4.2); and baselines (Sec. 4.3) followed by results
of VLM4HOI on HOI-QA (Sec. 4.4), ablations (Sec. 4.6), and qualitative results
(Sec. 4.5).

###  4.1 Implementation Details

We use EVA [20] as our vision encoder for VLM4HOI and keep it frozen during
training. We train the linear projection layer and fine-tune the large
language model. We use LLaMA2-chat (7B) [51] as our LLM which is finetuned
using LoRA [24] with rank $r=64$. Following [8], we train VLM4HOI with an
image resolution of $448\times 448$ and before projecting to LLaMA-2’s
representation space, four adjacent visual output tokens from EVA are
concatenated. We initialise the language model using the weights from the
third stage of MiniGPT-v2 [8]. This allows us to finetune a model that has
knowledge about referral in third-person images and task tags. Additionally,
we adapt the instruction template from LLaMA-2 as our input to the model:
[INST] <ImageFeature> [task tag] Instruction [/INST]. Here,
ImageFeature is replaced by the visual features in LLaMA-2’s embedding space
and the relevant task tag and instruction are filled in. Finally, all the
bounding boxes in the question-answer pairs are normalised to $(100,100)$. We
train VLM4HOI using $8$ Nvidia V100 GPUs for 1 epoch on HOI-QA that takes
approximately $28$ hours.

###  4.2 Metrics

Based on answer type, we have two evaluation schemes. For questions involving
a noun as answer, we perform word-level matching and calculate accuracy. For
questions generating bounding boxes, we consider the predicted bounding box as
correct if its IoU with the ground truth is above $0.5$. For questions with
multiple bounding boxes in their answers, we consider the answer to correctly
only when each ground-truth bounding box has prediction with IoU above $0.5$.

###  4.3 Baselines

To evaluate VLM4HOI on HOI-QA we consider the following baselines:  1. Random.
For noun generation, we select a noun from the distribution of nouns within
the training set. For bounding boxes, we use the bounding box obtained by
calculating the mean value of top-left and bottom-right corners of the
bounding boxes over the training set as our prediction.  2. MiniGPT-v2 [8].
Trained for referral on third-person images and utilises task tags to flag the
type of response expected. This is the closest baseline to VLM4HOI. We use the
publicly available model from the final stage of MiniGPT-v2 as our baseline.
There are additional works performing referral on third-person images (_e.g_.
[9, 38]), however, they do not utilise task tags for a fair comparison and are
outperformed on existing datasets by MiniGPT-v2 based on the results reported
in [8].

###  4.4 Quantitative Evaluation

In this section, we first present results of VLM4HOI on HOI-QA, before
analysing how well VLM4HOI performs on each task: HO-Ref and I-Ref.

####  4.4.1 Results of VLM4HOI on HOI-QA

Table 2 (left) contains results on the complete test set of HOI-QA. The random
baseline performs poorly on both noun and bounding box accuracy. This
highlights the challenging nature of HOI-QA. Secondly, VLM4HOI outperforms
both the baselines by a large margin. Specifically, VLM4HOI achieves $11.08\%$
higher noun accuracy and $27.85\%$ higher bounding box accuracy as compared to
MiniGPT-v2 [8]. This shows that VLMs trained on third-person images fall short
to generalise on egocentric images, and that improving their performance on
egocentric data requires a specialised dataset. Furthermore, as questions in
HOI-QA require understanding hand-object interactions, the significant
difference in performance between MiniGPT-v2 and VLM4HOI indicates that even
large scale third-person training does not enable hand-object interaction
understanding and localisation.

By analysing the two subsets of our benchmark (Tab. 2, right), HO-Ref vs
I-Ref, we note that our model’s focus on interaction referral results in a
significant increase in performance for noun accuracy during referral
($30.39\%$ on I-Ref compared to $10.10\%$ on HO-Ref). Improvements in bounding
box accuracy are comparable. We provide a detailed breakdown of results on the
eight sub-divisions of HO-Ref and I-Ref in the supplementary.

Table 2: Results on HOI-QA and analysis on HO-Ref and I-Ref. Left shows the
results on all of HOI-QA’s test set. Right compares the performance on HO-Ref
and I-Ref. VLM4HOI outperforms the baselines highlighting its capability to
understand hands/objects and refer interactions in an egocentric image. Here,
N-A stands for Noun Accuracy, BB-A stands for Bounding Box Accuracy, and Avg.
IOU stands for Average bounding box accuracy

Model HOI-QA HO-Ref I-Ref N-A BB-A Avg. IOU N-A BB-A Avg. IOU N-A BB-A Avg.
IOU Random $0.50$ $1.22$ $7.00$ $0.05$ $1.15$ $6.77$ $0.02$ $0.07$ $12.45$
MiniGPT-v2 [8] $6.70$ $22.46$ $24.49$ $6.68$ $22.13$ $24.15$ $7.06$ $30.12$
$32.53$ VLM4HOI (ours) $\mathbf{17.78}$ $\mathbf{50.26}$ $\mathbf{43.21}$
$\mathbf{16.78}$ $\mathbf{49.98}$ $\mathbf{42.83}$ $\mathbf{37.45}$
$\mathbf{56.86}$ $\mathbf{52.40}$

###  4.5 Qualitative Results

Figure 5 shows qualitative results for VLM4HOI and MiniGPT-v2. VLM4HOI
performs adequately for the cases shown where MiniGPT-v2 falls short. VLM4HOI
is capable of referring hands even when in glove (row-1, col-1), identify the
exact object in hand amongst others on the shelf (row-1, col-2) and better
predict the bounding box for in-hand objects (row-2, col-3).

VLM4HOI fails in the ambiguous cases. For example, when asked about what is in
the right hand, VLM4HOI identifies the fork as a spoon (row-1, col-5) due to
the small size of the object as well as motion blur. When the hand side is
missing, both models are confused by the present hand or even feet in view
(row-1, col-4). At times, the annotations might be confusing themselves. For
example when asked about the bounding box (53, 54, 65, 55) (row-2, col-4), the
hand fully capturing the cloth makes a perfectly overlapping bounding box.
VLM4HOI identifies the object as opposed to the hand.

Figure 5: Qualitative Results on VLM4HOI and MiniGPT-v2 [8] on HOI-QA. For
questions with correct bounding box output, the ground truth bounding box is
omitted. When both models are incorrect, we add the ground truth in blue.
VLM4HOI performs well on most of the cases where MiniGPT-v2 falls short.
VLM4HOI fails in case of ambiguity. For example, it identifies the hand as
cloth as the hand is holding the cloth (MiniGPT-v2 predicts it as a waffle).

###  4.6 Ablation Study

In this section, we first ablate whether spatial information is important by
removing bounding box supervision during training. Next, we showcase the
importance of dataset selection of both EPIC and Ego4D data within HOI-QA.
Finally, we evaluate the importance of the unique task tags in training
VLM4HOI.

Table 3: Importance of Spatial Information. Train VLM4HOI without any spatial
supervision. We show that for enabling hand-object interaction referral,
spatial information is crucial

Model Noun Accuracy Bounding Box Accuracy Avg. Bounding Box IOU w/o Bbox.
$10.72$ $8.98$ $14.74$ w/ Bbox. $\mathbf{17.78}$ $\mathbf{50.26}$
$\mathbf{43.21}$

####  4.6.1 Importance of spatial information.

Here, we study the impact of bounding box supervision in training VLM4HOI. We
select the subset of question-answer pairs from HOI-QA that do not contain any
bounding box supervision. This leaves us with a subset of $691,380$ QA pairs
from Ego4D and $279,458$ QA pairs from EPIC-Kitchens (approximately $971K$ QA
pairs) for training VLM4HOI. As shown in Tab. 3, the bounding box accuracy
significantly drops across all results. It is intuitive that without training
for spatial localisation, VLM4HOI will not be able to perform well. The
results however show that the model is unable to leverage its previous
training on third-person data and LLaMA-2’s large training corpus to locate
objects in egocentric images. Furthermore, the drop in noun accuracy
highlights the importance of benefit of localisation supervision to
understanding an object’s appearance.

####  4.6.2 Dataset Contributions.

As we train on annotations from two datasets, we analyse the contribution of
each dataset to the results by training on QA pairs solely from each dataset.
Accordingly, these are: the $1.7$M pairs of HOI-QA from EPIC-Kitchens; and the
$1.5$M QA pairs of HOI-QA from Ego4D. As shown in Tab. 4 Bounding box accuracy
drops when we train on only one dataset highlighting that both the datasets
play a crucial role for hand-object interaction referral. The model trained
solely on Ego4D performs better as compared to training on EPIC-Kitchens, this
is because of the larger and diverse nature of the Ego4D dataset. However, the
noun accuracy drops when we utilise both the datasets. we believe this is
because of the bias in our test set being more aligned with the smaller subset
of noun classes available in EPIC-Kitchens.

Table 4: Dataset Contributions. Here we train the model on a specific dataset
and test on the test set of HOI-QA. This highlights dataset’s contribution to
the performance.

Model Noun Accuracy Bounding Box Accuracy Avg. Bounding Box IOU Only EPIC
$\mathbf{20.63}$ $25.45$ $28.16$ Only Ego4D $13.71$ $43.42$ $38.12$ Full
Training $17.78$ $\mathbf{50.26}$ $\mathbf{43.21}$

####  4.6.3 Importance of Task Tags.

In this section, we evaluate the importance of the unique task identifiers in
the form of task tags in which we train VLM4HOI with/without the task tags.
The results can be seen in Tab. 5 where the model with task tags outperforms
by a small margin of $0.31\%$ on bounding box accuracy and $0.56\%$ on noun
accuracy. As in [8], the inclusion of task tags marginally helps the model
disambiguate the questions and improves overall performance.

Table 5: Importance of Task Tags. Here we train and evaluate VLM4HOI without
the task tags. Similar to [8], we see a slight improvement in the performance

Model Noun Accuracy Bounding Box Accuracy Avg. Bounding Box IOU w/o Task Tags
$17.22$ $49.95$ $43.07$ w/ Task Tags $\mathbf{17.78}$ $\mathbf{50.26}$
$\mathbf{43.21}$

###  4.7 Limitations

There are two sources of bias in VLM4HOI—the first is inherit bias in the pre-
trained LLM. Despite fine-tuning, language biases are known to exist in all
VLMs. Secondly, bias in the HOI-QA, which corresponds to the natural
occurrences of certain actions, objects and interactions over others. As
VLM4HOI is trained on unconstrained, long-tailed data of objects and actions
[21, 17], it will more commonly predict frequent objects and interactions over
infrequent ones.

VLM4HOI is also reliant on the strength of the visual encoder. _i.e_. Small
objects and motion blur can cause errors in the model’s prediction and would
require a stronger visual encoder or multiple frames from a video.

##  5 Conclusion

In this work, we propose and explore the HOI-Ref task for egocentric vision to
enable hand-object interaction referral using VLMs. We further investigate two
aspects of HOI-Ref, referral of hands and objects (HO-Ref) and understanding
of the interactions between them (I-Ref). We propose the HOI-QA dataset that
consists of 3.9M question-answer pairs for training and evaluation of VLMs for
HOI-Ref. We train a unified VLM, VLM4HOI, on HOI-QA and significantly improve
hand-object interaction referral performance over baselines. The dataset,
models, and task proposed will pave the way for hand-object interaction
referral in egocentric images.

### Acknowledgements

This work uses public datasets—code and models are publicly available. The
research is supported by EPSRC UMPIRE EP/T004991/1 and EPSRC Programme Grant
VisualAI EP/T028572/1. S Bansal is supported by a Charitable Donation to the
University of Bristol from Meta. We acknowledge the use of the EPSRC funded
Tier 2 facility JADE-II EP/T022205/1.

The authors would like to thank Alexandros Stergiou, Kranti Kumar Parida,
Samuel Pollard, and Rhodri Guerrier for their comments on the manuscript.

## References

  * [1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023)
  * [2] Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS) (2022)
  * [3] Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al.: OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models. arXiv preprint arXiv:2308.01390 (2023)
  * [4] Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities. arXiv preprint arXiv:2308.12966 (2023)
  * [5] Bansal, S., Arora, C., Jawahar, C.: My View is the Best View: Procedure Learning from Egocentric Videos. In: European Conference on Computer Vision (ECCV) (2022)
  * [6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [7] Chen, J., Zhu, D., Haydarov, K., Li, X., Elhoseiny, M.: Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions. arXiv preprint arXiv:2304.04227 (2023)
  * [8] Chen, J., Zhu, D., Shen, X., Li, X., Liu, Z., Zhang, P., Krishnamoorthi, R., Chandra, V., Xiong, Y., Elhoseiny, M.: MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 (2023)
  * [9] Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic. arXiv preprint arXiv:2306.15195 (2023)
  * [10] Cheng, S., Guo, Z., Wu, J., Fang, K., Li, P., Liu, H., Liu, Y.: Can Vision-Language Models Think from a First-Person Perspective? arXiv preprint arXiv:2311.15596 (2023)
  * [11] Cheng, T., Shan, D., Hassen, A., Higgins, R., Fouhey, D.: Towards A Richer 2D Understanding of Hands at Scale. Advances in Neural Information Processing Systems (NeurIPS) (2023)
  * [12] Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality (March 2023)
  * [13] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research (JMLR) (2023)
  * [14] Dai, G., Shu, X., Wu, W.: GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition. arXiv preprint arXiv:2401.10039 (2024)
  * [15] Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In: Advances in Neural Information Processing Systems (NeurIPS) (2023)
  * [16] Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. In: European Conference on Computer Vision (ECCV) (2018)
  * [17] Damen, D., Doughty, H., Farinella, G.M., Furnari, A., Ma, J., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., Wray, M.: Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. International Journal of Computer Vision (IJCV) (2022)
  * [18] Darkhalil, A., Shan, D., Zhu, B., Ma, J., Kar, A., Higgins, R., Fidler, S., Fouhey, D., Damen, D.: EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. In: Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks (2022)
  * [19] Dou, Z.Y., Kamath, A., Gan, Z., Zhang, P., Wang, J., Li, L., Liu, Z., Liu, C., LeCun, Y., Peng, N., et al.: Coarse-to-fine vision-language pre-training with fusion in the backbone. Advances in Neural Information Processing Systems (NeurIPS) (2022)
  * [20] Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., Cao, Y.: EVA: Exploring the Limits of Masked Visual Representation Learning at Scale. In: Computer Vision and Pattern Recognition (CVPR) (2023)
  * [21] Grauman, K., et al.: Ego4D: Around the World in 3,000 Hours of Egocentric Video. In: Computer Vision and Pattern Recognition (CVPR) (2022)
  * [22] Guo, Q., De Mello, S., Yin, H., Byeon, W., Chun Cheung, K., Yu, Y., Luo, P., Liu, S.: Regiongpt. Computer Vision and Pattern Recognition (CVPR) (2024)
  * [23] Hazra, R., Chen, B., Rai, A., Kamra, N., Desai, R.: EgoTV: Egocentric Task Verification from Natural Language Task Descriptions. In: International Conference on Computer Vision (ICCV) (2023)
  * [24] Hu, E.J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-Rank Adaptation of Large Language Models. In: International Conference on Learning Representations (ICLR) (2022)
  * [25] Journal, W.S.: I Spent 24 Hours Wearing Apple’s Vision Pro Headset | WSJ (2024), https://youtu.be/8xI10SFgzQ8?t=283, Last accessed: 2024-03-04
  * [26] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: MDETR – Modulated Detection for End-to-End Multi-Modal Understanding. In: International Conference on Computer Vision (ICCV) (2021)
  * [27] Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: International Conference on Computer Vision (ICCV) (2011)
  * [28] Kurita, S., Katsura, N., Onami, E.: RefEgo: Referring Expression Comprehension Dataset from First-Person Perception of Ego4D. In: International Conference on Computer Vision (ICCV) (2023)
  * [29] Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In: International Conference on Machine Learning (ICML) (2022)
  * [30] Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: VideoChat: Chat-Centric Video Understanding. arXiv preprint arXiv:2305.06355 (2023)
  * [31] Li, Y., Liu, M., Rehg, J.M.: In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video. In: European Conference on Computer Vision (ECCV) (2018)
  * [32] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual Instruction Tuning. Advances in Neural Information Processing Systems (NeurIPS) (2023)
  * [33] Liu, Z., He, Y., Wang, W., Wang, W., Wang, Y., Chen, S., Zhang, Q., Yang, Y., Li, Q., Yu, J., et al.: InternChat: Solving Vision-Centric Tasks by Interacting with Chatbots Beyond Language. arXiv preprint arXiv:2305.05662 (2023)
  * [34] Munasinghe, S., Thushara, R., Maaz, M., Rasheed, H.A., Khan, S., Shah, M., Khan, F.: PG-Video-LLaVA: Pixel Grounding Large Video-Language Models. arXiv preprint arXiv:2311.13435 (2023)
  * [35] Narasimhaswamy, S., Nguyen, T., Nguyen, M.H.: Detecting Hands and Recognizing Physical Contact in the Wild. Advances in Neural Information Processing Systems (NeurIPS) (2020)
  * [36] Narasimhaswamy, S., Wei, Z., Wang, Y., Zhang, J., Hoai, M.: Contextual Attention for Hand Detection in the Wild. In: International Conference on Computer Vision (ICCV) (2019)
  * [37] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.: Training language models to follow instructions with human feedback. Neural Information Processing Systems (NeurIPS) (2022)
  * [38] Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Wei, F.: Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv preprint arXiv:2306.14824 (2023)
  * [39] Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. In: International Conference on Computer Vision (ICCV) (2015)
  * [40] Pramanick, S., Song, Y., Nag, S., Lin, K.Q., Shah, H., Shou, M.Z., Chellappa, R., Zhang, P.: EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone. In: International Conference on Computer Vision (ICCV) (2023)
  * [41] Qinghong Lin, K., Jinpeng Wang, A., Soldan, M., Wray, M., Yan, R., Zhongcong Xu, E., Gao, D., Tu, R., Zhao, W., Kong, W., et al.: Egocentric Video-Language Pretraining. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)
  * [42] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning Transferable Visual Models From Natural Language Supervision. In: International Conference on Machine Learning (ICML) (2021)
  * [43] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research (JMLR) (2020)
  * [44] Ragusa, F., Furnari, A., Farinella, G.M.: MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain. Computer Vision and Image Understanding (CVIU) (2023)
  * [45] Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., Yao, A.: Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities. Computer Vision and Patter Recognition (CVPR) (2022)
  * [46] Shan, D., Geng, J., Shu, M., Fouhey, D.F.: Understanding Human Hands in Contact at Internet Scale. In: Computer Vision and Pattern Recognition (CVPR) (2020)
  * [47] Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y.: HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. Advances in Neural Information Processing Systems (NeurIPS) (2023)
  * [48] Sigurdsson, G.A., Gupta, A., Schmid, C., Farhadi, A., Alahari, K.: Actor and Observer: Joint Modeling of First and Third-Person Videos. In: Computer Vision and Pattern Recognition (CVPR) (2018)
  * [49] Sigurdsson, G.A., Gupta, A., Schmid, C., Farhadi, A., Alahari, K.: Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos. arXiv preprint arXiv:1804.09626 (2018)
  * [50] Soomro, K., Zamir, A.R., Shah, M.: UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
  * [51] Touvron, H., et al.: Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288 (2023)
  * [52] Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N.A., Khashabi, D., Hajishirzi, H.: Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv preprint arXiv:2212.10560 (2022)
  * [53] Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Selvan Dhanasekaran, A., Naik, A., Stap, D., et al.: Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks. arXiv preprint arXiv:2204.07705 (2022)
  * [54] Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., Duan, N.: Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models. arXiv preprint arXiv:2303.04671 (2023)
  * [55] Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., Wang, L.: MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action. arXiv preprint arXiv:2303.11381 (2023)
  * [56] Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., et al.: mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality. arXiv preprint arXiv:2304.14178 (2023)
  * [57] Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al.: GLM-130B: An Open Bilingual Pre-trained Model. arXiv preprint arXiv:2210.02414 (2022)
  * [58] Zhang, R., Han, J., Zhou, A., Hu, X., Yan, S., Lu, P., Li, H., Gao, P., Qiao, Y.: LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention. arXiv preprint arXiv:2303.16199 (2023)
  * [59] Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X.V., et al.: OPT: Open Pre-trained Transformer Language Models. arXiv preprint arXiv:2205.01068 (2022)
  * [60] Zhao, Y., Misra, I., Krähenbühl, P., Girdhar, R.: Learning Video Representations from Large Language Models. In: Computer Vision and Pattern Recognition (CVPR) (2023)
  * [61] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)

##  6 Appendix

###  6.1 Overview

In the supplementary, we provide in-depth details of the proposed HOI-QA
dataset (Sec. 7). We then provide additional quantitative results in Sec. 8,
showcasing example questions from both EPIC-Kitchens [16] and Ego4D [21]
datasets. Finally, we present zero-shot interactive results using the proposed
VLM4HOI model on the HOI-QA dataset in Sec. 9.

##  7 Dataset details

###  7.1 Additional dataset statistics

####  7.1.1 HOI-QA distribution details.

In this section, we dive deep into the details of the proposed HOI-QA dataset.
As mentioned in the main paper, HOI-QA consists of $3.9$M question-answer
pairs to train and test various Vision Language Models (VLMs). Figure 6 shows
different statistics of HOI-QA regarding the QA pairs.

Figure 6 (a) contains the distribution of QA pairs with and without bounding
boxes. As we particularly target referral, the dataset has a larger number of
questions with bounding boxes to enable hand-object interaction referral for
VLMs. In Figure 3 (main paper) we show the distribution of QA pairs based on
the question type, here (Fig. 6 (b)) we additionally show the distribution of
task tags within the dataset. Questions with tags [refer] and [identify] make
up the majority of questions, approximately $67\%$ of the dataset. Figure 6
(c) further demonstrates the distribution across the parent datasets—EPIC-
Kitchens and Ego4D. Due to the availability of more videos and annotations in
Ego4D, there are more QA pairs from Ego4D in HOI-QA over EPIC-Kitchens.
However, we do not have [vqa] questions from Ego4D as it does not contain
hand-object interaction information as provided in EPIC-Kitchens VISOR [18].
This also leads to less questions from Ego4D for [detect], [caption], and
[grounding].

Figure 6 (d) shows the distribution of the train and validation splits of
EPIC-Kitchens and Ego4D across the proposed HOI-QA dataset. As can be seen,
the train set of HOI-QA contains a similar set of QA pairs from EPIC-Kitchens
and Ego4D for training, however, due to a large number of videos in the
validation set of Ego4D’s FHO benchmark, the number of QA pairs from Ego4D in
the test set of HOI-QA are greater than EPIC-Kitchens.

Figure 6:  HOI-QA distribution details: (a) shows the distribution of QA pairs
across the HOI-QA dataset with and without bounding boxes; (b) contains the
distribution of QA pairs in HOI-QA based on the task tags; (c) provides the
distribution of QA pairs across EPIC-Kitchens and Ego4D; and (d) showcases the
distribution of EPIC-Kitchens and Ego4D across the train and test split of
HOI-QA. As can be seen, the majority of QA pairs contains bounding boxes to
enable hand-object interaction referral.

####  7.1.2 Bounding Box Distribution Heatmaps.

To analyse the distribution of bounding boxes across the proposed HOI-QA
dataset, we create heatmaps of the bounding boxes in EPIC-Kitchens and Ego4D.
To generate the heatmaps, we overlay all the bounding boxes on an $100\times
100$ image and take the $\log$ of it. Figure 7 shows the two heatmaps. Due to
the nature of egocentric videos, the bounding boxes are close to the centre.
However, as we are considering bounding boxes for both hands and objects, the
distribution is more spread. Moreover, owing to the diversity of activities in
Ego4D, the distribution of bounding boxes show a greater deviation in
comparison to EPIC-Kitchens. Finally, it is important to note that the Random
baseline from the main paper highlights that using the mean bounding box
position/size (calculated from the training data) will lead to only $1.22\%$
bounding box accuracy. Therefore, we can conclude that the bounding box
distribution is not strongly biased towards the centre and well-performing
models must understand the visual and textual aspects of the task to succeed.

Figure 7: Bounding Box’s heatmaps: (a) shows the heat map of bounding boxes in
answers from EPIC-Kitchens and (b) shows the same for Ego4D used to generate
the question-answer pairs for HOI-QA.

####  7.1.3 Bounding Box Size Analysis.

To understand the hand and object bounding box’s size distribution in HOI-QA,
we calculate the IOU between the image and the bounding box. Figure 8 shows
the histogram of IOU distribution for data from EPIC-Kitchens and Ego4D in
HOI-QA. Both the histograms are right skewed showing that majority of the
bounding boxes are small in the dataset. Therefore, for a model to achieve
high accuracy for referral, understanding small objects is crucial.

Figure 8: Bounding Box Size Distribution. The right skewed distribution shows
that majority of the objects in the dataset are small in size.

####  7.1.4 Word Clouds for Nouns.

In Fig. 9, we show the word clouds of nouns in the proposed HOI-QA dataset
from EPIC-Kitchens and Ego4D. As can be seen in the figure, there is a clear
difference between the two datasets due to the nature of their collection.
EPIC-Kitchens consists of nouns surrounding kitchen objects whereas Ego4D
contains more diverse nouns. Due to this, identifying objects and generating
the nouns on Ego4D is challenging as also highlighted in Tab. 9.

Figure 9: HOI-QA Word cloud for nouns in EPIC-Kitchens and Ego4D. Owing to the
nature of datasets, we see more diverse vocabulary in Ego4D and a specific
vocabulary in EPIC-Kitchens. Due to this, the overall vocabulary of the HOI-QA
dataset is diverse.

####  7.1.5 Images and Videos Count.

In Tab. 6, we show the count of images and videos used from EPIC-Kitchens and
Ego4D to generate the QA pairs for the HOI-QA dataset.

Table 6:  Number of images and videos used from EPIC-Kitchens and Ego4D to
generate $3.9$M question-answer pairs in the HOI-QA dataset

Dataset Number of Images Number of Videos EPIC-Kitchens [17, 16, 18] $208,656$
$495$ Ego4D [21] $665,217$ $2,461$

###  7.2 List of all the question pairs to generate the data

Apart from the question-answer pairs shown in the main paper, there are a
variety of ways in which the same question has been phrased. Here are all the
prompts used as questions in the HOI-QA dataset divided based on the task
tags:

  1. 1.

captions

    1. (a)

This is a photo from an ongoing video of hand object interactions. What is
happening in the photo?

    2. (b)

What is going on in the photo?

    3. (c)

This is a photo has hand object interactions. Describe the ongoing action in
the photo.

    4. (d)

What is the hand-object interaction happening in the image?

    5. (e)

What actions are the hands doing in this photo?

  2. 2.

vqa

    1. (a)

This is a photo from an ongoing video of hand object interactions. Is there an
object in the left hand?

    2. (b)

Is the left hand holding or touching an object?

    3. (c)

This is a photo from an ongoing video of hand object interactions. Is the left
hand holding an object or is object-free?

    4. (d)

Is the left hand object-free?

    5. (e)

This is a photo from an ongoing video of hand object interactions. Is there an
object in the right hand?

    6. (f)

Is the right hand holding or touching an object?

    7. (g)

This is a photo from an ongoing video of hand object interactions. Is the
right hand holding an object or is object-free?

    8. (h)

Is the right hand object-free?

    9. (i)

This is a photo from an ongoing video of hand object interactions. What is the
object in the left hand?

    10. (j)

What is the object in the hand of the person?

    11. (k)

What is the person holding in their left hand?

    12. (l)

What is the object in the right hand?

    13. (m)

This is a photo from an ongoing video of hand object interactions. What is the
person holding in their right hand?

    14. (n)

What is the object in the hands of the person?

    15. (o)

What hand is holding the {object’s name}?

    16. (p)

What hand is holding the {object’s name}?

    17. (q)

What hand is holding the {object’s name} in the image?

  3. 3.

identify

    1. (a)

${\\{<X_{\text{left}}><Y_{\text{top}}><X_{\text{right}}><Y_{\text{bottom}}>\\}}$

  4. 4.

refer

    1. (a)

Where is the left hand of the person?

    2. (b)

Where is the left hand?

    3. (c)

Where is the left hand of the person in the image?

    4. (d)

left hand

    5. (e)

Where is the right hand of the person?

    6. (f)

Where is the right hand?

    7. (g)

Where is the right hand of the person in the image?

    8. (h)

right hand

    9. (i)

Where are the hands of the person?

    10. (j)

Where are the hands?

    11. (k)

Where are the hands of the person in the image?

    12. (l)

Locate the object being manipulated by the hand

    13. (m)

Where is the object being manipulated by the hand?

    14. (n)

Locate the manipulated object

    15. (o)

Where is the manipulated object?

    16. (p)

Which hand is holding the {object’s name} in?

    17. (q)

Which hand is holding the {object’s name} in the image?

    18. (r)

Which hand has the {object’s name}?

  5. 5.

grounding

    1. (a)

This is a photo from an ongoing video of hand object interactions. Describe
hand-object interaction in detail

    2. (b)

Describe the ongoing action in this image in detail

    3. (c)

This is a photo from an ongoing video of hand object interactions. Describe
what actions are the hand doing in this photo

    4. (d)

This is a photo from an ongoing video of hand object interactions. Describe
the ongoing action in this image

    5. (e)

Describe the ongoing action in this image

  6. 6.

detection

    1. (a)

{object’s name}

##  8 Additional Quantitative Results

###  8.1 Analysis on HO-Ref and I-Ref

Here we take a deeper look into the results for HOI-Ref on HOI-QA from HO-Ref
and I-Ref’s perspective. We do this by further dividing the results into the
four categories of question-answer types described in Section 3.3 in the main
paper. The results are divided across two tables, Tab. 7 and Tab. 8.

Table 7:  Results on subsets in HO-Ref showcases the ability of baseline and
VLM4HOI on each category of QA pairs. Here, Obj N stands for Object Noun, Obj
BB stands for object bounding box, Hand S stands for Hand Side (Left/Right),
and Hand BB stands for hand bounding box

Model Obj N $\rightarrow$ Obj BB Hand S $\rightarrow$ Hand BB Obj BB
$\rightarrow$ Obj N Hand BB $\rightarrow$ Hand S MiniGPT-v2 [8] $25.37$
$19.90$ $15.67$ $0.01$ VLM4HOI (ours) $\mathbf{34.27}$ $\mathbf{60.38}$
$\mathbf{31.85}$ $\mathbf{5.60}$

As seen in Tab. 7, VLM4HOI outperforms baseline in all the four categories.
VLM4HOI has high performance when localising the object or hands showing its
superior capability for referral in egocentric images. Moreover, the baseline
falls short on generating hand side’s name provided hand’s bounding box,
whereas, VLM4HOI has $5.59\%$ improvement over it.

Table 8 shows similar results for I-Ref. Here, we test the VLMs capability to
refer to object/hand provided hand/object information. As can be seen, VLM4HOI
significantly outperforms the baseline. Especially for the case where object
name is provided and hand’s bounding box or noun is supposed to be generated,
VLM4HOI performs $43.41\%$ better on average. Furthermore, when object’s name
is provided, generating hand’s name is challenging for MiniGPT-v2 however,
VLM4HOI performs well on such cases.

Table 8:  Results on subsets in I-Ref showcases the ability of baseline and
VLM4HOI on each category of QA pairs. Here, Obj N stands for Object Noun, Obj
BB stands for object bounding box, Hand S stands for Hand Side, and Hand BB
stands for hand bounding box

Model Hand S $\rightarrow$ Obj BB Obj N $\rightarrow$ Hand BB Hand S
$\rightarrow$ Obj N Obj N $\rightarrow$ Hand S MiniGPT-v2 [8] $27.69$ $32.55$
$14.12$ $0.00$ VLM4HOI (ours) $\mathbf{41.13}$ $\mathbf{72.59}$
$\mathbf{28.11}$ $\mathbf{46.78}$

###  8.2 Results per parent dataset: EPIC-Kitchens and Ego4D

In Tab. 9 we show the results the validation splits of EPIC-Kitchens and Ego4D
used to create the test set of HOI-QA. There is a large boost in performance
when compared to MiniGPT-v2 [8] on both EPIC and Ego4D. The boost in noun
accuracy for Ego4D is less compared to EPIC-Kitchens due to the diversity of
nouns in Ego4D as shown in Fig. 9.

Table 9:  Results on EPIC-Kitchens and Ego4D in HOI-QA. Our VLM4HOI
outperforms MiniGPT-v2 on both the subsets within HOI-QA

Model EPIC-Kitchens [16, 17, 18] Ego4D [21] Noun Acc. Bounding Box Acc. Avg.
BB IOU Noun Acc. Bounding Box Acc. Avg. BB IOU MiniGPT-v2 [8] $13.56$ $34.88$
$33.96$ $5.38$ $19.56$ $22.04$ VLM4HOI (ours) $\mathbf{43.32}$
$\mathbf{65.30}$ $\mathbf{58.24}$ $\mathbf{12.86}$ $\mathbf{46.76}$
$\mathbf{39.76}$

##  9 Zero-shot Interactive Results

In this section, we perform zero-shot qualitative analysis using the VLM4HOI
trained on HOI-QA. For this purpose, we randomly select images from the test
set of HOI-QA and prompt the model with novel questions interactively. As
there are no ground truth for these questions, we qualitatively assess the
generation. This highlights the generalisability of VLM4HOI over various
prompts for hand-object interaction referral on egocentric images.

Figure 10 shows the results on images from the EPIC-Kitchens used in HOI-QA.
As can be seen in the figure, the model generates correct results for majority
of the cases. For example, the model correctly identifies the hands, objects
in hands, and generates the name of the objects. It confuses for the cases
where the objects are overlapping, for example, in row $6$ column $1$, the
model confuses between the fish and bowl. Figure 11 presents the results on
the images from the Ego4D used in HOI-QA. Here, the model is able to refer to
various objects in the wild for example, golf club, dog, and a wagon. However,
it confuses for the cases where objects are similar looking for example, in
row $5$ column $2$, the model confuses sanding the wood with spraying nitrogen
on the wood.

Figure 10: Zero-shot results on the EPIC-Kitchen’s subset of HOI-QA validation
set using VLM4HOI. Green bounding boxes represent VLM4HOI’s output and orange
bounding boxes represent input to the model. After qualitative analysis the
correct generations are marked by a tick and wrong generations are marked by a
cross. Figure 11: Zero-shot results on the Ego4D’s subset of HOI-QA validation
set using VLM4HOI. Green bounding boxes represent VLM4HOI’s output and orange
bounding boxes represent input to the model. After qualitative analysis the
correct generations are marked by a tick and wrong generations are marked by a
cross.