11institutetext:  University of Kaiserslautern-Landau, Kaiserslautern, Germany
22institutetext: Deutsches Forschungszentrum für Künstliche Intelligenz,
Kaiserslautern, Germany 33institutetext: ETH Zürich, Rämistrasse 101, Zurich,
Switzerland 44institutetext: Technical University of Munich, Munich, Germany

# FocusCLIP: Multimodal Subject-Level Guidance for Zero-Shot Transfer in
Human-Centric Tasks

Muhammad Saif Ullah Khan 1 University of Kaiserslautern-Landau,
Kaiserslautern, Germany 12Deutsches Forschungszentrum für Künstliche
Intelligenz, Kaiserslautern, Germany 2 0000-0002-7375-807X Muhammad Ferjad
Naeem 3ETH Zürich, Rämistrasse 101, Zurich, Switzerland 3 0000-0001-7455-7280
Federico Tombari 4Technical University of Munich, Munich, Germany 4
0000-0001-5598-5212 Luc Van Gool 3ETH Zürich, Rämistrasse 101, Zurich,
Switzerland 3 Didier Stricker 1 University of Kaiserslautern-Landau,
Kaiserslautern, Germany 12Deutsches Forschungszentrum für Künstliche
Intelligenz, Kaiserslautern, Germany 2 Muhammad Zeshan Afzal 1 University of
Kaiserslautern-Landau, Kaiserslautern, Germany 12Deutsches Forschungszentrum
für Künstliche Intelligenz, Kaiserslautern, Germany 21 University of
Kaiserslautern-Landau, Kaiserslautern, Germany 12Deutsches Forschungszentrum
für Künstliche Intelligenz, Kaiserslautern, Germany 2 0000-0002-7375-807X 3ETH
Zürich, Rämistrasse 101, Zurich, Switzerland 3 0000-0001-7455-7280 4Technical
University of Munich, Munich, Germany 4 0000-0001-5598-5212 3ETH Zürich,
Rämistrasse 101, Zurich, Switzerland 31 University of Kaiserslautern-Landau,
Kaiserslautern, Germany 12Deutsches Forschungszentrum für Künstliche
Intelligenz, Kaiserslautern, Germany 21 University of Kaiserslautern-Landau,
Kaiserslautern, Germany 12Deutsches Forschungszentrum für Künstliche
Intelligenz, Kaiserslautern, Germany 2

###### Abstract

We propose FocusCLIP, integrating subject-level guidance–a specialized
mechanism for target-specific supervision–into the CLIP framework for improved
zero-shot transfer on human-centric tasks. Our novel contributions enhance
CLIP on both the vision and text sides. On the vision side, we incorporate ROI
heatmaps emulating human visual attention mechanisms to emphasize subject-
relevant image regions. On the text side, we introduce human pose descriptions
to provide rich contextual information. For human-centric tasks, FocusCLIP is
trained with images from the MPII Human Pose dataset. The proposed approach
surpassed CLIP by an average of 8.61% across five previously unseen datasets
covering three human-centric tasks. FocusCLIP achieved an average accuracy of
33.65% compared to 25.04% by CLIP. We observed a 3.98% improvement in activity
recognition, a 14.78% improvement in age classification, and a 7.06%
improvement in emotion recognition. Moreover, using our proposed single-shot
LLM prompting strategy, we release a high-quality MPII Pose Descriptions
dataset to encourage further research in multimodal learning for human-centric
tasks. Furthermore, we also demonstrate the effectiveness of our subject-level
supervision on non-human-centric tasks. FocusCLIP shows a 2.47% improvement
over CLIP in zero-shot bird classification using the CUB dataset. Our findings
emphasize the potential of integrating subject-level guidance with general
pretraining methods for enhanced downstream performance.

###### Keywords:

Multimodal Learning Zero-Shot Classification Vision-Language Models

##  1 Introduction

Pretraining techniques that leverage multiple modalities have transformed deep
learning, enabling models to capture intricate patterns from vast, unlabeled
datasets [36, 55, 19, 18, 40, 35]. This approach is pivotal for zero-shot
capabilities, which allow models to recognize concepts not seen during
training [59, 66, 34, 22]. Within this evolving landscape, CLIP [40] emerges
as a significant advancement, particularly in language-image pretraining. CLIP
demonstrates impressive performance across various visual tasks by aligning
images and text in a shared embedding space. CLIP-like methods [26, 21, 63,
66, 14] are effective but require a large amount of data. The original CLIP
model used 400 million image-text pairs due to its broad pretraining approach,
making training complex. Recent works [30] have attempted to narrow the
pretraining objective by guiding self-supervision using known priors. Inspired
by that, we argue that when certain downstream tasks are known in advance, we
can limit the learning space by asking the model to ignore irrelevant input
regions during pretraining. In human-centric tasks, the target subject is the
people in images. For such tasks, we propose FocusCLIP, which augments the
CLIP framework on both the vision and text sides to guide the model focus to
pertinent image regions.

ActivityAgeEmotionMean$0$$20$$40$ChanceCLIPFocusCLIP (Ours)

Figure 1: FocusCLIP (left) uses heatmap modality (red pathways) and dual
contrastive loss to learn task-relevant embeddings. It outperforms the CLIP
baseline (right) across three human-centric zero-shot tasks on unseen
datasets.

On the vision side, we draw inspiration from the human gaze, characterized by
rapid movements and fixations, where high-acuity vision is concentrated on
limited spatial regions during fixation [17, 7]. We extend CLIP with an
additional Region of Interest (ROI) heatmap component, imitating the fixation
stage of human vision. These procedurally generated heatmaps act as attention
guidance, allowing the model to concentrate on task-relevant image regions. On
the text side, we create a curated dataset with detailed pose descriptions
generated from LLMs. Our domain-specific text data contains only around 14k
samples with detailed natural language descriptions of activities and body
postures.

Our model aligns three components: text encoder, image encoder, and ROI
encoder—in a unified embedding space. It is trained with a dual contrastive
loss [61], simultaneously aligning the original and heatmap-focused image to
the textual pose descriptions. This two-fold alignment strategy ensures that
the model benefits from the attention offered by the heatmap while retaining
its ability to interpret and analyze raw images effectively. While our image-
to-text contrastive loss is self-supervised, integrating heatmaps provides an
additional supervised modality, enhancing the learning of specialized
representations for human-centric tasks (Fig. 1). Our main contributions
include:

  * •

Integrating subject-level attention in generic contrastive pretraining via ROI
heatmaps, establishing a novel paradigm for focused embedding learning.

  * •

Introduction of a single-shot, structured LLM-prompting method for describing
images in datasets by leveraging both class and image-level annotations.

  * •

Superior zero-shot performance on three human-centric tasks across five unseen
datasets compared to the CLIP baseline.

  * •

Empirical validation showing that subject-level guidance during pretraining
results in more effective feature representations for specialized tasks.

  * •

Public release of the MPII Pose Descriptions dataset generated using GPT [37]
and Llama [54] models, promoting further research in the field.

By emphasizing task-related features, our approach provides a promising
direction for enhancing performance in a wide range of applications requiring
specialized knowledge, using less pretraining data.

##  2 Related Work

Vision-Language Models (VLMs) Multimodal learning [26, 21, 63, 66, 14, 6, 34,
62, 67, 65, 35] has seen significant advancements, primarily focused on
aligning and integrating textual and visual data. Despite the promising
results of CLIP-like models [40, 26, 63, 66, 14, 60, 16], which map images and
text into a shared embedding space, these models struggle with specialized
tasks due to their reliance on generic pretraining data [30]. Furthermore,
their learning objective is global, with no mechanism for fine-grained
alignment between image and text regions [34, 33]. Several works have
attempted to address this limitation by introducing image-to-text cross-
attention [34], random token masking [26], attentive token masking [63], and
masked self-distillation [14]. Our work, FocusCLIP, explores another approach
for improving fine-grained alignment between image and text by introducing an
ROI heatmap during pretraining, explicitly guiding the focus toward task-
relevant areas. This novel approach adds a degree of supervision to the self-
supervised CLIP pretraining and enhances the alignment between text and visual
data, boosting performance on specialized tasks.

Attention using Heatmaps Many VLMs develop an over-reliance on superficial
language priors [46, 38, 43]. Additionally, vision networks often focus on
image areas that do not correlate with the regions humans look at when
performing the same tasks [45, 12]. To address this, HINT [46] used human-
generated attention heatmaps to guide model focus. These heatmaps served as
explicit hints, showing which parts of an image humans considered essential
for the task. The technique effectively improved the model performance on
several tasks by providing a more reliable basis for grounding the
predictions. Similarly, [42] also uses heatmaps of human gaze to highlight
image regions humans deemed meaningful for bird classification. Notably, this
use of heatmaps is distinct from the attention mechanisms commonly found in
Transformer models [56]. Using explicit human guidance to tune the model’s
focus has been similarly explored in various Visual Question Answering studies
[29, 41]. Our work introduces a conceptually similar attention map through the
heatmap input. Integrating it into the CLIP framework sets our work apart,
broadening applicability to multiple zero-shot tasks. Furthermore, unlike
[46], where heatmaps always need to be created manually, our proposed method
does not require additional manual effort by human annotators in cases where
keypoint annotations are already available.

Large Language Models (LLMs) as Annotators In [50], the authors explored the
possibility of leveraging few-shot learning abilities of LLMs [5, 2, 11, 53,
54, 37, 44] to generate new text datasets. Concurrently, other works
demonstrated their capability to automate laborious annotation tasks [52, 25].
Recently, [33] introduced a few-shot prompting strategy for writing natural
language descriptions for images of animals and birds using LLMs. While [33]
did not provide a framework for validating the text quality, they successfully
used LLM-text to improve model performance. Drawing on this, we develop a
prompting strategy in Sec. 3.4 for effectively describing images.

##  3 Method

Figure 2: FocusCLIP pretraining adds ROI heatmaps as a third input to the CLIP
framework [40], highlighting important areas for target tasks. It is trained
with a dual contrastive loss to learn a joint embedding space between the raw
image, heatmap-highlighted image, and text. The text consists of pose
descriptions for people in images written by an LLM, which uses annotations
from the dataset.

In this section, we introduce FocusCLIP, designed to enhance human-centric
task performance by connecting visual and textual information with an emphasis
on human subjects. Our method, depicted in Fig. 2, is differentiated by its
use of a Region of Interest (ROI) heatmap, a novel addition to CLIP that
focuses on human figures within images to guide the learning process. For
human-centric tasks, FocusCLIP pretraining uses the MPII Human Pose dataset
[1], which provides images and detailed keypoint annotations. We procedurally
generate ROI heatmaps from these keypoints, highlighting the human figures. We
also employ LLMs to create detailed descriptions of human pose and activity.

The following subsections describe the methodology. We start with the formal
problem definition in Sec. 3.1 and explain our encoders in Sec. 3.2. Next, we
describe the heatmap generation in Sec. 3.3 and the pose description
generation in Sec. 3.4. Finally, we conclude with our pretraining method in
Sec. 3.5.

###  3.1 Problem Formulation

Let $\mathcal{D}=\\{(I_{i},T_{i},H_{i})\mid 1\leq i\leq N\\}$ be our training
dataset, containing $N$ samples. In this dataset, $I_{i}$ represents the input
image, $T_{i}$ denotes the corresponding textual description of the human
pose, and $H_{i}$ is the associated ROI heatmap for each $i$-th sample. Let
$\phi_{I}:I\mapsto E_{I}$, $\phi_{H}:H\mapsto E_{H}$, and $\phi_{T}:T\mapsto
E_{T}$ be encoding functions that map the image, heatmap, and text,
respectively, into an embedding space. The objective is to learn a function
$f:(E_{I},E_{H})\mapsto E_{T}$ that maps the visual embeddings $E_{I}$ and
$E_{H}$ to the textual embedding $E_{T}$. This mapping aims to minimize a
contrastive loss function $\mathcal{L}$, that quantifies the distance between
matched and unmatched $(E_{I},E_{H},E_{T})$ triplets.

During zero-shot prediction, the goal is to apply the learned function $f$ to
predict the class label $\hat{c}$ for a new image $\hat{I}$ without the aid of
the heatmap. This is achieved by defining a set of potential class labels
$\mathcal{C}=\\{c_{1},c_{2},\ldots,c_{K}\\}$ and generating a corresponding
set of texts $\mathcal{T}=\\{T_{1},T_{2},\ldots,T_{K}\\}$ through a task-
specific sentence template populated with each class label. These texts are
then encoded into embeddings by $f$, producing
$\mathcal{Z}=\\{E_{T_{1}},E_{T_{2}},\ldots,E_{T_{K}}\\}$. Concurrently, the
embedding of the new image $E_{\hat{I}}$ is obtained. The class label $c$
associated with the text embedding $E_{T_{c}}$ that has the highest cosine
similarity with $E_{\hat{I}}$ is predicted as the class for $\hat{I}$,
formalized as:

$\hat{c}=\operatorname*{argmax}_{c\in\mathcal{C}}\left(\text{s}(E_{\hat{I}},E_{T_{c}})\right)$
(1)

The performance of the function $f$ is measured on an unseen test dataset
$\mathcal{D}_{\text{test}}$, which examines the model’s ability to generalize
and accurately predict classes not provided during training.

###  3.2 Encoders

FocusCLIP integrates three encoders: a visual encoder, a text encoder, and a
Region of Interest (ROI) encoder. The visual and text encoders are analogous
to those in the original CLIP architecture, transforming images ($I$) and text
($T$) into corresponding embeddings in a shared space.

The ROI encoder, denoted as $\phi_{R}$, generates embeddings from masked
images, highlighting salient regions for downstream tasks. The masked image,
$I_{H}$, is formed by the Hadamard product (element-wise multiplication) of
the input image, $I$, and a scaled ROI heatmap, $H\in[0,1]$:

$E_{H}=\phi_{R}(I\odot H)$ (2)

This masking acts as an attention mechanism, selectively enhancing image
features corresponding to the regions of interest, such as human figures.

(a) Heatmap Generation Process.

(b) Heatmap Examples.

Figure 3: The process of generating person heatmaps from keypoints involves
grouping keypoints into body parts, encircling them with ellipses, and
blending these elements to produce comprehensive heatmaps, as shown in (a).
The technique effectively highlights individuals within diverse environments,
illustrated by the heatmap examples in (b).

###  3.3 Heatmaps

The procedural algorithm for generating heatmaps uses the MPII Human Pose
dataset’s keypoint annotations, provided as (x, y) coordinates and visibility
labels, to produce heatmaps that emphasize human figures. An empty mask is
initialized with dimensions $H\times W$, matching the input image. For each
person $p$, visible keypoints $K_{visible}$ are used to define a bounding
ellipse $E_{p}$ with center $C_{p}$ and axes $A_{p}$. A Gaussian distribution
$G_{p}$ is computed around $C_{p}$:

$G_{p}(x,y)=e^{-(x-x_{0})^{2}/2\sigma_{x}^{2}-(y-y_{0})^{2}/2\sigma_{y}^{2}}$
(3)

where $(x_{0},y_{0})$ and $(\sigma_{x},\sigma_{y})$ are derived from $C_{p}$
and $A_{p}$, respectively. The initial heatmap for person $p$, $H_{p}$,
combines the bounding ellipse and Gaussian distribution:

$H_{p}=E_{p}\times G_{p}$ (4)

To refine the elliptical heatmap, sub-masks for individual body parts are
created by repeating Eq. 3 and (4) for subsets of keypoints, as depicted in
Fig. 3(a). The heatmaps for each body part are then added together, and the
resulting heatmap is clipped to a range of [0, 1] to produce the final
heatmap, denoted $H$. This yields heatmaps that outline the shape of each
individual (Fig. 3(b)), facilitating focused attention by the model during
pretraining.

While specific to human figures, our approach provides a foundation for
exploring more generalized automated solutions, as discussed in our ablations.

###  3.4 Pose Descriptions

We propose a single-shot prompting strategy for LLMs, illustrated in Fig. 2
(right), to generate pose descriptions of people from keypoints datasets. We
used insights from prompt engineering works [58, 28, 32] to design a generic
template prompt. This prompt can describe images in pre-annotated datasets
using class labels and additional image-specific attributes.

The opening sentence of the prompt establishes the LLM role:

> You are an experienced {role}, with a deep understanding of {dataset name}
> dataset, which has {dataset description}.

This is inspired from the persona patterns in [58, 48, 47]. In our case, the
role is “human pose estimation expert”, dataset name is “MPII Human Pose”, and
the description includes an explanation of available keypoints and other
annotations. Including this effectively morphs the LLM into a knowledgeable
entity about the dataset. Inspired by the experiments in [32], we interface
this persona specification with LLM system prompt to further enhance context-
aware responsiveness.

Next, we define the required content in the generated descriptions, including
a captioning objective for additional context.

> Given the following annotations of an image from the {dataset name} dataset,
> describe the {target} in the image in terms of {required content}, and any
> other discriminatory attributes necessary for {captioning objective}:

This is followed by the raw image-specific attributes from the dataset
annotations as key-value pairs. We use the activity label, people count,
keypoint locations, visibility, center, and scale for each annotated person.
This is our main difference from previous works [33, 32] who harness the few-
shot learning capabilities of LLMs to generate data, providing a handful of
examples and prompting LLMs to synthesize analogous data. In our approach, we
instead leverage the well-documented ability of LLMs to parse structured data
[27, 20].

The next section of our prompt defines the required response format. In our
experiments, we ask the LLM to fill in a semi-standard template: Your response
should {response format}

> “There are [num2word($count)] people in image who are [getVerb($activity)
> parseName($activity)]. [General attributes describing $activity in keypoints
> context.]” For each person in image: “The [parseLocation($center,$scale)]
> person is [predictStateFromContext()] with their [limb]…”

This syntax effectively acts as semantic tagging, guiding the LLM to treat
certain portions of the JSON string as specialized entities. This makes the
LLM "invoke" these tags as if they were functions, thereby facilitating a more
effective and nuanced parsing of the annotations, as corroborated by our
ablation study in Tab. 4(b).

Lastly, we reinforce the persona by repeating it and informing the LLM about
the intended usage of the generated data. We also include any response
restrictions, which tell LLMs the kind of language to use.

> Draw on your professional expertise as a {role}, image-specific features
> mentioned in the annotation, general facts known about the {target}, and any
> other relevant knowledge that can be used to teach {task}. {response
> restrictions}.

Appendix Sec. B provides complete prompts and further details. We also provide
evaluations to quantify the quality of the generated pose descriptions in Sec.
4.2 and use human feedback to assess their correctness in Appendix Sec. A.4.

###  3.5 Contrastive Pretraining

FocusCLIP is pretrained with a contrastive loss that aligns visual and textual
representations in a shared embedding space. To accomplish this, we utilize
dual contrastive losses: one aligning the image embeddings with text
embeddings and another aligning the embeddings of the heatmap-highlighted
images with the text embeddings. The objective is to bring the representations
of corresponding pairs closer together while distancing non-corresponding
pairs.

The loss function for each pair is defined using the Normalized Temperature-
Scaled Cross-Entropy (NTXent) loss, a common choice in contrastive learning
scenarios due to its effectiveness in previous studies like SimCLR [10]. The
NTXent loss for a single positive pair within a batch is given by:

$l(V,T)=-\log\frac{e^{\text{s}(E_{V},E_{T})/\tau}}{\sum_{k=1}^{2N}\mathbb{1}_{[k\neq
i]}e^{\text{s}(z_{i},z_{k})/\tau}}$ (5)

where $\text{s}(u,v)=u^{\top}v/(\|u\|\|v\|)$ is the cosine similarity, $E_{V}$
is the embedding of the i-th image or heatmap-highlighted image, $E_{T}$ is
the corresponding text embedding, $N$ is the batch size, $\tau$ is the
temperature parameter, and $\mathbb{1}_{[k\neq i]}$ is the indicator function
equal to 1 iff $k\neq i$.

Our total loss function, $\mathcal{L}$, is the sum of two NTXent losses—one
for the original images and one for the heatmap-augmented images:

$\mathcal{L}=\frac{1}{2N}\sum_{(I,T,H)\in\mathcal{D}}[l(I,T)+l(I_{H},T)]$ (6)

This combined loss function facilitates learning a robust embedding space that
captures both the global context of images and the localized information
highlighted by the heatmaps.

##  4 Experiments

We perform two sets of experiments: one to assess our FocusCLIP model in zero-
shot classification on human-centric tasks, and another to optimize our pose
descriptions dataset. We also demonstrate its applicability in non-human
domains by evaluating it on the bird classification task.

Implementation details. We initialize the visual encoder with ImageNet pre-
trained weights and freeze it, in line with recommendations by [51, 66]. Our
models are trained using a contrastive loss, with a fixed temperature value of
0.5, consistent with the approach used in SimCLR [10]. We utilize a Stochastic
Gradient Descent (SGD) optimizer, with a learning rate of 0.001 and momentum
of 0.9, and train all models for 64 epochs. This process takes approximately
12 hours on an A100-40GB GPU. For all our experiments, we use ViT-B/16 [15]
for visual and ROI encoders and BERT-Base [13] for the textual encoder.

###  4.1 Evaluation on human-centric tasks

In this set of experiments, we evaluate the zero-shot classification abilities
of FocusCLIP, which was trained on MPII Human Pose dataset [1] images and our
pose descriptions. We apply the model to three human-centric tasks, using five
previously unseen datasets. Our main baseline is vanilla CLIP training [40]
from scratch on the same image-text pairs as FocusCLIP for a fair comparison.
We also report a lower bound for accuracy by simulating a random guess
averaged over three runs.

Tables 1 and 2 present the quantitative and qualitative results of our
experiments. We report the top-k accuracy for image-based activity
classification, age classification, and emotion recognition. For datasets with
numerical age labels, we categorize the age into groups such as adult,
teenager, kid, etc., to transform it into a classification task.

Table 1: Comparing FocusCLIP with baselines for human-centric tasks. Our method improves the CLIP baseline by 8.61 points on the top-k accuracy metric for three zero-shot classification tasks covering five unique datasets. Best values are bold and our method is highlighted. Average is reported across task categories Task | Baselines | Ours
---|---|---
Category | Dataset | # Classes | k | Random | CLIP | FocusCLIP
Activity |  Stanford40 [64] | 40 | 3 | 8.24 | 6.49 | 10.47
Age Classification |  Emotic [23] | 3 | 1 | 30.54 | 37.56 | 41.80
LAGENDA-Body [24] | 3 | 1 | 30.46 | 39.48 | 59.44
LAGENDA-Face [24] | 3 | 1 | 33.13 | 44.56 | 71.41
UTKFace [68] | 5 | 1 | 20.86 | 27.02 | 35.13
Emotion Recognition |  Emotic [23] | 26 | 3 | 11.54 | 10.39 | 13.73
FER+ [3] | 8 | 3 | 36.79 | 52.56 | 63.35
Task-wise Mean | 20.88 | 25.04 | 33.65
Table 2: Qualitative comparison of FocusCLIP with a baseline trained without ROI heatmaps (HM). We randomly select images from the test datasets and show the top three predicted classes by both models. The correct label is highlighted in  bright purple . A prediction is correct if the purple bar appears at the top. The model without heatmaps either fails to predict the correct class (highlighted in red) or has lower confidence than the model with heatmaps | Activity Recognition | Age Classification | Emotion Recognition
---|---|---|---
w/o HM |  |  cutting treesthrowing frisbywalking the dog |  |  teenageradultkid |  |  disconnectionfearpain
with HM |  throwing frisbycutting treeswalking the dog |  adultteenagerkid |  disconnectionconfusionsympathy
w/o HM |  |  feeding a horseriding a horseriding a bike |  |  adultteenagerkid |  |  happinesssadnessdisgust
with HM |  riding a horsefeeding a horsewalking the dog |  adultteenagerkid |  sadnesshappinessneutral

FocusCLIP outperforms the baseline CLIP by 3.98% with 10.47% accuracy on the
Stanford40 [64] dataset for activity recognition. Similarly, significant
improvements are observed across various age classification datasets,
including Emotic [23] and LAGENDA-Body [24], which contain full body images,
and LAGENDA-Face [24] and UTKFace [68], which contain cropped facial images.
The improvements range from 4.24% to 26.85%. In the emotion recognition task,
there is a similar trend with 3.34% improvement for Emotic [23] and 10.79%
improvement for FER+ [3]. The improvements in zero-shot classification
performance across diverse human-centric tasks illustrate FocusCLIP’s enhanced
ability to understand and interpret human-centric features more effectively
than the CLIP baseline. Notably, the improvements in age classification across
various datasets highlight its capacity to grasp implicit human attributes,
and significant gains in emotion recognition tasks emphasize its proficiency
in identifying emotional cues from both body language (Emotic) and facial
expressions (FER+).

Furthermore, FocusCLIP’s performance in the activity recognition task
underscores its capability to contextualize human actions within a scene,
suggesting that the model can extract meaningful information from both the
foreground and background. As we will see in our ablation studies, heatmaps
focusing on background objects in addition to human subjects can further
improve performance for this task. This supports our claim that heatmaps are a
versatile and effective way to guide model focus depending on the specific
downstream task.

These results suggest that combining pose descriptions with heatmap-based
guidance in FocusCLIP effectively directs the model’s focus toward relevant
features within an image, enabling it to make more accurate predictions. This
highlights the potential of specialized training for models that not only
perform well in narrow, task-specific benchmarks but also exhibit a broader
understanding of human-centric concepts in a more generalized, zero-shot
context.

LLaMA-2GPT-3.5GPT-4$50$$60$$70$$80$$90$CLIP ScoreHuman Score (a) Image-Text
Correlation Metrics. GPT-4’s pose descriptions correlate more with images, as
evidenced by automated CLIP Score and human feedback.

LLaMA-2GPT-3.5GPT-4$0.2$$0.4$$0.6$$0.8$$1$$1.2$ReadabilityDiversityRepetition
(Inv) (b) Text Quality Metrics. GPT-4 surpasses human-written text in
diversity and readability while maintaining a low repetition rate.

|  There are two people in the image who are playing music, specifically
drums, while sitting.
---|---
|  The left leg seems to be bearing most of their weight, as it’s straightened
and firmly planted on the ground.
|  The person’s right arm seems to be extended outwards and slightly
downwards, possibly holding onto a tool such as a shovel handle.
|  Their left arm is extended overhead, with their hand holding a tool or
brush near the top of the wall.
(c) Sample pose descriptions reflect the LLM’s nuanced understanding of
activities and contextual relevance, offering precise interpretations of body
language without seeing the image. Additional examples are provided in the
supplementary materials.

Figure 5: Comprehensive analysis of pose descriptions. Table 3: Evaluating pose descriptions with quality metrics and human feedback. *Using shorter response length and prompt without personas or structured syntax. $\dagger$ CLIP Score is influenced by the 77-token limit, favoring shorter captions |  Readability $\uparrow$ |  Errors $\downarrow$ |  Diversity $\uparrow$ |  Repetition $\downarrow$ |  Correlation $\uparrow$ |  Correctness $\uparrow$
---|---|---|---|---|---|---
arXiv | 14.47 | 7.75 | 118.73 | 0.16 | 9.54 | -
GPT-3.5∗ | 5.61 | 1.32 | 41.85 | 0.40 |  18.07$\dagger$ | 74.8
LLaMA-2 | 7.52 | 2.45 | 44.37 | 0.57 | 17.28 | 72.0
GPT-3.5 | 7.63 | 2.79 | 69.23 | 0.36 | 17.46 | 77.6
GPT-4 | 10.29 | 2.01 | 138.68 | 0.17 | 17.97 | 82.6

ReadabilityDiversityRepetitionCorrelationUnrefined PromptOur Prompt (a) Using
contextual cues and personas in our prompt enhances vocabulary diversity,
reduces repetition, increases language readability, and enhances image
correlation for generated text. This is studied in more detail in Sec. B.1

Model |  Auxiliary Information | ZSL
---|---|---
T1
GloVe [39] |  Wiki | 29.0
LongFormer [4] |  Wiki | 22.6
MPNet [49] |  Wiki | 25.8
CLIP |  1-LLM | 28.59
FocusCLIP |  1-LLM + HM (ours) | 31.06
(b) Bird Classification. FocusCLIP pretrained on seen classes in CUB dataset
[57] and human-gaze attention heatmaps from [42] outperforms the CLIP baseline
on unseen classes.

Figure 6: Impact of choice of LLM and prompt refinement on pose descriptions
quality

###  4.2 Quality evaluation of pose descriptions dataset

As part of our MPII Human Pose Descriptions dataset quality analysis, we
compared LLM-written descriptions with a baseline of human-written
text–specifically scientific abstracts from arXiv. We used several metrics to
assess the textual quality of the descriptions, including Flesch-Kincaid Grade
Level for readability level, number of grammatical errors for text integrity,
MTLD for lexical diversity, and a 3-gram repetition score for text uniqueness.
Moreover, we used the CLIP Score metric with OpenAI’s CLIP-ViT-L/14 for image-
description correlation. We also designed a human evaluation scheme to
evaluate the "correctness" of generated text, with results shown in Fig. 5(a).
This is described in more detail in Appendix Sec. A.4.

Results in Tab. 3 reveal that GPT-4 generated descriptions not only offer
greater readability and fewer grammatical errors but also display enhanced
linguistic diversity and lower repetition compared to other models. This
suggests GPT-4’s proficient use of an extensive vocabulary and its detailed
understanding of depicted activities, surpassing even human-generated text in
diversity (Fig. 5(b), Tabs. A2 and A3). GPT-4 provides valuable insights in
addition to limb locations and orientations. For instance, when describing a
person playing football with one leg straight and the other bent (Fig. 5(c)),
only GPT-4 reasoned that the person’s weight must be more on the straight leg.
The human evaluation further confirmed GPT-4’s superiority in crafting
contextually relevant and interpretive descriptions, ranking highest among
human evaluators.

###  4.3 Evaluation on non-human-centric tasks

We evaluated our framework for zero-shot bird classification on the CUB
dataset [57], following splits from [59]. CLIP and FocusCLIP are trained on
150 seen classes with GPT-3.5-written image descriptions (see Appendix Sec. D
for details) and human-gaze attention heatmaps [42]. Our results in Fig. 6(b)
indicate that integrating heatmaps improves CLIP by 2.47%. This validates our
claim that the heatmap component adds subject-level guidance, making our
approach adaptable to non-human classification tasks. For comparison, we also
report results for embedding-based methods [39, 4, 49].

Table 4: Ablation studies examining impact of different parts of our work

(a) Ablating over individual components in FocusCLIP, we show that each component is important for performance with heatmap-masked images (MIX) and sharing weights in visual encoders having the most impact ROI | $\mathcal{L}_{R,T}$ | SE | MIX | Activity | Age | Emotion |  Mean
---|---|---|---|---|---|---|---
|  |  |  | 6.49 | 37.16 | 31.48 |  25.04
✓ |  | ✓ | ✓ | 3.47 | 49.28 | 39.68 |  30.81
✓ | ✓ |  | ✓ | 4.66 | 40.96 | 26.68 |  24.10
✓ | ✓ | ✓ |  | 8.06 | 46.32 | 21.52 |  25.30
✓ | ✓ | ✓ | ✓ | 10.47 | 51.94 | 38.54 |  33.65

(b) Ablations over prompting. Generating text with function syntax in prompts improves performance for most tasks Task | Structured | Plain
---|---|---
Activity | 10.47 | 6.18
Age | 51.94 | 45.40
Emotion | 38.54 | 40.32
Mean | 33.65 | 30.63

(c) Ablating over Heatmaps: Evaluation of keypoint-based, DINO, TCL, and bounding-box heatmaps on various tasks Task | Default | Box | DINO | TCL | None
---|---|---|---|---|---
Activity | 10.47 | 3.74 | 13.38 | 3.49 | 6.49
Age | 51.94 | 41.16 | 39.54 | 46.86 | 37.16
Emotion | 38.54 | 32.20 | 35.90 | 35.87 | 31.48
Mean | 33.65 | 25.70 | 29.61 | 28.74 | 25.04

(d) Ablating over LLMs: Performance comparison across different language models with varied parameter count. Task | GPT-4 | GPT-3.5 | LLaMA-2
---|---|---|---
Activity | 10.47 | 7.97 | 4.88
Age | 51.94 | 36.41 | 40.69
Emotion | 38.54 | 38.71 | 39.82
Mean | 33.65 | 27.70 | 28.46

##  5 Ablations studies

The primary components of our work include the ROI encoder and the additional
contrastive loss between ROI and textural components. Furthermore, we multiply
the heatmap with the original image to create a highlighted image before
feeding it into the ROI encoder, and we share the weights between both visual
encoders. We analyze the value of each of these four components in Tab. 4(a).
By removing each component separately and observing the impact on performance,
we determine that all components work together to boost performance. The ROI
encoder with heatmap input and the multiplication operation to highlight image
regions (MIX) played important roles and notably impacted performance (rows 1
and 4). However, the most significant decrease in performance was observed
when the two visual encoders did not share weights (row 3). This was due to
the increased complexity of the learning objective and the doubling of model
parameters, resulting in overfitting on our small training data and inferior
zero-shot performance on unseen datasets. The additional loss function (row 2)
also contributes to the overall performance of our network (row 5).

Impact of the ROI encoder. The primary contribution of our work is the
heatmap-based attention provided by the ROI encoder. The second contribution
is the pose descriptions dataset. As shown in Tab. 4(a), when the model is
trained with our pose descriptions, it achieves a task-wise mean accuracy of
25.04% (row 1). This is 4.16% above the random baseline at 20.88%,
demonstrating the impact of our text dataset. When the person heatmaps are
provided as an auxiliary input, the task-wise mean improves to 33.65% (row 5),
an additional 8.61%. This confirms our initial hypothesis that heatmaps can
provide subject-focused supervision, allowing FocusCLIP to learn feature
representations better suited for the intended downstream tasks.

Ablations over Prompting Method: Fig. 6(a) compares the statistics of
responses generated by two different prompts: the one we proposed, which
includes persona and multiple contextual cues, and an unrefined prompt that
lacks these features. Quantitative ablations in Tab. 4(b) reveal that
structured prompts, which utilize function-like syntax, yield a higher average
performance across various tasks than plain prompts. The results show that our
proposed prompt leads to higher-quality responses regarding language and
correlation with the images. We present detailed qualitative results in Sec.
B.1 illustrating how adding activity labels and persons enhances the
contextual richness of responses and complements the keypoint data. Similarly,
our evaluations in Sec. A.4 validate our hypothesis that embedding image-
specific attributes and other contextual data in the prompt results in more
precise generated descriptions. This can be explained by multiple contextual
cues and image attributes that we add to the prompt. It reduces the
expectations from LLM to extrapolate new information by instead asking it to
parse structured data, mitigating hallucinations. This emphasizes the
significance of prompt engineering when using LLMs as annotators.

Impact of heatmap quality. We compare three heatmap sources: Gaussian ellipses
within object bounding boxes, self-attention maps from a DINO model [6], and
zero-shot heatmaps generated from TCL [8]. Bounding box-based heatmaps, while
less annotation-dependent, lack the shape detail provided by keypoint-based
annotations. DINO-derived maps eliminate manual annotation but offer limited
control over shape or image regions. TCL heatmaps, generated through textual
description, provide flexibility but have low resolution and contrast. As
shown in Tab. 4(c), including any heatmap consistently enhances performance
over a CLIP-only baseline. Our original fine-grained, keypoint-based heatmaps
excel, particularly in age and emotion recognition, achieving the highest
average performance of 33.65. DINO-based heatmaps demonstrate a unique
advantage in activity recognition, scoring 13.38, as they effectively
highlight multiple significant regions within the scene, not just the subject.
This contributes to their overall average score of 29.61. TCL-based heatmaps
also show competitive performance with an average score of 28.74. However,
bounding-box-based heatmaps offer the least improvement, with an average score
of 25.70, reiterating their limitations in capturing object intricacies.
Notably, DINO and TCL-based heatmaps surpass the no-heatmap baseline without
manual annotation. This demonstrates the ability of FocusCLIP to benefit from
heatmaps.

Impact of LLM choice on performance. We examine how different LLMs influence
zero-shot classification performance using our single-shot prompting method,
as summarized in Tab. 4(d). GPT-4-generated captions enable our FocusCLIP
model to achieve the best performance on the MPII dataset, closely followed by
GPT-3.5. Conversely, captions generated by LLaMA-2 lag, likely due to its
fewer parameters hindering the effective parsing of our JSON-formatted
prompts. These observations are consistent with quality evaluations in Fig.
A2.

##  6 Conclusion

This paper introduced FocusCLIP, an enhancement for CLIP pretraining using a
new ROI encoder. This encoder uses heatmaps to help the model focus on key
image areas, improving performance. We also proposed a systematic prompt for
using LLMs to describe annotated images and used the proposed method to create
the MPII Pose Descriptions dataset (Sec. A). This approach provides a
practical way to generate rich and context-aware captions for images,
enhancing the model’s ability to understand and describe visual content.
Finetuning tasks, such as keypoint estimation, stand to gain significantly
from the detailed pose descriptions present in our proposed dataset.
Additionally, there is potential to explore more sophisticated attention-based
fusion strategies for the two visual input streams rather than the current
method of computing the loss twice. Moreover, there is potential for adapting
our approach for non-human tasks. This paper sets the foundation for these
advancements, and we are optimistic about our method’s future iterations and
our dataset’s usage.

Ethical Considerations. We use LLM-generated pose descriptions, but validating
their correctness through human evaluators covered only a small fraction of
the generated text. We introduced gender-neutral language constraints to
reduce the risk of stereotype propagation, but LLMs may still inadvertently
exhibit biases from their training data.

## References

  * [1] Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New benchmark and state of the art analysis. In: CVPR (June 2014)
  * [2] Anil, R., Dai, A.M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al.: Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023)
  * [3] Barsoum, E., Zhang, C., Canton Ferrer, C., Zhang, Z.: Training deep networks for facial expression recognition with crowd-sourced label distribution. In: ACM International Conference on Multimodal Interaction (ICMI) (2016)
  * [4] Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)
  * [5] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NeurIPS 33, 1877–1901 (2020)
  * [6] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the International Conference on Computer Vision (ICCV) (2021)
  * [7] Carrasco, M.: Visual attention: The past 25 years. Vision research 51(13), 1484–1525 (2011)
  * [8] Cha, J., Mun, J., Roh, B.: Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11165–11174 (2023)
  * [9] Chao, W.L., Changpinyo, S., Gong, B., Sha, F.: An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. pp. 52–68. Springer (2016)
  * [10] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [11] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al.: Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022)
  * [12] Das, A., Agrawal, H., Zitnick, L., Parikh, D., Batra, D.: Human attention in visual question answering: Do humans and deep networks look at the same regions? Computer Vision and Image Understanding 163, 90–100 (2017)
  * [13] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [14] Dong, X., Bao, J., Zheng, Y., Zhang, T., Chen, D., Yang, H., Zeng, M., Zhang, W., Yuan, L., Chen, D., et al.: Maskclip: Masked self-distillation advances contrastive language-image pretraining. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10995–11005 (2023)
  * [15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [16] Fang, Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., Cao, Y.: Eva: Exploring the limits of masked visual representation learning at scale. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19358–19369 (2023)
  * [17] Foulsham, T.: Eye movements and their functions in everyday tasks. Eye 29(2), 196–199 (2015)
  * [18] Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., Gao, J.: Vision-language pre-training:: Basics, recent advances, and future trends. Foundations and Trends® in Computer Graphics and Vision 14(3-4), 163–352 (2022)
  * [19] Goyal, P., Caron, M., Lefaudeux, B., Xu, M., Wang, P., Pai, V., Singh, M., Liptchinsky, V., Misra, I., Joulin, A., et al.: Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01988 (2021)
  * [20] Guo, J., Du, L., Liu, H.: Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066 (2023)
  * [21] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y.H., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. In: International conference on machine learning. pp. 4904–4916. PMLR (2021)
  * [22] Khan, M.G.Z.A., Naeem, M.F., Van Gool, L., Stricker, D., Tombari, F., Afzal, M.Z.: Introducing language guidance in prompt-based continual learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11463–11473 (2023)
  * [23] Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotic: Emotions in context dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 61–69 (2017)
  * [24] Kuprashevich, M., Tolstykh, I.: Mivolo: Multi-input transformer for age and gender estimation (2023)
  * [25] Kuzman, T., Mozetic, I., Ljubešic, N.: Chatgpt: Beginning of an end of manual linguistic data annotation? use case of automatic genre identification. ArXiv, abs/2303.03953 (2023)
  * [26] Li, Y., Fan, H., Hu, R., Feichtenhofer, C., He, K.: Scaling language-image pre-training via masking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 23390–23400 (2023)
  * [27] Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 (2023)
  * [28] Liu, Y., Deng, G., Xu, Z., Li, Y., Zheng, Y., Zhang, Y., Zhao, L., Zhang, T., Liu, Y.: Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860 (2023)
  * [29] Lu, J., Xiong, C., Parikh, D., Socher, R.: Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 375–383 (2017)
  * [30] Miao, K., Gokul, A., Singh, R., Petryk, S., Gonzalez, J., Keutzer, K., Darrell, T.: Prior knowledge-guided attention in self-supervised vision transformers. arXiv preprint arXiv:2209.03745 (2022)
  * [31] MMPose Contributors: OpenMMLab Pose Estimation Toolbox and Benchmark (Aug 2020), https://github.com/open-mmlab/mmpose
  * [32] Møller, A.G., Dalsgaard, J.A., Pera, A., Aiello, L.M.: Is a prompt and a few samples all you need? using gpt-4 for data augmentation in low-resource classification tasks. arXiv preprint arXiv:2304.13861 (2023)
  * [33] Naeem, M.F., Khan, M.G.Z.A., Xian, Y., Afzal, M.Z., Stricker, D., Van Gool, L., Tombari, F.: I2mvformer: Large language model generated multi-view document supervision for zero-shot image classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15169–15179 (2023)
  * [34] Naeem, M.F., Xian, Y., Gool, L.V., Tombari, F.: I2dformer: Learning image to document attention for zero-shot image classification. Advances in Neural Information Processing Systems 35, 12283–12294 (2022)
  * [35] Naeem, M.F., Xian, Y., Zhai, X., Hoyer, L., Van Gool, L., Tombari, F.: Silc: Improving vision language pretraining with self-distillation. arXiv preprint arXiv:2310.13355 (2023)
  * [36] Newell, A., Deng, J.: How useful is self-supervised pretraining for visual tasks? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7345–7354 (2020)
  * [37] OpenAI: Gpt-4 technical report (2023)
  * [38] Peng, D., Wei, W., Mao, X.L., Fu, Y., Chen, D.: An empirical study on the language modal in visual question answering. arXiv preprint arXiv:2305.10143 (2023)
  * [39] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532–1543 (2014)
  * [40] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [41] Ramakrishnan, S., Agrawal, A., Lee, S.: Overcoming language priors in visual question answering with adversarial regularization. Advances in Neural Information Processing Systems 31 (2018)
  * [42] Rong, Y., Xu, W., Akata, Z., Kasneci, E.: Human attention in fine-grained classification. In: British Machine Vision Conference (BMVC 2021) (2021)
  * [43] Salin, E., Farah, B., Ayache, S., Favre, B.: Are vision-language transformers learning multimodal representations? a probing perspective. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 11248–11257 (2022)
  * [44] Scao, T.L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A.S., Yvon, F., Gallé, M., et al.: Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 (2022)
  * [45] Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. pp. 618–626 (2017)
  * [46] Selvaraju, R.R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., Parikh, D.: Taking a hint: Leveraging explanations to make vision and language models more grounded. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2591–2600 (2019)
  * [47] Sisaengsuwanchai, K., Nananukul, N., Kejriwal, M.: How does prompt engineering affect chatgpt performance on unsupervised entity resolution? arXiv preprint arXiv:2310.06174 (2023)
  * [48] Sivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S., Wang, Y.: An empirical evaluation of prompting strategies for large language models in zero-shot clinical natural language processing. arXiv preprint arXiv:2309.08008 (2023)
  * [49] Song, K., Tan, X., Qin, T., Lu, J., Liu, T.Y.: Mpnet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems 33, 16857–16867 (2020)
  * [50] Su, H., Kasai, J., Wu, C.H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N.A., et al.: Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 (2022)
  * [51] Sun, Q., Fang, Y., Wu, L., Wang, X., Cao, Y.: Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 (2023)
  * [52] Tekumalla, R., Banda, J.M.: Leveraging large language models and weak supervision for social media data annotation: an evaluation using covid-19 self-reported vaccination tweets. arXiv preprint arXiv:2309.06503 (2023)
  * [53] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * [54] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
  * [55] Trinh, T.H., Luong, M.T., Le, Q.V.: Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv:1906.02940 (2019)
  * [56] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. NeurIPS 30 (2017)
  * [57] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)
  * [58] White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., Schmidt, D.C.: A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023)
  * [59] Xian, Y., Lampert, C.H., Schiele, B., Akata, Z.: Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence 41(9), 2251–2265 (2018)
  * [60] Xu, H., Xie, S., Tan, X.E., Huang, P.Y., Howes, R., Sharma, V., Li, S.W., Ghosh, G., Zettlemoyer, L., Feichtenhofer, C.: Demystifying clip data. arXiv preprint arXiv:2309.16671 (2023)
  * [61] Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, L., Gao, J.: Unified contrastive learning in image-text-label space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19163–19173 (2022)
  * [62] Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., Huang, J.: Vision-language pre-training with triple contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15671–15680 (2022)
  * [63] Yang, Y., Huang, W., Wei, Y., Peng, H., Jiang, X., Jiang, H., Wei, F., Wang, Y., Hu, H., Qiu, L., et al.: Attentive mask clip. arXiv preprint arXiv:2212.08653 (2022)
  * [64] Yao, B., Jiang, X., Khosla, A., Lin, A.L., Guibas, L., Fei-Fei, L.: Human action recognition by learning bases of action attributes and parts. In: ICCV. pp. 1331–1338. IEEE (2011)
  * [65] Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343 (2023)
  * [66] Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., Beyer, L.: Lit: Zero-shot transfer with locked-image text tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18123–18133 (2022)
  * [67] Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang, L., Choi, Y., Gao, J.: Vinvl: Revisiting visual representations in vision-language models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5579–5588 (2021)
  * [68] Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5810–5818 (2017)

FocusCLIP: Multimodal Subject-Level Guidance for Zero-Shot Transfer in Human-
Centric Tasks Supplementary Material

Muhammad Saif Ullah Khan Muhammad Ferjad Naeem Federico Tombari Luc Van Gool
Didier Stricker Muhammad Zeshan Afzal

In this supplementary, we perform more in-depth quantitative and qualitative
studies to justify our design choices. In particular, we ablate over LLM-
prompting strategy by studying the impact of major prompt components.
Moreover, we describe in detail the composition and suggested use cases of the
pose descriptions dataset introduced in the paper, and show some data samples.
We also describe our framework for evaluating the quality of generated data
using statistical measures and human feedback. Next, we provide some useful
details about our inference method and model hyperparameters to help
researchers reproduce our results. We conclude the supplementary materials
with a brief discussion of how our work can be adapted for non-human tasks.

##  A The MPII Pose Descriptions Dataset

This section describes the pose descriptions dataset created in this paper,
including sample data and results of qualitative ablations.

The data is derived from the MPII Human Pose [1] dataset, using the LLM-
prompting method described in the paper. For each annotated image, we provide
up to four pose descriptions, each written by a different LLM. These natural
language descriptions enable fine-grained understanding and analysis of human
poses, activities, and object interactions within the images.

###  A.1 Selected LLMs

For the dataset creation, we used three state-of-the-art Large Language Models
(LLMs) available at the time of dataset creation. These comprised OpenAI’s
GPT-3.5-Turbo and GPT-4, along with the open-source LLaMA-2 model from Meta.
We based our selection on the reported performance of the models and their
parameters that vary in scale: GPT-3.5-Turbo with 375 billion parameters,
GPT-4 with 1.76 trillion parameters, and LLaMA-2 with 70 billion parameters.
The rationale behind selecting these particular models was to leverage their
diverse architectural strengths and parameter scales, ensuring a robust
comparison across varying capacities for language comprehension and
generation.

For OpenAI models, we used the chat completion endpoint of OpenAI’s API.
Specifically, we used the gpt-3.5-turbo-0613 and gpt-4-0613 models. For
LLaMA-2, we used the llama-2-70b-chat-hf model with HuggingFace’s Inference
API. We also used an older GPT-3.5 release, gpt-3.5-turbo-0301 with an
unrefined version of the prompt from the paper. This model is denoted as
GPT-3.5∗ in the following sections and is used as one of the baselines for
comparison.

###  A.2 Dataset Structure

The dataset is provided in JSON format, following the same structure and
splits as the MPII
annotations111https://download.openmmlab.com/mmpose/datasets/mpii_annotations.tar
available in the MMPose library [31]. It consists of 14644 training and 2723
validation samples. We add a description field for each sample, containing the
generated pose description. Separate training and validation files are
available for each LLM. The complete train set is captioned for each LLM.
However, descriptions for the validation set using GPT-4 are not provided.

Table A1: MPII Pose Descriptions Dataset contains captions written by various LLMs, which will be made accessible through the HuggingFace Datasets API using the corresponding configuration names. Descriptions for the complete training set of MPII [1] are provided for each LLM. The validation set descriptions for GPT-4 are not available. LLM | Config | Train | Val
---|---|---|---
gpt-3.5-turbo-0301 | gpt-3.5-turbo-legacy | ✓ | ✓
gpt-3.5-turbo-0613 | gpt-3.5-turbo | ✓ | ✓
gpt-4-0613 | gpt-4 | ✓ | -
LLaMA-2-70b-chat-hf | llama-2 | ✓ | ✓

###  A.3 Accessing the Dataset

The dataset will be made publicly available through HuggingFace Datasets. Pose
descriptions by a particular LLM will be accessible using the appropriate
config and split names from Table A1.

Figure A1: Human Evaluation UI. Evaluators chose the correctness level of an
image paired with a sentence from its caption.

###  A.4 Human Evaluation

We conducted a human evaluation study using LabelStudio as our annotation
framework to more rigorously assess the quality of the AI-generated captions.

Framework and Methodology We utilized LabelStudio, a versatile data-labeling
framework, to facilitate the evaluation process. We randomly selected 100
images from the original 14,644 training images for this evaluation. Each
image was annotated with captions generated by four different Language
Learning Models (LLMs): GPT-3.5∗, GPT-3.5, GPT-4, and LLaMA-2. Each caption
comprises a variable number of sentences to describe the respective image.

The evaluation was carried out by human volunteers who were presented with a
straightforward user interface, as shown in Figure A1. For each image,
evaluators were shown a sentence extracted from the caption generated by one
of the LLMs. They were then asked to rate the caption’s correctness on a scale
of 1-5, with one being entirely inaccurate and five being completely accurate.
The results of this evaluation are illustrated in Fig. A2. Some sample
sentences in the pose descriptions dataset written by different LLMs are shown
in Tables A2 and A3 together with evaluation scores. Furthermore, in Fig. A4,
we take a closer look at one sample with assigned correctness scores by
evaluators.

GPT-3.5LLaMA-2GPT-4$20$$40$$60$WrongPartly WrongNeutralMostly CorrectCorrect Figure A2: Human evaluation of LLM outputs on a scale of 1 (completely wrong) to 5 (perfect). GPT-4 shows the greatest difference between perfect and completely wrong sentences, indicating that it is the best LLM for describing images. Table A2: Examples of GPT output. We show randomly-selected sentences from descriptive paragraphs GPT-3.5 and GPT-4 wrote for each image. These image-sentence pairs were given to human evaluators who rated them on a five-step scale from completely wrong (1) to perfect (5), with three intermediate steps  2  3  4 . The evaluators marked the largest proportion of GPT-4 outputs as "correct", with only $\leq 15\%$ of GPT-4 sentences partially or completely incorrect. GPT-3.5 | GPT-4
---|---
|  |  There are two people in the image who are playing music, specifically drums, while sitting.
---|---
|  They have their right arm bent at the elbow, holding a paintbrush near
their head.
|  These limb positions suggest that this person might be preparing or
executing a shot during their racquetball game.
|  The torso appears to be slightly twisted to the right side while
maintaining balance during climbing.
|  The right person is also standing with their left arm slightly bent and
holding onto a fishing rod with their left hand.
|  Overall, these individuals exhibit focused posture while playing music,
leaning slightly forward to engage with their instruments and maintain balance
during sitting position
|  There are 2 people in the image who are playing doubles tennis.
|  |  The person’s right arm seems to be extended outwards and slightly downwards, possibly holding onto a tool such as a shovel handle.
---|---
|  Their right arm is extended downwards with elbow bent at an angle greater
than 90 degrees, suggesting that it’s possibly resting or pressing against
something - likely the double bass.
|  The left leg seems to be bearing most of their weight, as it’s straightened
and firmly planted on the ground.
|  Lastly, their head seems to be tilted downwards possibly focusing on
maintaining balance during this complex exercise routine.
|  The centrally located person is engaged in an activity with their limbs
positioned in a way that suggests they are interacting with objects on a
surface like a kitchen counter.
|  Their torso is inclined forward slightly which is typical for this kind of
activity where power from the upper body needs to be transferred down through
the legs into the ground.
Table A3: Examples of LLaMA-2 output. We show randomly selected sentences from descriptive paragraphs LLaMA-2 wrote for each image. These image-sentence pairs were given to human evaluators who found the highest percentage of "incorrect" sentences (red) from LLaMA-2. However, many sentences still "correctly" described images (green). |  Their left arm is extended overhead, with their hand holding a tool or brush near the top of the wall.
---|---
|  The person’s pose suggests that they are in the process of trimming a
branch on the right side of the shrub or tree, and their body is positioned at
an angle to allow them to reach the desired area.
|  In summary, the person in the image is grooming a horse while standing with
a slight lean forward, with their arms extended and partially bent, and their
head looking downwards.
|  The person’s left leg is extended behind them, with their foot pointing
towards the ground.

loading and unloading a truck

GPT-3.5*: There is 1 person in the image who is loading and unloading a truck,
tying down the load, standing, walking, and carrying a heavy load. The person
is standing with their legs slightly apart and their arms extended towards the
truck bed. Their left leg is slightly bent while their right leg is straight.
Their left arm is holding a rope to tie down the load while their right arm is
holding a heavy box. Their torso is slightly twisted towards the truck bed.
Their head is facing forward.

GPT-4: There is one person in the image who is working, specifically truck
driving, loading and unloading a truck, tying down load, standing, walking and
carrying heavy loads. This activity involves a lot of movement and physical
exertion. The centrally located person is actively engaged in work with their
limbs positioned to carry out tasks. Their right leg seems to be bent at the
knee as it’s visible from the hip to the knee but the ankle isn’t visible
suggesting it might be obscured behind an object or positioned towards the
back. The left leg appears straighter with both hip and knee visible but like
the right leg, its ankle isn’t visible either. For their arms; their right arm
appears extended outward with a bend at elbow while holding something given
that both wrist and elbow are higher than shoulder level. Their left arm also
seems to be bent at elbow but unlike right arm, it’s likely hanging downwards
since its wrist is lower than shoulder level. Their torso leans slightly
towards right indicating they might be trying to balance something heavy
they’re lifting or carrying on their right side. Lastly for head position;
since upper neck and head top keypoints are close together compared to other
parts of body suggests they’re looking straight ahead or slightly downwards
which could mean they’re focused on task at hand.

LLaMA-2: There is one person in the image who is occupation, truck driving,
loading and unloading truck, tying down load, standing, walking and carrying
heavy loads. The person is standing with their feet shoulder-width apart, with
their left foot slightly in front of their right foot. Their left leg is bent
at the knee, and their right leg is straight. Their left arm is extended
overhead, and their right arm is bent and holding onto something. Their torso
is straight, and their head is facing forward. The person’s left arm is
positioned above their right shoulder, and their right arm is positioned below
their left shoulder. Their left leg is positioned in front of their right leg,
and their right leg is positioned behind their left leg. The person is wearing
a jacket and pants, and they have a serious expression on their face. They are
in a standing pose, with their weight evenly distributed on both feet. They
are holding a rope in their right hand, and their left hand is on their hip.
The person’s body language suggests that they are focused on their task and
are exerting effort to tie down the load. Their facial expression indicates
that they are concentrating and may be slightly straining from the weight of
the load. Overall, the person in the image is engaged in a physically
demanding activity that requires strength, endurance, and attention to detail.
They are dressed appropriately for the task, and their body language conveys a
sense of determination and focus.

Figure A4: Qualitative Comparison of LLM Output. GPT-4 descriptions are more
descriptive compared to LLaMA-2. They also contain a higher percentage of
sentences rated correct by human evaluators. Furthermore, as corroborated by
the statistical evaluations in the paper, GPT-4 uses a diverse vocabulary with
few repetitions. In comparison, LLaMA-2 output exhibits sentences with a
similar pattern. It is also more often incorrect (i.e., not correlated to the
image), and the contents of its descriptions are sometimes trivial. We also
show the output of GPT-3.5* with the unrefined prompt, which is worse than
GPT-4.

###  A.5 Supported Tasks

The proposed dataset is useful for multimodal learning, fine-grained activity
recognition, and text-enhanced keypoint estimation tasks. It contains detailed
captions that correlate images with text descriptions, providing a rich source
of information about human activities, poses, and individuals involved.

###  A.6 Considerations for Using the Data

Social Impact of Dataset The dataset has the potential to advance research and
applications in various fields such as human pose estimation, activity
recognition, human-computer interaction, sports analytics, healthcare, and
more. Providing rich, descriptive captions generated by multiple LLMs enables
diverse analyses and comparisons of models’ understanding and interpretations
of visual human activities and poses.

Discussion of Biases As the descriptions are generated by various LLMs, biases
inherent in these models could be reflected in the dataset. These biases might
include model-specific understanding or interpretations of activities, poses,
or contexts within the images. Furthermore, the varying degrees of accuracy
across different LLMs could introduce inconsistencies. Users should be aware
of these aspects when utilizing the dataset and consider conducting a detailed
analysis of potential biases for their specific use cases.

Other Known Limitations Some dataset limitations include potential
inaccuracies in captions due to the automated generation process by LLMs.
There may also be disparities in the quality of captions across different
models or for different activities or poses. Users should consider these
factors when using the dataset for training or evaluation purposes.

##  B LLM Prompt

The paper describes the prompt structure, its rationale, and design process.
The specific prompt used for data generation is as follows:

> You are an expert human activity and pose analyzer with deep understanding
> of MPII Human Pose dataset, which has 16 keypoints in order: 0 - right
> ankle, 1 - right knee, 2 - right hip, 3 - left hip, 4 - left knee, 5 - left
> ankle, 6 - pelvis, 7 - thorax, 8 - upper neck, 9 - head top, 10 - right
> wrist, 11 - right elbow, 12 - right shoulder, 13 - left shoulder, 14 - left
> elbow, 15 - left wrist. Given a set of 2D keypoint coordinates from MPII
> dataset as (x,y) with -1 for invisible joints, you will precisely describe
> body poses in terms of relative limb locations. Your descriptions will
> follow this template: “There are [num2word($count)] people in image who are
> [getVerb($activity) parseName($activity)]. [General attributes describing
> $activity in keypoints context.]” For each person in image: “The
> [parseLocation($center,$scale)] person is [predictStateFromContext()] with
> their [limb]…” For each limb (left leg, right leg, left arm, right arm,
> torso, head): “[Describe how these limbs are positioned relative to other
> limbs, bend angles, and other similar pose information.]” Use concise,
> precise, and gender-neutral language.

standing on a rock, holding a sling

There is one person in the image who is standing on a rock, holding a sling.
This activity is characterized by positioning on one’s feet with one arm
extended.

The person in the center of the image is standing, which is evident from the
location of the hips, knees, and ankles. The distribution of weight appears
balanced, as both ankles are planted on the ground.

The right leg of this person is slightly bent at the knee, with the ankle
firmly on the ground, suggesting the leg might be bearing more weight. The
left leg is also slightly bent at the knee, with the ankle grounded too,
providing balance and support.

The torso seems upright with a small inclination to the right, possibly for
stability. The neck and head are erect, suggesting the individual could be
focused on some target.

The right arm is bent at an almost 90-degree angle at the elbow, with the
wrist closer to the waist, implying the person is holding something, perhaps
the sling’s pouch. The left arm is markedly bent at the elbow, with a raised
wrist level, suggesting the person might be reaching out or preparing to
launch something with the sling.

The head is looking straight forward or possibly slightly downward, indicating
focus on the task at hand.

posing for a painting, smiling

There is one person in the image who is posing for a painting, smiling. The
activity is characterized by a relaxed stance, with the person’s body fairly
upright and their expression visibly happy.

The person is centrally positioned in the image. They are in a semi-seated
pose, with their lower half obscured, implying they’re sitting. The torso is
slightly tilted to the right from center.

Their right arm is bent at the elbow, extending down from the shoulder with
the wrist located close to the hip near the pelvis. They may be supporting
themselves in the seated position using their right hand.

The left arm is outstretched away from the torso, forming a slight curve with
a bent elbow. The wrist of their left hand is also at the same level as their
right hand, which suggests it might be resting on a surface like an armrest.

There’s no visible information about the legs, as the keypoints for the knees
and ankles return the "-1" value, hence they may be out of frame or obscured.

The head is straight and upright with the neck-less curved. This, in
conjunction with their activity, indicates that they might be looking directly
towards the viewer, enhancing their smiling pose for the painting.

Figure B5: Pose Description Examples. We use GPT-4 to generate descriptions
for two famous artworks, the Statue of David and the Mona Lisa, to showcase
the versatility of our prompt. The LLM was provided 16 MPII keypoints
(manually annotated) and an activity name (shown under the image). Note: These
two samples are not part of the MPII Pose Descriptions dataset.

Fig. B5 shows two examples generated from keypoint data using our prompt.

###  B.1 Qualitative Ablations for the Prompt

This section systematically evaluates the impact of different components of
our prompt. As described in the paper, our prompt has several essential
elements. We look at the response of the GPT-4 model with different prompt
variations using the reference images in Fig. 6(a) and Fig. 7(a). We manually
labeled the 16 MPII keypoints for each individual in the reference images and
also defined activity labels for each image, which can be found in the
respective caption.

Impact of activity labels. In the paper, we said that supplementing the raw
keypoint coordinates with activity labels enhances the context-richness of the
generated response. In some cases, it also leads to better parsing of the
keypoints. This can be observed by comparing the default description in Fig.
6(b) with the description in Fig. 6(c). The second description was generated
using our complete prompt, with only the activity label missing from the
image-specific attributes we provide as key-value pairs. As seen in Fig. 6(c),
the model response lacks the additional context provided by activity labels,
which are marked bold in Fig. 6(b). This context can enable a better
understanding of both the activity and body posture.

Also, it is interesting to note that the activity label can help the LLM infer
what objects the person might be interacting with. For example, consider the
first image (top row) in Tab. A3. It shows a person painting a wall with their
left arm raised high, similar to the Statue of Liberty. The activity label for
this image is “painting a wall," which produces a widely different (yet
accurate) description: “Their left arm is extended overhead, with their hand
holding a tool or brush near the top of the wall."

(a) The Statue of Liberty is used with the label "holding a torch aloft as an
example to demonstrate the effect of including activity in the prompt.

There is one person in the image who is holding a torch aloft. The person is
depicted in a classic torch-bearer pose, with their arm raised, suggesting a
sense of victory or celebration. The person occupying the central portion of
the image is standing straight with their right arm held high. The right leg
is slightly bent at the knee, with the ankle in front of the right hip,
suggesting a slight forward movement. The left leg is straight and aligned
with the left hip, providing stable support to the body. The right arm is
positioned in an elevated and bent pose, with the wrist being slightly higher
than the shoulder, indicative of holding a torch aloft. The left arm appears
relaxed and slightly bent forward at the elbow, hanging naturally by the side.
The alignment of the pelvis, thorax, and upper neck suggests an upright
posture for the torso. The head is also upright, in line with the neck,
indicating a forward gaze. This overall body posture implies a sense of
determination and focus. (b) Prompt: Complete prompt from Sec. B with an
activity label for the image and keypoint data of the statue.

There is one person in the image who is standing upright. The person occupies
the majority of the center and lower half of the image. The centrally placed
person is standing straight with their limbs showing clear, natural bends and
placements. Their right leg features the knee slightly bent forward,
positioning the right ankle slightly ahead of the right hip. Conversely, their
left leg is straight, with the left ankle is directly beneath the hip. Their
right arm is elevated and bent at the elbow, with the wrist slightly higher
than the shoulder. Conversely, the left arm hangs down naturally, with a
slight forward bend at the elbow that positions the wrist slightly in front of
the shoulder. The person’s torso is upright, as shown by the alignment of the
pelvis, thorax, and upper neck. The head is also aligned with the neck,
suggesting a forward gaze. (c) Prompt: Modified prompt containing only
keypoint data (i.e., no activity label).

Figure B6: Impact of activity labels. We manually labeled the keypoints on the
statue and defined an activity name (a). Using the GPT-4 model, we compare the
pose descriptions generated from both activity labels and keypoint data (b)
with that generated from keypoint data only (c). The additional contextual
information included in the LLM response when the activity label is used is
bolded. These details are absent when the activity label is omitted.

Impact of personas. Another principal component of our prompt (Sec. B) is the
persona specification using a role label. In Fig. B7, we compare the LLM
response with and without role labels specifying the persona. When we ask the
LLM to act as an expert pose analyzer, it makes fewer mistakes, uses more
engaging language, and provides higher-quality insights about the pose.
Compared to this, when directly asked to describe pose without specifying a
role, the LLM focuses on insignificant details (i.e., legs, which are not
important to the activity), writes monotonic sentences, makes more mistakes,
and does not provide useful insights about the interaction.

(a) The Arnolfini Portrait shows a married couple. Using the activity label
"holding hands, indoors," we demonstrate the effect of persona specification
in the LLM prompt when asked to describe this image from keypoints.

There are two people in the image who are indoors, holding hands. Both
individuals appear to be standing upright, facing each other. Their arms are
extended towards each other, indicating the action of holding hands. Their
legs and torsos seem to be in a neutral position, suggesting a relaxed state.
The person on the left side of the image is standing upright with their body
slightly tilted towards the other person. Their right arm is extended towards
the person on the right, with a slight bend at the elbow, indicating they are
holding hands. The left arm is bent at the elbow with the wrist positioned
near the torso. The legs are straight with a slight bend at the knees,
suggesting a relaxed stance. The person on the right side of the image is also
standing upright, with a similar body tilt towards the left. Their left arm is
extended towards the person on the left, mirroring the hand-holding action.
The right arm is bent at the elbow with the wrist positioned near the torso.
The legs are straight with a slight bend at the knees, mirroring the relaxed
stance of the other person. Their heads are slightly inclined towards each
other, suggesting a sense of intimacy or engagement in conversation. The
overall body language suggests a comfortable and relaxed interaction between
the two individuals. (b) Prompt: You are an expert human activity and pose
analyzer with deep understanding of MPII Human Pose dataset, which has 16
keypoints in order …. Given a set of 2D …

There are two people in the image who are holding hands indoors. The
individuals are standing close to each other, and their hands are connected,
indicating a shared activity. The leftmost person is standing upright with
their body slightly tilted towards the right. The right leg is straight, with
the ankle, knee, and hip aligned vertically. The left leg is also straight,
but it is slightly angled towards the right. The torso is upright and slightly
turned to the right. The head is straight, looking forward. The right arm is
bent at the elbow, and the hand is extended towards the other person. The left
arm is slightly bent at the elbow, and the hand is also extended towards the
other person, indicating they are holding hands. The rightmost person is also
standing upright with their body slightly tilted towards the left. The right
leg is straight, with the ankle, knee, and hip aligned vertically. The left
leg is also straight, but it is slightly angled towards the left. The torso is
upright and slightly turned to the left. The head is straight, looking
forward. The right arm is slightly bent at the elbow, and the hand is extended
towards the other person. The left arm is bent at the elbow, and the hand is
also extended towards the other person, indicating they are holding hands. (c)
Prompt: Given a set of 2D keypoints from the MPII Human Pose dataset, which
has 16 keypoints in order …

Figure B7: Impact of personas. We manually defined an activity name and
labeled keypoints for two people in a sample image (a) and compared GPT-4
output using our prompt, which specifies a persona in the first sentence (b)
with a modified prompt omitting the LLM role definition (c). For easier
comparison, we segment the LLM output into three parts: the first talking
about the overall image, the second talking about the person on the left, and
the third talking about the person on the right. The text segments describing
body pose are bold, whereas the text segments drawing insights from the pose
are italic. The incorrect or superfluous pose descriptions are bold-italic.
When we ask the LLM to act as an expert pose analyzer (b), it makes fewer
mistakes, uses more engaging language, and provides higher-quality insights
about the pose. Compared to this, when directly asked to describe pose without
specifying a role (c), the LLM focuses on insignificant details (i.e., legs,
which are not important to the activity), writes monotonic sentences, makes
more mistakes, and does not provide useful insights about the interaction.

##  C Classification Tasks and Model Details

In our experiments, we used the zero-shot classification setup of CLIP, which
is shown in Fig. C8. We populated a task-specific template sentence with all
candidate classes to generate candidate sentences. These sentences were then
passed through the pretrained text encoder to obtain text embeddings.
Similarly, we passed the query image through the pretrained image encoder to
get its image embedding. We used a distance metric to select the text
embedding most similar to the image embedding, and we chose the corresponding
class label as the predicted class. Below, we describe the sentence templates
we used for each task.

Figure C8: Zero-Shot Classification Setup. Given a query image and a set of
possible class labels, a task-specific sentence template is populated by the
class labels and corresponding text embeddings are computed. The query image
is passed through the visual encoder to get the image embeddings. Then, we
compute a similarity score between the image embedding and each text
embeddings and select the class label corresponding to the most similar text
embedding as the predicted class.

###  C.1 Sentence Templates

####  C.1.1 Action Recognition

This task aims to predict the action category from images of people performing
actions. We use the sentence template: “a photo of a person [activity verb-
ing]”.

####  C.1.2 Age Classification

This task classifies people into a discrete age category using the sentence
template: “a photo of a [age group] person”.

####  C.1.3 Emotion Recognition

This task predicts the emotion category from images of cropped faces or
people. For facial emotion, the sentence template “a photo of a/an [emotion
adjective] looking face” is used. For body images, the sentence template “a
photo of a person who is feeling [emotion noun]” is used.

###  C.2 Model Hyperparameters

We provide detailed hyperparameters used to train out models in Tab. C4.

Table C4: FocusCLIP hyperparameters | Attribute | Value
---|---|---
Training Hyperparams | Batch size | 32
Training epochs | 64
Optimizer | SGD
Learning rate | $1\times 10^{-3}$
Momentum | 0.9
Embedding dimension | 512
Temperature | 0.5
Vision Transformer | Variant | ViT-Base
Patch size | 16
Input resolution | 224
Num. layers | 12
Num. heads | 12
Hidden dimension | 768
Text Transformer | Variant | BERT-Base-Uncased
Vocabulary | 30522
Num. layers | 12
Num. heads | 12
Hidden dimension | 768
Sequence length | 512
Perfo- rmance | Parameters (M) | 195
GFLOPs | YY
Inference (ms) | XX

##  D Beyond CLIP and Human-Centric Tasks

In this section, we investigate the potential application of our proposed
approach for non-human tasks and try integrating heatmap-based focus into
other VLMs. In particular, we selected the I2MVFormer [33] model, a recent VLM
for zero-shot learning, and adapted it similarly to how we modified CLIP. We
added a new vision branch that takes in a heatmap-focused image and then
computed the vision-to-text loss twice, once for each vision branch.

We used the CUB dataset [57] for fine-grained bird classification. The dataset
consists of 200 bird species classes, of which 150 are seen during training.
The unseen 50 classes are reserved for zero-shot classification, as
recommended in [59]. We obtained the bird heatmaps from CUB-GHA [42]
highlighting image regions that human annotators found meaningful for
identifying the birds.

In addition to the bird species labels, the CUB dataset provides a rich list
of image-specific attributes. Relying on these auxiliary annotations, we
generated textual descriptions for each image in the dataset using our
proposed LLM-prompting method. We prompted GPT-3.5 as follows:

> You are an experienced ornithologist, a branch of zoology that concerns the
> study of birds, with a deep understanding of the CUB dataset. Given the
> following annotations of an image from the CUB dataset, describe the bird in
> the image in terms of shape/size (use part locations to infer), color,
> unique characteristics or distinct markings, and any other discriminatory
> attributes necessary for bird classification: “attributes”
>
> Your response should be a natural language paragraph starting with “a photo
> of…”. Draw on your professional expertise as an ornithologist, image-
> specific features mentioned in the annotation, general facts known about the
> bird, and any other relevant knowledge that can be used to teach bird
> classification. Use simple, clear, and concise language. Do not include raw
> numbers from the annotation, but you may use them to inform your description
> using words.

Figure D9: Integrating Heatmap-Based Supervision in I2MVFormer. Our heatmap-
component and dual loss formulation improves performance on unseen CUB [57]
classes with I2MVFormer [33] modified similarly to FocusCLIP.

We trained two models using this data: the original I2MVFormer with CUB images
and our LLM-generated text and the modified I2MVFormer with the additional
heatmap input. We tracked the generalized zero-shot learning (GZSL)
performance [9] during training. We noticed a constant improvement of 2-3% on
the 50 unseen classes throughout the training process, with no negative impact
on the seen classes, as shown in Fig. D9. This improvement validates our claim
that heatmap-based supervision can enhance the zero-shot learning capabilities
of vision-language models. This promising direction of research can lead to
further advancements in the field.