# KG-SP: Knowledge Guided Simple Primitives
for Open World Compositional Zero-Shot Learning

Shyamgopal Karthik1 Massimiliano Mancini1 Zeynep Akata1,2
1University of Tübingen 2Max Planck Institute for Intelligent Systems

###### Abstract

The goal of open-world compositional zero-shot learning (OW-CZSL) is to
recognize compositions of state and objects in images, given only a subset of
them during training and no prior on the unseen compositions. In this setting,
models operate on a huge output space, containing all possible state-object
compositions. While previous works tackle the problem by learning embeddings
for the compositions jointly, here we revisit a simple CZSL baseline and
predict the primitives, i.e. states and objects, independently. To ensure that
the model develops primitive-specific features, we equip the state and object
classifiers with separate, non-linear feature extractors. Moreover, we
estimate the feasibility of each composition through external knowledge, using
this prior to remove unfeasible compositions from the output space. Finally,
we propose a new setting, i.e. CZSL under partial supervision (pCZSL), where
either only objects or state labels are available during training, and we can
use our prior to estimate the missing labels. Our model, Knowledge-Guided
Simple Primitives (KG-SP), achieves state of the art in both OW-CZSL and
pCZSL, surpassing most recent competitors even when coupled with semi-
supervised learning techniques. Code available at:
https://github.com/ExplainableML/KG-SP.

##  1 Introduction

As humans, we interact with objects depending on their state. For instance, we
use ripe lemons rather than moldy ones to prepare a lemonade, and we clean
dirty dishes after using them. Algorithms that can recognize objects together
with their state are crucial for autonomous agents to show the same high-level
interactions capabilities we have. In the literature, this problem is studied
under the name of Compositional Zero-shot Learning (CZSL). In CZSL, we are
given a training set with images of objects in a subset of their possible
states and, at test time, the goal is to recognize compositions of the same
set of objects and states, even unseen during training. Since an object has a
different appearance depending on its state (_e.g_. dry dog vs wet dog) and a
state modifies objects in different ways (_e.g_. wet dog vs wet car), the
challenge of CZSL is modeling how states and objects interact with each other,
extrapolating this knowledge from seen to unseen compositions. Under this
perspective, multiple works modeled the interactions of objects and states,
either through compositional classifiers [27, 32], or a shared embedding space
[30, 24, 29].

Figure 1: We consider the problems of open-world CZSL (OW-CZSL), where we lack
priors on unseen compositions at test time, and CZSL under partial supervision
(pCZSL) where we also lack compositional labels during training (left). We
tackle them by independently predicting object (red) and state (blue) labels
and by using external knowledge (bottom) to estimate the feasibility of
compositions, reducing the search space during inference and improving pseudo-
labeling during training in pCZSL.

Despite their effectiveness, [25] showed how the performance of CZSL methods
degrade in the open-world setting (OW-CZSL). In OW-CZSL, there are no priors
on the unseen compositions, and models must consider all possible compositions
at test time. Due to the large cardinality of the output space, it is
difficult to produce discriminative embeddings for the unseen compositions
[25]. Inspired by the findings of [25], in this work we explore a completely
different direction. Specifically, we design an architecture that disregards
the compositional nature of the problem and produces the initial predictions
independently for objects and states. The idea is that while discriminating
between compositions is hard in OW-CZSL due to the large search space,
recognizing primitives (_i.e_. objects and states) in isolation is easier
since 1) the cardinality of the two sets is much lower and 2) the sets are
fixed at both training and test time.

Inspired from [27] and [20], we design a simple method that predicts objects
and states with two independent classifiers. Since recognizing states requires
different features w.r.t. recognizing objects, instead of having a shared
feature representation, we train our model with two different non-linear
feature extractors. Furthermore, since not all compositions are equally
feasible in reality (_e.g_. ripe dog) we can refine the predictions of our
model by eliminating less feasible compositions from the output space. With
this goal, we use external knowledge (_i.e_. ConceptNet [41]) to estimate the
compatibility between a state and an object, using these estimates to remove
less feasible compositions at test time. We name our model Knowledge-Guided
Simple Primitives (KG-SP). As our KG-SP method does not require compositional
labels during training, we explore a new challenging setting, _i.e_. CZSL
under partial supervision (pCZSL). In pCZSL, training samples have either only
object or state annotation, but not both. Here we use our prior on feasible
compositions to aid pseudo-labeling during training. Experiments show that KG-
SP is either competitive or surpasses the current state of the art in OW-CZSL
and outperforms recent CZSL approaches on pCZSL setting. Figure 1 provides an
overview of KG-SP and the two tasks.

Contributions. To summarize, 1) inspired by [27, 20], our model predicts state
and objects independently while at the same time removing less feasible
compositions from the output space based on external contextual information
about the feasibility of certain compositions; 2) we explore the problem of
CZSL under partial supervision, where either object or state information is
missing in the ground-truth; 3) we adapt recent baselines for pCZSL showing
that KG-SP outperforms them even when coupled with semi-supervised learning
techniques in both OW-CZSL and pCZSL settings.

##  2 Related works

Compositional zero-shot learning aims to recognize compositions of states and
objects in images, even unseen during training. The main challenge of this
setting is modeling how states modify objects, generalizing this capability to
unseen compositions. Most of the previous works focused on how to model the
interactions between states and objects either at the parameter level or in a
given representation space. For instance, [27, 32] proposed to generate a
classifier for a given state-object composition given two classifiers (or
embeddings) for specific state and object primitives, using either a
compositional module [27] or a gating network [32]. Differently, [30, 24]
model each state as an operator transforming object embeddings, imposing
properties on the state operators (_e.g_. commutativity, symmetry). In [30]
the state operators are linear, while in [24] they are coupling and decoupling
networks. Recently, [29, 38] used graph convolutional networks [18] to model
the interactions between state, objects and their compositions. Differently,
[2] tackles CZSL from a causality perspective, learning disentangled objects
and states representations. In this work, we revisit VisProd [27], predicting
objects and states in isolation, showing that this strategy is effective in
OW-CZSL. As in [25], we estimate the feasibility of each composition to
improve the model’s performance. However, we use ConceptNet to this aim,
rather than compositional annotations, being the first to tackle CZSL without
compositional labels during training.

Multi-task learning. Since we predict state and object independently, our work
is related to Multi-Task [8, 16, 28, 39, 37] and Multi-Domain learning [34,
35, 23, 4], where the goal is to learn a unique model able to address
different visual tasks. Most of the approaches in this domain either learn
task-specific parameters [34, 35, 23, 4, 39] and how to combine them [28, 37],
or focus on re-weighting different loss functions [16]. While we use multi-
task learning to design primitive classifiers for CZSL, our final goal is
different since we compose predictions from separate output spaces.

Learning from Partial Supervision. Our CZSL setting without compositional
labels is related to semi-supervised learning and learning with missing
labels. In semi-supervised learning, both labeled and unlabeled samples are
available, and the goal is to effectively use the unlabeled samples. Popular
ideas revolve around consistency regularization [6, 5, 40], and self-training
[11, 21, 36]. Unlike semi-supervised learning, in pCZSL, all samples are
labeled, but partially. Thus, we can also exploit the prior on how objects
interact with states to estimate the missing labels.

For what concerns learning with missing labels, this is most prevalent in
multi-label scenarios where it is unfeasible to annotate all labels that are
present in a single image. Approaches in this field usually model the
correlation among labels [10, 13, 19, 9] to impose semantic objectives on
missing ones. While we are also interested in learning from partial
supervision, our labels lie on two separate spaces (_i.e_. objects and
states), and the missing labels (_e.g_. state) influences the appearance of
the positive one (_e.g_. object). In this setting, the main challenge is to
model how the two spaces influence each other without any compositional
supervision.

##  3 Knowledge Guided Simple Primitives

Problem formulation. CZSL [25] aims to recognize compositions of a set of
objects $O$ and a set of states $S$. Formally, we are given a training set
$\mathcal{T}=\\{(x,y)\\}_{i=1}^{N}$, where $N$ is the size of the training
set, $x\in X$ denotes an image in the input space $X$ and $y\in Y_{s}$ is its
label in the set of seen compositions $Y_{s}$. The goal is to learn a model
that can recognize a set of compositions $Y_{t}=Y_{s}\cup Y_{u}$, where
$Y_{u}$ is a set of unseen compositions (_i.e_. $Y_{u}\cap Y_{s}=\emptyset$)
and $Y_{t}\subseteq Y$, with $Y$ being the set of all possible compositions,
_i.e_. $Y=S\times O$.

OW-CZSL and pCZSL settings. In this work, we consider two different CZSL
settings. Open-World CZSL (OW-CZSL) [25] assumes no prior on the set of unseen
compositions at test time. This means that the model needs to operate on the
full compositional space, _i.e_. $Y_{t}=Y$. Consequently, the number of unseen
compositions is much larger than the number of seen ones _i.e_.
$Y_{u}=Y\setminus Y_{s}$, thus the main challenge is operating in a very large
output space where most of the compositions are unseen and thus hard to
discriminate.

In this work, we also consider a new challenging task, namely CZSL under
partial supervision (pCZSL), where the training set does not contain any
compositional label and all training images have either object or state label,
but not both. This setting is more realistic than standard CZSL since most
datasets are collected with single labels (_e.g_. only object-level
information) and collecting multiple labels is expensive and time-consuming.
Formally, we consider the labels of our training set $\mathcal{T}$ to be of
the form $y=(s,\mathtt{u})\vee y=(\mathtt{u},o),\;\forall(x,y)\in\mathcal{T}$,
with $s\in S$, $o\in O$ and $\mathtt{u}$ denoting an unknown label. Note that,
as a consequence of this formulation, the set of training compositions $Y_{s}$
is not known a priori anymore. This implies that, as in OW-CZSL, we need to
consider the full compositional space at test time, _i.e_. $Y_{t}=Y$.
Moreover, since no training image contains both object and state labels, we do
not have explicit supervision on how states modify objects and vice-versa.

In the following, we describe the two components of our framework, Simple
Primitives (SP) where we predict the primitives, e.g. object and states,
independently and Knowledge Guidance (KG) where we use external resources that
guide our model on the feasibility of certain compositions.

###  3.1 Simple Primitives (SP) in KG-SP

Inspired by the early Visual Product (VisProd) baseline [27], our model
completely disregards the compositional nature of the problem and predicts
states and objects independently. This idea contrasts with recent approaches
(_i.e_. [30, 32, 24, 29, 2, 25]), explicitly modeling the interactions between
objects and states within the model.

Formally, given an image $x$, we extract its feature representation
$z=\omega(x)$ through a function $\omega$, mapping images into a feature space
$Z$, _i.e_. $\omega:X\rightarrow Z$. We then have an object classifier
$\phi_{o}:Z\rightarrow\Delta_{O}$ that maps $z$ to a vector in the probability
simplex $\Delta_{O}$, spanning all object categories. Similarly, we have
another classifier that maps $z$ to a probability over the states, _i.e_.
$\phi_{s}:Z\rightarrow\Delta_{S}$. During training, we minimize the cross-
entropy loss for both the object and state predictions. Specifically, we
minimize:

$\displaystyle\mathcal{L}_{\text{visprod}}$
$\displaystyle=\sum_{i=1}^{N}\mathbb{I}_{s_{i}\neq\mathtt{u}}\mathcal{L}_{\text{state}}(x_{i},s_{i})+\mathbb{I}_{o_{i}\neq\mathtt{u}}\mathcal{L}_{\text{obj}}(x_{i},o_{i})$
(1)
$\displaystyle=-\sum_{i=1}^{N}\mathbb{I}_{s_{i}\neq\mathtt{u}}\log\phi_{s}(z_{i},s_{i})+\mathbb{I}_{o_{i}\neq\mathtt{u}}\log\phi_{o}(z_{i},o_{i})$

where $z_{i}=\omega(x_{i})$, $\phi_{o}(z,o)$ is the probability of the object
$o$ assigned by $\phi_{o}$ to the input $z$, and $\phi_{s}(z,s)$ is the
probability of the state $s$ assigned by $\phi_{s}$ to the input $z$. In Eq.
(1), $\mathbb{I}$ is an indicator function used to not compute the loss in
pCZSL, in the absence of primitive labels. Our prediction function is:

$f=\operatorname*{arg\,max}_{(s,o)\in
Y}\,\phi_{o}(w(x),o)\cdot\phi_{s}(w(x),s).$ (2)

Although learning simple primitives independently like this may not be
effective in standard CZSL, we argue that the capability to separate state and
objects predictions is crucial in OW-CZSL, where the search space is too large
if predictions are made over the full compositional space.

Figure 2: Knowledge-Guided Simple Primitives (KG-SP). We train a separate
object (red) and state (blue) predictor on top of a shared feature extractor
(green) using the available state and object labels. We use external knowledge
to estimate feasibility scores of compositions (yellow, bottom), using this
prior during inference to directly remove unfeasible compositions from the
output space. In pCZSL , we use this knowledge to re-weight the class scores
and perform pseudo-labeling (grey) of missing labels, sampling them through
the Gumbel-softmax.

In the original VisProd formulation, objects and state predictors are simple
linear layers operating over the same feature vector. However, this choice is
suboptimal and leads to low results in practice. In fact, by using separate
linear layers, VisProd addresses CZSL as a multi-task learning problem [8, 16,
34, 35, 28, 39, 37], where there are two different tasks (_i.e_. states and
objects prediction) that share the same feature extractor while differing only
for the classification head. However, multiple works in multi-task (MTL) and
multi-domain learning (MDL) discussed how fully sharing the parameters to
extract the feature representation for different tasks (_i.e_. hard-sharing
[8]) is suboptimal when the tasks are not strictly related [34, 35, 28] and
may even lead to negative transfer [22].

In CZSL, recognizing objects is different than recognizing their states.
Specifically, the former requires focusing on global features: for instance,
distinguishing an animal from another requires focusing on their shapes and
skins while distinguishing fruits requires detecting texture-based cues. On
the contrary, recognizing states requires focusing on local patterns: for
instance, the difference between dry and wet can be detected by the presence
of drops in cars and apples while in animals it requires looking at the shape
of the fur. With this premise, we need to overcome the limits imposed by hard-
parameter sharing to ensure the objects and state classifiers have enough
flexibility to learn primitive-specific features. While advanced MTL and MDL
techniques can be used for this purpose, in this work we found that it is
sufficient to implement the two classifiers as multi-layer perceptrons (MLP)
with non-linear activations.

###  3.2 Knowledge Guidance (KG) in KG-SP

In the large output space of OW-CZSL and pCZSL, not all compositions are
equally feasible (_e.g_. ripe dog, hairy apple) and taking this prior into
account can help in correcting incompatible state-object predictions of our
model. In the following we describe how we estimate the feasibility scores and
how we use them in our model.

Estimating feasibility scores. Formally, let us associate to each composition
$(s,o)$ a compatibility score $c_{s}^{o}\in[0,1]$. Since there exist no
database contain such information, previous works exploited the set of seen
compositions $Y_{s}$ to estimate $c_{s}^{o}$[25]. Here we explore an
alternative direction by using external knowledge. In this way, our estimation
is independent of the actual availability of the set $Y_{s}$ and can also be
applied in pCZSL, where $Y_{s}$ is unknown. While we explored different
strategies (see supplementary), we found ConceptNet [41] to give reliable
feasibility estimates.

ConceptNet is a knowledge graph connecting words and phrases with labeled
edges, extracted from various sources [41]. We can use ConceptNet in two ways.
The first is querying for the existence of a composition and the second is
querying for the relatedness between two entries (_i.e_. object and state).
Since direct queries are very sparse, we follow the second approach, defining
the scores as:

$c_{s}^{o}=\rho_{\text{KB}}(s,o)$ (3)

where $\rho_{\text{KB}}(s,o)$ returns the relational score between $s$ and
$o$. In ConceptNet, these scores are computed from the cosine-similarity of
ConceptNet Numberbatch embeddings [42]. The latter are built from ConceptNet
adjacency matrix and existing word embeddings (e.g. word2vec [26], GloVe
[31]).

Using the feasibility scores during inference. Similarly to [25], the most
straightforward-way to use the feasibility sores is to remove from the output
space less feasible compositions during inference. Thus, our prediction
function becomes:

$f_{\text{KG}}=\operatorname*{arg\,max}_{(s,o)\in
Y,c_{s}^{o}>0}\,\phi_{o}(z,o)\cdot\phi_{s}(z,s)$ (4)

where we consider feasible all compositions with $c^{s}_{o}>0$.

Using the feasibility scores during training for pCZSL.  In pCZSL, we may
obtain additional supervision by estimating missing labels. One
straightforward way to achieve this is through pseudo-labeling [21], a semi-
supervised learning technique that takes the model predictions as ground-truth
for unlabeled samples. In pCZSL, this means that, when the state (object)
label is missing, pseudo-labeling will impute as label the object (state)
predicted with the highest score. To avoid that the pseudo-labels form
unfeasible compositions, we can use our prior to aid the pseudo-labeling
process.

Given either an object label $o$ or a state label $s$, we estimate their
respective state and object pseudo-labels as:

$\hat{s}\thicksim\mathtt{Gumbel}\left(\phi_{s}(z)\odot
c^{o}\right),\;\hat{o}\thicksim\mathtt{Gumbel}\left(\phi_{o}(z)\odot
c_{s}\right)$ (5)

where $c^{o}$ is the vector containing the compatibility scores for all states
given the object $o$, _i.e_. $c^{o}=[c_{s}^{o}]_{s\in S}$, and $c_{s}$ is its
counterpart for all objects given the state $s$, _i.e_.
$c_{s}=[c_{s}^{o}]_{o\in O}$111We assume $S$ and $O$ to be alphabetically
ordered.. Note that in both equations we sample the pseudo-labels using
Gumbel-softmax ($\mathtt{Gumbel}$) [15]. We found this choice helpful to make
the model more robust to noisy predictions and less biased toward the training
set latent label distribution. Our objective function becomes:

$\displaystyle\begin{split}\mathcal{L}^{\text{pCZSL}}_{\text{visprod}}=&\sum_{(x_{s},s)\in\mathcal{T}_{s}}\mathcal{L}_{\text{state}}(x_{s},s)+\mathcal{L}_{\text{obj}}(x_{s},\hat{o})\\\
&+\sum_{(x_{o},o)\in\mathcal{T}_{o}}\mathcal{L}_{\text{obj}}(x_{o},o)+\mathcal{L}_{\text{state}}(x_{o},\hat{s}).\end{split}$
(6)

We use this objective in pCZSL during training and Eq.(1) for OW-CZSL. In both
cases we perform inference through Eq.(4). Since we couple independent
primitive prediction with external knowlede to refine them, we name the method
Knowledge-guided Simple Primitives (KG-SP). Figure 2 illustrates our approach
during training and inference.

Method | MIT-States | UT Zappos | C-GQA
---|---|---|---
S | U | HM | AUC | S | U | HM | AUC | S | U | HM | AUC
| TMN [32] | 12.6 | 0.9 | 1.2 | 0.1 | 55.9 | 18.1 | 21.7 | 8.4 | NA | NA | NA | NA
| AoP [30] | 16.6 | 5.7 | 4.7 | 0.7 | 50.9 | 34.2 | 29.4 | 13.7 | NA | NA | NA | NA
| LE+ [27] | 14.2 | 2.5 | 2.7 | 0.3 | 60.4 | 36.5 | 30.5 | 16.3 | 19.2 | 0.7 | 1.0 | 0.08
| VisProd [27] | 20.9 | 5.8 | 5.6 | 0.7 | 54.6 | 42.8 | 36.9 | 19.7 | 24.8 | 1.7 | 2.8 | 0.33
| SymNet [24] | 21.4 | 7.0 | 5.8 | 0.8 | 53.3 | 44.6 | 34.5 | 18.5 | 26.7 | 2.2 | 3.3 | 0.43
| CompCos${}^{\text{CW}}$ [25] | 25.3 | 5.5 | 5.9 | 0.9 | 59.8 | 45.6 | 36.3 | 20.8 | 28.0 | 1.0 | 1.6 | 0.20
| CGE${}_{\text{ff}}$ [29] | 29.6 | 4.0 | 4.9 | 0.7 | 58.8 | 46.5 | 38.0 | 21.5 | 28.3 | 1.3 | 2.2 | 0.30
| CompCos [25] | 25.4 | 10.0 | 8.9 | 1.6 | 59.3 | 46.8 | 36.9 | 21.3 | 28.4 | 1.8 | 2.8 | 0.39
| CGE [29] | 32.4 | 5.1 | 6.0 | 1.0 | 61.7 | 47.7 | 39.0 | 23.1 | 32.7 | 1.8 | 2.9 | 0.47
| KG-SPff | 23.4 | 7.0 | 6.7 | 1.0 | 58.0 | 47.2 | 39.1 | 22.9 | 26.6 | 2.1 | 3.4 | 0.44
| KG-SP | 28.4 | 7.5 | 7.4 | 1.3 | 61.8 | 52.1 | 42.3 | 26.5 | 31.5 | 2.9 | 4.7 | 0.78

Table 1: Open World CZSL results on MIT-States, UT Zappos and C-GQA. We
measure best seen (S) and unseen accuracy (U), best harmonic mean (HM), and
area under the curve (AUC) on the compositions. KG-SPff refers to our proposed
method with a frozen backbone.

##  4 Experiments

Datasets. We use the three standard datasets for Compositional Zero-Shot
Learning, namely UT-Zappos [45, 46], MIT-States [14] and the recently proposed
C-GQA [29] dataset. UT-Zappos contains 12 object categories (shoe types) and
16 state categories (material types), with $83$ seen compositions and a
compositonal space of 192 compositons. MIT States is a larger dataset that
contains 245 object categories in 115 possible states. In total, it contains
$1,262$ seen compositions and an output space of $28,175$ compositions in OW-
CZSL. Finally, C-GQA is a recently proposed dataset222We refer to the updated
split in https://github.com/ExplainableML/czsl. with 674 object categories and
413 state categories. It contains $5,592$ training compositions and a full
compositional space with $278,362$ compositions.

Baselines. In OW-CZSL, we compare KG-SP against standard CZSL approaches;
namely Attributes as Operators (AoP) [30], Label Embed+ (LE+) [27], Task
Modular Networks (TMN) [32], SymNet [24], Compositional Graph Embeddings (CGE)
[29] and Compositional Cosine Logits (CompCos) [25]. In the tables, we refer
to the closed world version of CompCos as CompCos${}^{\text{CW}}$ and the
variant of CGE with a frozen feature extractor as CGEff.

In pCZSL, we compare KG-SP with CGE [29] and CompCos [25], the state-of-the-
art models in the closed and open-world settings respectively. These methods
are adapted to pCZSL by marginalizing their predictions over states/objects,
when the state/object information is available, minimizing the cross-entropy
loss on the ground-truth annotation. We also experiment with popular semi-
supervised learning techniques such as entropy minimization [11] and pseudo-
labeling [21], adding them to both CompCos and CGE.

Evaluation Protocol. For the OW-CZSL setting, we follow the standard splits of
[32, 29], evaluate all the methods on the generalized setting, where the model
recognizes samples from both seen and unseen compositions. Following the
protocol in [32], we apply a bias on the seen compositions at test time,
measuring the performance as best seen (S) and best unseen (U) accuracy, best
harmonic mean (HM) as well as the area under the curve (AUC) by varying the
bias.

For the pCZSL setting, we propose a new split of the training split set,
separating samples with object and state labels. This is done by keeping only
the object label for half the samples, while for the remaining half, only the
state label, ensuring that every object and state is seen in the training
set.Furthermore, for pCZSL setting, the model has no access to seen
compositons $Y_{s}$. Thus, we evaluate the model in the full compositional
space, without subtracting any bias on $Y_{s}$. Therefore, we use as metric
the seen (S) and unseen (U) accuracy, and their harmonic mean (HM), as it is
standard in Zero-Shot Learning [44].

Implementation Details. We follow the standard practices in the CZSL
literature [29, 25], by using a ResNet18 [12] feature extractor. For the state
and object classifiers, we follow [29] and use Multi-Layer Perceptrons with
three layers, comprising Layer Normalization [3] and Dropout [43]. The model
is optimized using Adam [17] with the default hyperparameters, a learning rate
and weight-decay of $5e$-$5$.

###  4.1 Open-World CZSL

The results on the challenging OW-CZSL setting are reported in Table 1. In
this setting KG-SP either outperforms or it is competitive with the state of
the art. Specifically, on UT-Zappos KG-SP outperforms the best competitor
(CGE) in all metrics, with an improvement of 3.4% in AUC (26.5 vs 23.1), 3.3%
in best HM (42.3 vs 39.0) and 4.4% best unseen (52.1 vs 47.7). Similarly,
without end-to-end training, KG-SP${}_{\text{ff}}$ surpasses the best baseline
(CGE${}_{\text{ff}}$) by 1.3 in AUC (22.9 vs 21.5) and by 1.3 in best HM (39.1
vs 38.0).

These results are confirmed in the challenging C-GQA dataset. Despite an
output space of almost $280$k compositions, KG-SP obtains 0.78 AUC vs 0.47 of
the best competitor (CGE), 4.7 HM (vs 3.3 of SymNet) and 2.9 on best unseen
(vs 2.2 of SymNet). When non-finetuned, the method achieves competitive
results w.r.t. SymNet (_e.g_. 3.4 HM) while being much easier to optimize,
since KG-SP does not impose any constraint on the compositional space. Our
results indicate that modeling states and objects independently may be an
effective approach to deal with the very large output space of OW-CZSL. This
independence assumption ensures that each predictor learns a discriminative
classifier over a few hundred classes rather than a single classifier over
thousands of compositions, which does not scale well even with powerful
architectures (_e.g_. graph-convolutional neural network of [29]) and
initialization through side-information (_e.g_. word-embeddings [26]).

Finally, the table also highlights the gap between KG-SP and the VisProd
baseline of [27]. In particular, our revised model (without fine-tuning)
consistently surpasses VisProd in AUC (_i.e_. 1.0 vs 0.7 on MIT-States, 22.8
vs 19.7 in Zappos, 0.44 vs 0.33 in C-GQA) and best harmonic mean (6.7 vs 5.6
on MIT-States, 39.3 vs 36.9 on Zappos and 3.4 vs 2.8 on C-GQA). These results
confirm the importance of our modifications to the original VisProd model, as
we will ablate in the following.

| Marginaliz. | Seen | Unseen | HM | AUC
---|---|---|---|---|---
CGEff |  | 25.5 | 5.7 | 6.5 | 1.0
✓ | 24.0 | 7.8 | 8.1 | 1.3
CGE |  | 27.2 | 6.6 | 7.0 | 1.3
✓ | 25.1 | 8.1 | 8.1 | 1.4
Table 2: OW-CZSL results in the validation set of MIT-States when using marginalization. CGEff is the approach of [29] with frozen backbone whereas CGE performs end-to-end training.  Figure 3: Ablation study on the importance of the depth of the object and state-classifiers for KG-SP${}_{\text{ff}}$ on UT-Zappos (red curve) and MIT-States (blue curve) validation set for the OW-CZSL setting. Performance is measured in AUC. | Mask | Seen | Unseen | HM | AUC
---|---|---|---|---|---
VisProd |  | 24.8 | 6.8 | 7.3 | 1.1
✓ | 24.7 | 7.2 | 7.6 | 1.2
KG-SPff |  | 26.3 | 7.4 | 7.9 | 1.3
✓ | 26.5 | 7.7 | 8.2 | 1.4
Table 3: OW-CZSL results on MIT states validation set while applying our
feasibility-based masks ($f_{\text{KG}}$) on different models.

####  4.1.1 Why KG-SP works in OW-CZSL?

Separately predicting objects and states. We argue that an important reason
for the success of KG-SP is the separate treatment of states and objects, As
stated in previous discussion, predicting state and objects independently
makes the OW-CZSL problem easier and more scalable w.r.t. predicting directly
over thousands of compositions. To verify this hypothesis, we take the state-
of-the-art method in standard CZSL, CGE [29] (with and without end-to-end
training) and we modify its classifier in such a way that it can output
separate objects and states. Specifically, we take the state predictions by
marginalizing their scores across all possible objects and, similarly, we
marginalize object predictions over the set of possible states. Results are
reported in Table 2. We see a consistent increase in best unseen accuracy (5.7
vs 7.8 for CGEff, 6.6 vs 8.1 for CGE) and in the best HM (6.5 vs 8.1 for
CGEff, 7.0 vs 8.1 for CGE) when we separate the two predictions. As a
consequence, the methods also improve in AUC (1.1 to 1.2 CGEff, 1.3 to 1.4 for
CGE). This indicates how providing a separate ground for objects and states
predictions is a useful strategy in the open-world setting. Operating in the
primitives rather than the compositional space, provides a simplification of
the problem that can improve the performance even of existing state-of-the-art
CZSL models.

Method | MIT-States | UT Zappos | C-GQA
---|---|---|---
S | U | HM | S | U | HM | S | U | HM
CGE${}_{\text{ff}}$ [29] |  | 19.6 | 1.3 | 2.4 | 50.3 | 3.4 | 5.0 | 17.4 | 0.4 | 0.9
| +Pseudo-Lab. | 19.7 | 0.9 | 1.8 | 48.5 | 1.1 | 2.2 | 19.8 | 0.2 | 0.4
| +Entropy Min. | 15.1 | 1.7 | 3.1 | 51.9 | 4.2 | 6.4 | 22.1 | 0.4 | 0.9
CompCos [25] |  | 10.8 | 2.0 | 3.6 | 52.4 | 4.1 | 7.6 | 24.3 | 0.4 | 0.7
| +Pseudo-Lab. | 9.2 | 1.9 | 3.2 | 52.9 | 3.7 | 6.8 | 23.6 | 0.3 | 0.5
| +Entropy Min. | 13.2 | 2.1 | 3.7 | 55.0 | 4.2 | 7.9 | 23.1 | 0.6 | 1.1
CGE [29] |  | 17.9 | 1.6 | 3.0 | 55.8 | 5.9 | 10.7 | 25.6 | 0.7 | 1.4
| +Pseudo-Lab. | 10.6 | 2.3 | 3.8 | 56.1 | 3.9 | 7.3 | 21.3 | 0.6 | 1.2
| +Entropy Min. | 17.8 | 1.6 | 3.0 | 60.1 | 4.7 | 8.7 | 24.8 | 1.0 | 1.8
KG-SPff | 13.5 | 2.6 | 4.4 | 53.8 | 6.9 | 12.3 | 22.3 | 0.9 | 1.7
KG-SP | 18.4 | 2.2 | 4.0 | 57.9 | 7.4 | 13.1 | 26.9 | 1.2 | 2.3
Table 4: Partial Open-World CZSL results on MIT-States, UT Zappos and C-GQA. We measure seen (S) and unseen accuracy (U) on the compositions and their harmonic mean (HM). KG-SP refers to our full model with our knowledge-guided pseudo-labeling and inference. CGE${}_{\text{ff}}$ and KG-SP${}_{\text{ff}}$ denotes the non-finetuned version of the methods. For each CZSL baseline, we show the results of the original methods and the same coupled with Entropy-Minimization (Entropy Min.) [11] or Pseudo-labeling (Pseudo-Lab.) [21].  Method | Seen | Unseen | HM
---|---|---|---
KG-SP | 16.6 | 2.8 | 4.8
\+ Pseudo-Labeling | 15.9 | 2.7 | 4.6
\+ Gumbel Softmax | 16.1 | 2.6 | 4.5
\+ ConceptNet-scores | 16.6 | 3.1 | 5.3
Table 5: Partial Open-World CZSL results on MIT-States validation set for
different methods in terms of Seen, and Unseen accuracy and their Harmonic
Mean (HM). Pseudo-Labeling, sampling the pseudo-label using Gumbel-Softmax,
and using ConceptNet to filter unfeasible labels are added one after another.

Effect of depth of the classifier on KG-SP. We ablate the impact of the depth
in Fig 3 for KG-SPff on both MIT-States and UT-Zappos validation sets. The
validation AUC on both UT-Zappos (red curve) and MIT-States (blue curve)
rapidly increases with the depth of the classifier. This shows the importance
of taking into account the multi-task nature of the problem and instantiate
objects and state classifiers that have enough capacity to extract primitive-
specific features. While deeper predictors help, after 3 layers the
performance degrades (_i.e_., going from 26.9 to 24.3 on UT-Zappos when depth
is increased from 3 to 5 layers). The reason behind this drop is mainly linked
to overfitting on seen compositions.

Effect of the knowledge-based masking. We ablate the impact of masking out
unfeasible compositions from the output space in Table 3 for MIT-states
validation set. We experiment with both VisProd and KG-SPff. As the table
shows, KG-SPff benefits from removing unfeasible compositions, with the unseen
accuracy going from 7.3 to 7.5 and the best HM from 7.6 to 8.1. Similarly,
also for VisProd we see a consistent improvement for both the unseen accuracy
(6.8 to 7.2) and the best HM (7.3 to 7.6) when its output space is filtered.
These results show how removing unfeasible compositions from the output space
benefit the performance of OW-CZSL models and that ConceptNet is a reliable
source for estimating feasibility scores of the compositions.

###  4.2 CZSL under Partial supervision

The results on our proposed pCZSL setting are reported in Table 4. In addition
to recognizing unseen compositions in an extremely large compositional space,
in pCZSL the model has to cope with the lack of compositional labels.

In this scenario KG-SP achieves state-of-the-art results on all the three
datasets. On UT-Zappos, KG-SP achieves a HM of 13.1 vs 10.7 of the best
competitor (CGE). Similarly, on MIT-States and C-GQA, the results obtained by
KG-SP exceed the performance of the competitors, with KG-SP achieving 4.4 HM
vs 3.7 of CompCos with entropy minimization on MIT-States, and 2.3 (1.7 when
non end-to-end trained) HM on C-GQA vs 1.8 of the best competitor (CGE). It is
interesting to note how, even incorporating semi-supervised learning
techniques in CGE and CompCos does not bridge the gap between KG-SP and these
methods. Interestingly, semi-supervised learning techniques do not bring
consistent improvements across CZSL models and tasks. Entropy minimization, by
pushing the method toward confident predictions, improves CGE (1.8 vs 1.4 AUC)
and CompCos (1.1 vs 0.7) on C-GQA, and improves CGE${}_{\text{ff}}$ on MIT-
States (3.1 vs 2.4 AUC). However, in the other settings either achieves the
same performances of the baseline (_e.g_. CGE on MIT-States) or degrades them
(_e.g_. CGE on UT-Zappos, 8.7 vs 10.7 AUC). On the other hand, pseudo-labeling
degrades the performance of the CZSL models in all settings (_e.g_. from 3.6
to 3.2 AUC of CompCos on MIT-States), providing advantages only for CGE on
MIT-States (3.8 vs 3.0 AUC). The low efficacy of standard semi-supervised
learning techniques, is due to the fact that the top predicted object/state
may form an unfeasible composition with the ground-truth state/object. Without
modeling the feasibility of state-object compositions, pseudo-labeling may
assign incorrect (and unfeasible) labels while entropy minimization may push
the model toward incorrect predictions. In the following section, we discuss
why this happens and why it is important to restore to our knowledge-aided
pseudo-labeling in pCZSL.

####  4.2.1 Ablation Study

We ablate the two important components of our proposed approach: the pseudo-
labeling strategy and the quality of the ConceptNet feasibility scores.

Effect of the pseudo-labeling strategy. In Table 5, we consider a few
alternatives to our proposed pseudo-labeling strategy for KG-SP. The first
option is to just introduce vanilla pseudo-labeling. This is done by replacing
the missing labels with the top predicted class, when the confidence of the
model is greater than a threshold. This strategy has a negative effect,
reducing the accuracy of the model on both seen and unseen compositions
(_i.e_. 16.6 vs 15.9 on the seen, 2.8 vs 2.7 on the unseen). This is because
pseudo-labeling alone is prone to confirmation bias [1] since the model can
become increasingly more confident about the predictions that, without
modeling the feasibility of compositions, may be not only incorrect, but also
unfeasible.

The second alternative is to use Gumbel-softmax [15], sampling a label instead
of performing direct pseudo-labeling. This approach may allow the model to
overcome the confirmation bias, especially in the initial training stages by
sampling different labels w.r.t. the top prediction. However, this strategy
alone does not achieve good results, degrading the performance of the base
model (_e.g_. 4.5 vs 4.8 HM) and performing slightly worse than standard
pseudo-labeling (4.5 vs 4.6 HM). This performance degradation is due in both
cases by not exploiting the external knowledge to correct the pseudo-labeling
process, avoiding using incompatible compositions as supervision.

Our pseudo-labeling strategy, on the other hand, provides a clear improvement
(5.3 vs 4.8 HM) over our method which already attains state-of-the-art results
on open-world CZSL benchmarks. Its benefits come from using external knowledge
to assign a feasibility score to each composition. These feasibility scores
are used to refine the pseudo-labeling process, avoiding incompatible
compositional labels. In addition to this, using the model scores to sample
the label in a probabilistic manner ensures that confirmation bias and model
drift can be avoided.

| Compositions
---|---
| Most Feasible (Top-5) | Least Feasible (Bottom-5)
| painted paint | grimy balloon
| muddy mud | steaming bracelet
| frozen ice | blunt clock
| mossy moss | thin garage
| cloudy cloud | unpainted belt
Objects | States
| Most Feasible (Top-3) | Least Feasible (Bottom-3)
chains | frayed, broken, loose | pureed, unripe, steaming
sugar | melted, whipped, caramelized | scratched, ancient, coiled
sword | blunt, sharp, splintered | filled, runny, closed
laptop | small, shattered, modern | cloudy, sunny, dull
tulip | bright, wilted, ruffled | grimy, raw, damp

Table 6: Examples of ConceptNet feasibility scores. Top: Top-5 (left) and
Least-5 (right) compositions per feasibility; Bottom: Top-3 highest and
Bottom-3 lowest feasible states per random objects.

Quality of the ConceptNet feasibility scores. An important aspect of KG-SP is
the quality of the estimated feasibility scores. In Table 6 we show some
qualitative results of our strategy, reporting detailed quantitative analyses
on the supplementary material. ConceptNet assigns the highest feasibility
scores to compositions where the state and the objects share the same root,
such as painted paint, muddy mud, and mossy moss. This is a consequence of the
very similar contexts in which such states and objects appear. On the other
hand, among the least feasible compositions we find objects with incompatible
physical transformations (_i.e_. steaming bracelet), or unclear states (_e.g_.
unpainted belt, blunt clock). These scores reflect the reliance of ConceptNet
relatedness scores to the context in which words appear. This is also a
limitation, since rare co-occurrences can be deemed as unfeasible compositions
(_e.g_. grimy balloon, thin garage).

When inspecting the most and least feasible compositions for random objects
(Table 6, bottom), we find that the compositions ranked as most feasible such
as bright tulip, caramelized sugar, and blunt sword are indeed feasible in the
reality, while the compositions marked as least feasible are not.
Interestingly, the unfeasible compositions merge objects and states from
different categories. Examples are food vs tools (_e.g_. pureed chains,
scratched sugar) and weather vs objects (_e.g_. cloudy laptop). This suggests
that the ConcepNet-based feasibility scores encode a high-level notion of
grouping, where objects and states can be separated depending on the contexts
where they commonly occur.

##  5 Conclusion

In this work, we addressed the problem of Open-World Compositional Zero-Shot
learning (OW-CZSL), where the goal is to recognize compositions of objects and
states in images given only a subset of them during training and without any
prior on unseen compositions at test time. We address the problem by
revisiting a simple CZSL method, VisProd, that independently predicts state
and object labels. The idea is to simplify the problem by exploiting the much
smaller cardinality of object and state sets w.r.t. the compositional labels.
Our model, KG-SP, uses two different feature extractors to account for the
dissimilarity between the two tasks, and uses external information to remove
less feasible compositions from the output space at test time. As KG-SP does
not require compositional labels, we explore a new challenging setting, CZSL
under partial supervision (pCZSL) where training images have either only
object or state annotation. In pCZSL, we use the feasibility scores to aid the
estimation of the missing labels. Experiments show that KG-SP achieves the
state of the art in OW-CZSL and pCZSL, outperforming recent CZSL approaches
coupled with standard semi-supervised learning techniques.

Limitations and Broader Impact. One weakness of our approach, shared with CZSL
methods, is that the absolute performance on all OW-CZSL benchmarks is quite
low (_e.g_. 2.9 unseen accuracy on C-GQA). This can significantly hamper the
real-world deployment of these models. However, we believe that this research
topic is crucial to bridge the gap between machine and human visual
compositional recognition abilities: our work contributes to the field by
making another step toward this direction. Another limitation of our work is
that automatic retrieving feasible state-object compositions is a process
without expert supervision, thus vulnerable to inaccuracies in the knowledge
base. For instance, if a valid composition is marked as unfeasible, it will
not be predicted during inference and removed from candidate pseudo-labels
during pCZSL training. This may lead to the model producing erroneous
outcomes, even reflecting potential biases in the knowledge base. Modeling
issues in the knowledge base and/or merging multiple external sources is an
important topic for future research in OW-CZSL and pCZSL.

Acknowledgments This work has been partially funded by the ERC (853489 -
DEXIM), by the DFG (2064/1 – Project number 390727645), and as part of the
Excellence Strategy of the German Federal and State Governments. The authors
thank the International Max Planck Research School for Intelligent Systems
(IMPRS-IS) for supporting Shyamgopal Karthik.

## References

  * [1] Eric Arazo, Diego Ortego, Paul Albert, Noel E O’Connor, and Kevin McGuinness.  Pseudo-labeling and confirmation bias in deep semi-supervised learning.  In IJCNN, 2020.
  * [2] Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik.  A causal view of compositional zero-shot recognition.  In NeurIPS, 2020.
  * [3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [4] Rodrigo Berriel, Stephane Lathuillere, Moin Nabi, Tassilo Klein, Thiago Oliveira-Santos, Nicu Sebe, and Elisa Ricci.  Budget-aware adapters for multi-domain learning.  In CVPR, 2019.
  * [5] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  ICLR, 2020.
  * [6] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  NeurIPS, 2019.
  * [7] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.  Enriching word vectors with subword information.  Transactions of the Association for Computational Linguistics, 5, 2017.
  * [8] Rich Caruana.  Multitask learning.  Machine learning, 28(1):41–75, 1997.
  * [9] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic.  Multi-label learning from single positive labels.  In CVPR, 2021.
  * [10] Thibaut Durand, Nazanin Mehrasa, and Greg Mori.  Learning a deep convnet for multi-label classification with partial labels.  In CVPR, 2019.
  * [11] Yves Grandvalet, Yoshua Bengio, et al.  Semi-supervised learning by entropy minimization.  NeurIPS, 2004.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [13] Dat Huynh and Ehsan Elhamifar.  Interactive multi-label cnn learning with partial labels.  In CVPR, 2020.
  * [14] Phillip Isola, Joseph J Lim, and Edward H Adelson.  Discovering states and transformations in image collections.  In CVPR, 2015.
  * [15] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In ICLR, 2018.
  * [16] Alex Kendall, Yarin Gal, and Roberto Cipolla.  Multi-task learning using uncertainty to weigh losses for scene geometry and semantics.  In CVPR, 2018.
  * [17] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [18] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In ICLR, 2017.
  * [19] Kaustav Kundu and Joseph Tighe.  Exploiting weakly supervised visual patterns to learn from partial annotations.  In NeurIPS, 2020.
  * [20] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling.  Learning to detect unseen object classes by between-class attribute transfer.  In CVPR, 2009.
  * [21] Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In ICML-WS, 2013.
  * [22] Hae Beom Lee, Eunho Yang, and Sung Ju Hwang.  Deep asymmetric multi-task feature learning.  In ICML, 2018.
  * [23] Yunsheng Li and Nuno Vasconcelos.  Efficient multi-domain learning by covariance normalization.  In CVPR, 2019.
  * [24] Yong-Lu Li, Yue Xu, Xiaohan Mao, and Cewu Lu.  Symmetry and group in attribute-object compositions.  In CVPR, 2020.
  * [25] Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata.  Open world compositional zero-shot learning.  In CVPR, 2021.
  * [26] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In NeurIPS, 2013.
  * [27] Ishan Misra, Abhinav Gupta, and Martial Hebert.  From red wine to red tomato: Composition with context.  In CVPR, 2017.
  * [28] Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert.  Cross-stitch networks for multi-task learning.  In CVPR, 2016.
  * [29] Muhammad Ferjad Naeem, Yongqin Xian, Federico Tombari, and Zeynep Akata.  Learning graph embeddings for compositional zero-shot learning.  In CVPR, 2021.
  * [30] Tushar Nagarajan and Kristen Grauman.  Attributes as operators: factorizing unseen attribute-object compositions.  In ECCV, 2018.
  * [31] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In EMNLP, 2014.
  * [32] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc’Aurelio Ranzato.  Task-driven modular networks for zero-shot compositional learning.  In ICCV, 2019.
  * [33] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  In ICML, 2019.
  * [34] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  In NeurIPS, 2017.
  * [35] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Efficient parametrization of multi-domain deep neural networks.  In CVPR, 2018.
  * [36] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah.  In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning.  In ICLR, 2021.
  * [37] Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard.  Latent multi-task architecture learning.  In AAAI, 2019.
  * [38] Frank Ruis, Gertjan J Burghouts, and Doina Bucur.  Independent prototype propagation for zero-shot compositionality.  In NeurIPS, 2021.
  * [39] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [40] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  NeurIPS, 2020.
  * [41] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In AAAI, 2017.
  * [42] Robyn Speer and Joanna Lowry-Duda.  Conceptnet at semeval-2017 task 2: Extending word embeddings with multilingual relational knowledge.  In International Workshop on Semantic Evaluation (SemEval-2017), 2017.
  * [43] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  JMLR, 15(1), 2014.
  * [44] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  IEEE TPAMI, 41(9), 2018.
  * [45] Aron Yu and Kristen Grauman.  Fine-grained visual comparisons with local learning.  In CVPR, 2014.
  * [46] Aron Yu and Kristen Grauman.  Semantic jitter: Dense supervision for visual comparisons via synthetic images.  In ICCV, 2017.

##  Appendix A Further implementation details

We set all hyperparameters of KG-SP following previous works [25, 29] and our
MLPs from the implementation of [29], using 3 layers with dimension 768 in the
first, 1024 in the second, the number of objects/states in the last, and
intermediate Dropout layers with ratio 0.5. For consistency with previous
works (_e.g_. [30, 24, 29]), we also test our model without fine-tuning the
backbone, denoting it as KG-SPff in the tables.

##  Appendix B Analysis of the ConceptNet Embeddings

In addition to the analysis presented in the main paper, we further analyze
the quality of the ConceptNet embeddings both quantitatively and
qualitatively.

Comparison with Alternative Word Embeddings. We compare ConceptNet [41] with
other popular word embeddings such as Word2Vec [26], Glove [31] and FastText
[7], as well as language models [33].

For what concerns the word embeddings, we reject compositions whose cosine
similarity between the state and object embeddings is less than 0. For the
language models, we use GPT-2 [33] by querying a specific prompt and
extracting the likelihood of the next word to be either Yes or No. We can then
set as feasible all compositions whose likelihood of Yes is higher than the
likelihood of No. We tested several prompts (_e.g_. Can OBJ be STA?, You can
see a STA OBJ), but we found the most effective to be: Question: Is this a STA
OBJ? Answer:, with STA and OBJ being a state and object respectively.

In Fig. 4, we report the results of the feasibility scores computed through
the various strategy. Specifically, we report in the x-axis the percentage of
compositions correctly considered as feasible (w.r.t. the ones present in the
dataset) and on the y-axis the percentage of compositions correctly rejected
(_i.e_. compositions not present in the datasets). We report these results for
both MIT-states (Fig. 1.a) and C-GQA (Fig. 1.b). In both these datasets,
FastText accepts nearly all the compositions therefore, rejecting very few
compostions. Similar trends can be seen for Word2Vec and Glove, which reject
more unfeasible compositions. While GPT-2 rejects more unfeasible
compositions, it also rejects compositions present in the dataset. ConceptNet
achieves the best trade-off by accepting most of the compositions present in
the dataset, while still rejecting the largest possible number of unfeasible
compositions among the competitors.

Feasibility scores | Seen | Unseen | HM | AUC
---|---|---|---|---
None | 26.3 | 7.4 | 7.9 | 1.3
CompCos (No Tuning) | 26.3 | 7.4 | 7.9 | 1.3
CompCos (Tuned) | 26.3 | 7.5 | 8.0 | 1.4
ConceptNet | 26.5 | 7.7 | 8.2 | 1.4
Table 7: Comparison of various feasibility scores on the MIT-States for KG-
SPff. We see that despite tuning the threshold for CompCos, ConceptNet is
still able to achieve better results.

(a) MIT-States

(b) C-GQA

Figure 4: Comparison of the ConceptNet embeddings against other word
embeddings (Word2Vec, Glove, FastText) and GPT-2. The x-axis denotes the
percentage of compositions in the dataset that are correctly accepted. The
y-axis denotes the percentage of compositions not present in the dataset that
are correctly rejected. We can see that ConceptNet provides an excellent
trade-off by rejecting a large number of unfeasible compositions, while still
accepting most of the compositions present in the dataset.

Quantitative Analysis. In CompCos [25], a method for estimating the
feasibility scores of each composition is presented. CompCos computes
feasibility scores by using the cosine similarity of learned object/state
embeddings, exploiting the available seen compositions during training.
Specifically, for a given unseen state-object composition, CompCos computes
the cosine similarity of the embeddings of the given object with those of any
object containing the same state as seen during training, taking the maximum
similarity as feasibility score. The same procedure is applied for the states,
and the two scores are averaged.

A natural question is whether our ConceptNet scores can outperform the ones of
CompCos. We analyze this in Table 7 for OW-CZSL on the MIT-States validation,
using different feasibility scores to perform hard-masking on KG-
SP${}_{\text{ff}}$, . From the table, we can see that hard masking with the
feasibility scores of CompCos does not bring any benefits when used with a
natural threshold of zero (_i.e_. half of the range of cosine-similarity
scores). When we tune a threshold to filter out less feasible compositions (as
in [25]), we see slight improvements (_e.g_. 8.0 vs 7.9 AUC, 1.4 vs 1.3 best
HM). However, our ConceptNet-based feasibility scores bring a larger
improvement in performance (_e.g_. 7.7 best unseen accuracy, 8.2 best HM),
without requiring tuning any threshold or accessing compositional annotations
during training. The latter feature makes ConceptNet feasibility scores
applicable also in pCZSL, while CompCos-based ones cannot be applied in such
setting.

Qualitative Analysis. In this section, we expand the analysis of Section 4.2.1
in the main paper, and we present the top-3 most feasible states and bottom-3
least feasible states for 25 randomly selected objects in Table 6. Similarly,
we show the top-3 most feasible objects and the bottom-3 least feasible
objects for 25 randomly selected states in Table 9 . As discussed earlier both
in Section 4.2.1 in the main paper and in Section B, ConceptNet provides good
estimates of the feasibility of state-object compositions. Overall, the most
feasible compositions selected by the embeddings usually turn out to be
commonly occurring combinations of the states and objects. For instance, clear
sky, burnt flame, and engraved jewelry are all frequently occurring state-
object combinations. An interesting aspect of the ConceptNet embeddings is the
implicit clustering of the most relevant object/state pairs. For instance, in
Table 8 food items (paste, pizza, salad) are linked to cooking-related states
states such as sliced and diced. Similarly, both chain and cord are linked to
the state frayed. We can see similar patterns also in Table 9, where murky and
muddy are linked with mud, and unripe with fruit, pear and orange. However,
the estimated feasibility scores are not perfect, and can lead to erroneous
outcomes. For instance, this happens for rarely occurring compositions that
might be considered unfeasible (_e.g_. steaming chains in Tab. 6, dirty
bracelet in Tab. 9), and for co-occurring words that might receive high
feasibility scores despite not being compatible (_e.g_. sunny sea in Tab. 8,
molten flame in Tab. 9). This is a limitation of feasibility scores based on
single words representations, since these models are biased by the context in
which word appear, and their co-occurring frequency. Using a combination of
cues from multiple sources may be an effective tool to deal with this issue.

Objects | States
---|---
| Most Feasible (Top-3) | Least Feasible (Bottom-3)
balloon | inflated, deflated, filled | grimy, rusty, raw
bear | heavy, large, huge | crinkled, dark, damp
bridge | narrow, curved, bent | barren, cluttered, pressed
building | standing, tall, moldy | ruffled, sharp, pureed
bush | mossy, blunt, wilted | ripped, broken, grimy
cable | coiled, frayed, loose | empty, filled, barren
camera | raw, sharp, lightweight | pierced, crinkled, ruffled
chains | loose, broken, frayed | pureed, unripe, steaming
chair | standing, upright, bent | pierced, thawed, sliced
concrete | unpainted, crushed, molten | ruffled, folded, whipped
cord | coiled, frayed, loose | foggy, dirty, grimy
flower | wilted, ruffled, verdant | grimy, cored, scratched
jewelry | engraved, pierced, worn | steaming, squished, full
mirror | painted, dented, shiny | unripe, pureed, cooked
paste | mashed, pureed, sliced | tall, standing, winding
pizza | sliced, cooked, diced | bright, clear, modern
salad | diced, mashed, sliced | tight, narrow, smooth
sea | sunny, murky, fresh | closed, unpainted, squished
shower | wet, damp, steaming | burnt, cored, thin
snake | coiled, winding, curved | creased, crumpled, pressed
steel | molten, rusty, shiny | runny, unripe, verdant
steps | standing, winding, straight | viscous, tight, dry
stream | muddy, winding, spilled | dented, rusty, caramelized
sugar | caramelized, whipped, melted | scratched, ancient, coiled
tile | painted, unpainted, moldy | whipped, inflated, ripe

Table 8: Examples of top-3 and bottom-3 states associated to 25 randomly
selected objects, according to ConceptNet feasibility scores.

States | Objects
---|---
| Most Feasible (Top-3) | Least Feasible (Bottom-3)
ancient | stone, bronze, ceramic | candy, sandwich, cake
bright | lightbulb, sky, orange | sandwich, pizza, beef
browned | sauce, butter, beef | key, vacuum, wall
brushed | coat, wool, dust | cave, lake, building
burnt | flame, fire, smoke | bathroom, shower, camera
clear | sky, concrete, glass | sandwich, pants, pizza
cooked | meat, soup, sauce | key, mirror, moss
crushed | ground, lemon, concrete | lake, pond, clock
curved | blade, steel, knife | oil, eggs, bag
deflated | balloon, bubble, tire | kitchen, wood, coffee
dirty | dirt, mud, dust | balloon, bracelet, cord
dull | blade, knife, sword | fig, mountain, cookie
filled | vacuum, bag, bucket | cable, cliff, sword
heavy | metal, bear, handle | fig, pool, pond
moldy | basement, dust, carpet | tiger, road, highway
molten | metal, flame, copper | book, cat, furniture
murky | cloud, mud, pond | tire, wheel, nut
old | building, roots, tree | sandwich, bubble, chocolate
open | door, window, gate | bus, cliff, granite
peeled | orange, potato, fruit | coffee, tea, tower
thin | paper, blade, paste | garage, car, garden
tiny | penny, toy, town | gear, beef, field
torn | fabric, paper, clothes | pond, bronze, clock
unripe | fruit, pear, orange | mirror, cat, steel
viscous | foam, mud, paste | garden, clock, city

Table 9: Examples of top-3 and bottom-3 objects associated to 25 randomly
selected states, according to ConceptNet feasibility scores.

##  Appendix C Additional Quantitative Experiments

In this section, we report additional experimental results, not included in
the main paper due to the lack of space. First, while in the main paper we
focused our ablation studies on MIT-States, here we provide additional results
also for UT-Zappos and C-GQA. Moreover, we report pCZSL results when applying
a bias over the seen compositions, for a direct comparison with OW-CZSL
results.

Ablations on other benchmarks. Due to space constraints, in the main paper we
performed the ablation studies only on MIT-States, following previous works
[29, 25, 24, 32]. However, our results are consistent across settings. As
examples, here we also provide ablation studies of the benefit of
marginalization on the UT-Zappos dataset in Tab 10. Again, we see that
marginalization consistently brings improvements to CGE. We also evaluate the
benefit of hard masking on the C-GQA dataset in Tab. 11. We again see hard
masking bringing improvements to the HM and AUC on the validation set of C-GQA
for both VisProd and KG-SP${}_{\text{ff}}$.

pCZSL: results with bias. In the main paper, we evaluate the models on the
pCZSL without applying a bias on the seen compositions. This is because, in
the partial setting, we do not have the explicit notion of seen or unseen
compositions, thus we cannot follow the same evaluation protocol of standard
CZSL and OW-CZSL. Here we show the results on pCZSL when assuming the seen
compositions to be known, and applying a bias to them. We underline that this
is not fair in the pCZSL setting, but allows us to directly compare results of
pCZSL and OW-CZSL. The results are shown in Tab. 12 for all 3 datasets.
Consistently with the results without bias, we find that KG-SP always
outperforms CompCos and CGE. Specifically, KG-SP achieves significantly better
AUC, best harmonic mean, and either comparable or superior best accuracy on
seen and unseen compositions. More importantly, the drop from the open-world
setting is quite small, _e.g_. 0.78 to 0.61 AUC on C-GQA. This is remarkable,
since in pCZSL we have half the labels of OW-CZSL, and indicates the
robustness of KG-SP to the available annotations.

| Marginaliz. | Seen | Unseen | HM | AUC
---|---|---|---|---|---
CGEff |  | 51.3 | 30.0 | 25.2 | 11.5
✓ | 51.4 | 46.6 | 30.0 | 15.4
CGE |  | 51.1 | 47.6 | 33.3 | 17.5
✓ | 53.9 | 48.5 | 32.3 | 18.1
Table 10: OW-CZSL results in the validation set of UT-Zappos when using marginalization. CGEff is the approach of [29] with frozen backbone whereas CGE performs end-to-end training.  | Hard Masking | Seen | Unseen | HM | AUC
---|---|---|---|---|---
VisProd |  | 24.8 | 14.8 | 13.2 | 2.8
✓ | 24.9 | 14.9 | 13.3 | 2.9
KG-SP |  | 30.5 | 16.9 | 15.4 | 3.9
✓ | 30.6 | 17.0 | 15.5 | 4.0
Table 11: OW-CZSL results in the validation set of C-GQA when using hard
masking.

Method | MIT-States | UT Zappos | C-GQA
---|---|---|---
S | U | HM | AUC | S | U | HM | AUC | S | U | HM | AUC
| CompCos [25] | 18.2 | 6.3 | 5.6 | 0.64 | 56.0 | 42.7 | 34.0 | 18.4 | 22.0 | 1.8 | 2.9 | 0.31
| CGE [29] | 19.2 | 5.5 | 5.2 | 0.61 | 60.4 | 43.4 | 34.7 | 19.5 | 26.7 | 1.2 | 2.1 | 0.25
| KG-SP | 20.5 | 6.3 | 5.9 | 0.77 | 60.0 | 43.3 | 40.2 | 22.6 | 29.2 | 2.4 | 4.1 | 0.61

Table 12: pCZSL results on MIT-States, UT Zappos and C-GQA. We measure best
seen (S) and unseen accuracy (U), best harmonic mean (HM), and area under the
curve (AUC) on the compositions.

##  Appendix D CZSL vs OW-CZSL: an example

In the main text we described the OW-CZSL setting, proposed in [25]. Here, we
provide an example to clarify how this setup differs from the more standard
CZSL. For more details on this setting, please refer to [25].

Let us consider a toy benchmark where the training set contains images of the
following compositions: wet cat, dry apple, dry dog, ripe apple. Similarly,
let us assume that the same benchmark contains images of the following unseen
compositions in the test set: wet dog, dry cat. Note that other three
compositions, _i.e_. wet apple, ripe dog, ripe cat, are not present in any
image of the dataset, either because they are unfeasible (_e.g_. ripe dog) or
because no image has been collected for them (_i.e_. wet apple).

The main difference between CZSL and OW-CZSL is that, in the latter, we have
no priors on unseen compositions, and we thus consider the full compositional
space at test time. To clarify, in standard CZSL, a model assumes to know
which unseen compositions are present in the test set of the dataset and which
are not. Thus, a CZSL model would predict 6 compositions: the 4 seen ones
during training, and the 2 unseen ones that have at least one image in the
test set (_i.e_. wet dog, and dry cat).

In OW-CZSL, we do not know which compositions are present in the test set. As
a consequence, the output space needs to consider all possible unseen
compositions. In our toy benchmark, this means predicting the 4 seen
compositions and all the 5 compositions for which we did not have training
images (_i.e_. wet dog, dry cat, wet apple, ripe dog, ripe cat). Note that in
OW-CZSL a model needs to cope with the presence of "distractors", _i.e_.
compositions close to other existing ones but not present in the dataset
(_e.g_. wet apple vs ripe apple) as well as modeling unfeasible compositions
(_e.g_. ripe dog) to simplify the task. While this is a toy example, the
difference between the settings is huge in CZSL benchmarks, where these
challenges are more pronounced. As an example, CZSL models on MIT-States
consider only 1’662 compositions out of the possible 28’175 compositions
considered in the output space of OW-CZSL.

As a final note, despite the difference in the output spaces, models built for
standard CZSL may perform well in OW-CZSL since they can still exploit what
learned from seen compositions to generalize to unseen ones. This can be seen
in Table 1 of the main paper, where CGE [29], designed for standard CZSL, is
competitive in most benchmarks, being either third or second best performing
model.

##  Appendix E Dataset Licenses

The datasets used in our work are: UT-Zappos [45, 46], MIT-States [14], and
C-GQA [29] and their licenses are summarized in Table 13. For MIT-States we
did not find an accompanying license, but the dataset is publicly
available333http://web.mit.edu/phillipi/Public/states_and_transformations/index.html.

Dataset | Licence
---|---
UT-Zappos | Custom: Non-commercial Usage
MIT-States | Not Available
C-GQA | Creative Commons Attribution 4.0 License
Table 13: The datasets employed in the paper and their licences.


# Evaluation of ChatGPT for NLP-based Mental Health Applications

Bishal Lamichhane,
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLM) have been successful in several natural language
understanding tasks and could be relevant for natural language processing
(NLP)-based mental health application research. In this work, we report the
performance of LLM-based ChatGPT (with gpt-3.5-turbo backend) in three text-
based mental health classification tasks: stress detection (2-class
classification), depression detection (2-class classification), and
suicidality detection (5-class classification). We obtained annotated social
media posts for the three classification tasks from public datasets. Then
ChatGPT API classified the social media posts with an input prompt for
classification. We obtained F1 scores of 0.73, 0.86, and 0.37 for stress
detection, depression detection, and suicidality detection, respectively. A
baseline model that always predicted the dominant class resulted in F1 scores
of 0.35, 0.60, and 0.19. The zero-shot classification accuracy obtained with
ChatGPT indicates a potential use of language models for mental health
classification tasks.

###### Index Terms:

ChatGPT, mental health, natural language processing, stress, depression,
suicidality

##  I Introduction

Mental health illnesses are widely prevalent. More than one billion people
worldwide are estimated to be suffering from mental illnesses [1]. In the US,
one in five individuals has mental health issues every year [2]. Proper
management of mental health with timely diagnosis, intervention, and
monitoring requires measuring mental health. Since most mental health
illnesses have bio-psycho-social origins, research on measuring mental health
has considered obtaining bio-behavioral signals to detect the mental health
state of the individual. Social media posts could be a source to detect the
mental health state of an individual. One’s mental health state could be
reflected in the sentiments or linguistic contents of their social media
posts. Accordingly, previous works have demonstrated mental health state
detection based on the analysis of users’ social media posts [3, 4, 5].

Mental health state detection from texts such as social media posts requires
natural language processing (NLP) to infer the sentiment and content shared in
the text. Recently, large language models (LLM) have shown unprecedented
success in natural language understanding. LLMs are neural networks based on
the transformer architecture [6] trained in a self-supervised setting with a
large text corpus, followed by fine-tuning in supervised and reinforcement
learning settings. ChatGPT is a popular LLM-based chat application/assistant
that has been receiving a lot of public attention. ChatGPT as a conversation
agent demonstrates its capability to understand the text and respond
accordingly. Though several limitations of ChatGPT and LLMs, in general, are
acknowledged, these models have shown to be good generalized models across
different applications. The applications of the GPT-4 model, the latest LLM
model from OpenAI, in several areas such as programming and mathematics were
demonstrated by the authors in [7]. Similarly, the authors in [8] demonstrated
GPT-4’s capability in the medical domain for answering USMLE (United States
Medical Licensing Examination) and MultMedQA dataset [9] questions.

One application area for LLM could be in mental health, given the model’s
capability for language understanding. Different application scenarios could
arise for LLM’s use in a mental health context. As a user interacts more with
LLMs, sometimes even sharing their thoughts and concerns, the posts shared
with LLMs could provide a better view of the person’s mental health compared
to social media posts. The LLMs could adapt to respond based on the person’s
mental health state or even provide interventions, e.g., by suggesting to seek
medical care. LLM’s language understanding could also be used as a backend in
many front-end mental health applications. Much effort today is focused on
building custom machine-learning models for mental health detection tasks
based on texts and other modalities. As the LLMs get more intelligent, it is
not inconceivable that these LLMs become a defacto backend for all language
understanding tasks such as in mental health applications

Given the current capability of ChatGPT as one of the most popular LLM-based
chat applications, in this work, we evaluated ChatGPT’s zero-shot
classification performance in three mental health application tasks: stress
detection, depression detection, and suicidality detection tasks based on
user’s social media posts. We used publicly available labeled datasets for our
evaluations.

##  II Method

###  II-A Dataset

####  II-A1 Stress Detection Dataset

We evaluated ChatGPT’s ability to detect stressed states using the labeled
stress detection dataset available at [10]. The dataset was inspired by the
work in [11] and consists of users’ social media posts in different Reddit
group categories. A train and test set of 2838 posts and 715 posts,
respectively, are available in the dataset. Each post has an associated label
of stress or non-stress. We evaluated the ChatGPT model on the test set posts
only.

An example post in the stress class is:
”By the fourth infusion, I was able to sleep through the night, and only got a
mild feeling of anxiety when talking or thinking about it. Today, while
leaving my psychologist’s office, I saw someone get hit by a car. It was not
pretty. He died. Right as I’m starting to move past one, another happens.”

Similarly, an example post in the non-stress class is:
”Best friend knows I have anxiety and I am always asking her if she’s mad at
me. Well, yesterday she asked if I wanted to go to the beach next weekend, I
said yes, and then today I told her I couldn’t because I was supposed to watch
my sister’s kids. She wants to know why I don’t take them with us. It’s a two
hour drive, and I really just don’t want to go. But then she says that I
always say no when she asks me to go somewhere with her.”

####  II-A2 Depression Detection Dataset

For the depression detection task, we used the dataset available at [12]. The
dataset consists of user posts in different Reddit groups (subreddits) and web
blogs. We used the user posts in depression-related Reddit groups as the data
for the depression class. The Reddit posts from the non-depression category in
the dataset [12] were taken as the negative class. The number of posts in the
depression class is 1293 and that in the non-depression class is 548.

####  II-A3 Suicidality Detection Dataset

A suicidality detection dataset based on social media posts is available in
[13]. We used the subset of 500 users in the dataset obtained from [14].
Experts labeled posts from these users as belonging to one of the five
classes: Supportive, Suicidal Ideation, Suicidal Behavior, Suicidal Attempt,
and Suicidal Indicator.

###  II-B ChatGPT Evaluation

For each detection task, we provided the input text from the user post to the
OpenAI ChatGPT API with the following prompt.

”Which of the classes can the following post be assigned to? Only reply with
answers from one of the classes: CLASS_NAMES..

For example, for the stress/non-stress detection task, the prompt would be:

”Which of the classes can the following post be assigned to? Only reply with
answers from one of the classes: stressed, non-stressed..

We used the python openai package to invoke the ChatComplete API of ChatGPT.
The gpt-3.5-turbo backend was used; we still do not have access to the GPT-4
backend. We compared ChatGPT’s detection from a single API call to the
annotations in the dataset and computed the F1 score of detection. In a multi-
class setting, we compute the weighted F1 score. We also evaluated the
confusion matrix of the predictions.

##  III Results

The F1 score obtained for the three detection tasks is provided in Table I. We
obtained an F1 score of 0.72 for the stress detection task. In this 2-class
classification task, the baseline model that predicted the dominant class
(stress) resulted in an F1 score of 0.35. For the depression detection task,
we obtained an F1 score of 0.86 which is much higher than the baseline F1
score of 0.60 obtained by always predicting the dominant class (depression).
Finally, we obtained a low F1 score of 0.37 for the 5-class suicidality
classification task. The F1 score was still higher than the baseline F1 score
of 0.19 obtained by always predicting the dominant class (Suicidal Ideation).

TABLE I: Classification performance obtained in the three text-based mental health applications using ChatGPT. Dataset | F1 score | Balanced Accuracy
---|---|---
Stress Detection | 0.73 | 0.73
Depression Detection | 0.86 | 0.85
Suicidality Detection | 0.37 | 0.33

The confusion matrix for each of the three classification tasks is shown in
Figure 1,2, and 3 respectively. Some posts in the dataset could not be
evaluated due to errors in reading these posts from the dataset or invoking
the API with the given posts.

Figure 1: Confusion matrix for the prediction from ChatGPT on the stress
detection task. Figure 2: Confusion matrix for the prediction from ChatGPT on
the depression detection task. Figure 3: Confusion matrix for the prediction
from ChatGPT on the 5-class suicidality detection task. When ChatGPT could not
assign any of the five classes to the input text, we labeled it as belonging
to the None class.

##  IV Discussion

LLMs show a good language understanding capability and could be useful in NLP-
based mental health applications. One such application is mental health state
detection based on users’ social media posts. We evaluated the zero-shot
classification accuracy obtained from ChatGPT, a popular LLM-based
application, across three mental health detection tasks. We found that ChatGPT
provides a good classification performance compared to the baseline model,
indicating that ChatGPT can infer a user’s mental health state to an extent
based on their social media posts. Since the ChatGPT is a general-purpose
language model, it could be further fine-tuned/adapted for a specific mental
health application or a range of related applications.

The classification performance obtained from ChatGPT for the stress detection
and depression detection task was quite promising (Table I). For the stress
detection task, the authors in [10] trained three machine learning models
using the in-domain training set (labeled user posts similar to the test
dataset). The BERT model [15] fine-tuned for the stress detection task
resulted in an F1 score of 0.81, higher than the F1 score obtained with a
random forest and logistic regression model trained using the TF-IDF (term
frequency-inverse document frequency) and Word2Vec features. The F1 score of
the fine-tuned BERT model reported in [10] is higher than the F1 score of 0.73
obtained from the ChatGPT model in our evaluation (Table I). However, we
evaluated ChatGPT in a zero-shot classification setting. Thus, the F1 score
obtained with ChatGPT could be considered encouraging. Fine-tuning the
underlying LLM model for the stress detection task could further improve the
classification. For the depression detection task, the authors in [12] did not
report any model performance. The authors in [16] reported an F1 score of 0.93
on the depression detection dataset using bigrams, LIWC (Linguistic Inquiry
and Word Count dictionary), and LDA (Latent Dirichlet Allocation)-based
features in a neural network model. With zero-shot classification from ChatGPT
already yielding a high F1 score of 0.86, fine-tuning can further improve
performance to match or exceed the performance from [16].

The lowest F1 score among our tasks was obtained for the suicidality detection
task. Suicidality detection was a 5-class classification problem which might
have caused a high classification error due to the higher inter-class
confusion. From the confusion matrix (Figure 3), we observed that inputs are
mostly predicted to be in the suicidal ideation category. Compared to the
binary stress and depression detection tasks, the class boundaries between the
closely related classes of suicidality (ideation, behavior, attempt, and
indicator) might not be as distinct. The authors in [14] reported an F1 score
of 0.65 on the suicidality detection dataset using word embeddings as input to
a CNN model. The authors evaluated their model with 5-fold cross-validation.
Thus, the model was trained with the in-domain data from four of the five
folds. The confusion matrix reported by the authors in [14] also indicates
that suicidal ideation is the most commonly predicted category, similar to our
results in Figure 3. The high confusion between the suicidality classes
obtained in our evaluations and those in [14] indicates that the class
boundary is highly likely to be overlapped.

Our evaluations have several limitations. First, we evaluated ChatGPT with
GPT-3.5-turbo backend. Recently, a much-improved backend of GPT-4 was
released. We plan to evaluate the performance of GPT-4 for the mental health
application tasks in future work. Second, we only explored a limited prompt
setting and considered the first response from the ChatGPT as its prediction.
There might be other prompts that can provide better classification. The
variations in responses over multiple calls and with different prompts could
also provide auxiliary information helpful for the classification tasks. We
will explore this in future work. Third, we evaluated detection tasks in
relatively small datasets. Future work should consider evaluations using
larger text corpora. Finally, the evaluations are highly dependent on the
annotations. We used the annotation provided with the stress and depression
detection dataset. These two datasets, unlike the suicidality detection
dataset, were likely not annotated by multiple experts. The datasets could be
re-annotated with experts to understand if the LLM model’s confusion aligns
with the disagreement between the annotators.

## References

  * [1] W. H. Organization _et al._ , “World mental health report: transforming mental health for all,” 2022.
  * [2] S. Abuse _et al._ , “Key substance use and mental health indicators in the united states: results from the 2019 national survey on drug use and health,” 2020.
  * [3] G. Gkotsis, A. Oellrich, S. Velupillai, M. Liakata, T. J. Hubbard, R. J. Dobson, and R. Dutta, “Characterisation of mental health conditions in social media using informed deep learning,” _Scientific reports_ , vol. 7, no. 1, p. 45141, 2017.
  * [4] S. Chancellor and M. De Choudhury, “Methods in predictive techniques for mental health status on social media: a critical review,” _NPJ digital medicine_ , vol. 3, no. 1, p. 43, 2020.
  * [5] J. Kim, J. Lee, E. Park, and J. Han, “A deep learning model for detecting mental illness from user content on social media,” _Scientific reports_ , vol. 10, no. 1, pp. 1–6, 2020.
  * [6] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [7] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg _et al._ , “Sparks of artificial general intelligence: Early experiments with gpt-4,” _arXiv preprint arXiv:2303.12712_ , 2023.
  * [8] H. Nori, N. King, S. M. McKinney, D. Carignan, and E. Horvitz, “Capabilities of gpt-4 on medical challenge problems,” _arXiv preprint arXiv:2303.13375_ , 2023.
  * [9] K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl _et al._ , “Large language models encode clinical knowledge,” _arXiv preprint arXiv:2212.13138_ , 2022.
  * [10] G. Chiang and A. Stepanyan, “Insight stress analysis,” https://github.com/gillian850413/Insight_Stress_Analysis, 2020.
  * [11] E. Turcan and K. McKeown, “Dreaddit: A reddit dataset for stress analysis in social media,” _arXiv preprint arXiv:1911.00133_ , 2019.
  * [12] P. Inna, “Identifying depression,” https://github.com/Inusette/Identifying-depression, 2018.
  * [13] A. Alambo, “Suicide risk assessment using reddit,” https://github.com/AmanuelF/Suicide-Risk-Assessment-using-Reddit, 2022.
  * [14] M. Gaur, A. Alambo, J. P. Sain, U. Kursuncu, K. Thirunarayan, R. Kavuluru, A. Sheth, R. Welton, and J. Pathak, “Knowledge-aware assessment of severity of suicide risk for early intervention,” in _The world wide web conference_ , 2019, pp. 514–525.
  * [15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [16] M. M. Tadesse, H. Lin, B. Xu, and L. Yang, “Detection of depression-related posts in reddit social media forum,” _IEEE Access_ , vol. 7, pp. 44 883–44 893, 2019.


* Karniadakis et al. (2021) Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L., 2021.  Physics-informed machine learning.  Nature Reviews Physics 3, 422–440.
  * Krishnapriyan et al. (2021) Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., Mahoney, M.W., 2021.  Characterizing possible failure modes in physics-informed neural networks.  Advances in Neural Information Processing Systems 34, 26548–26560.
  * Leok and Shingel (2012) Leok, M., Shingel, T., 2012\.  General techniques for constructing variational integrators.  Frontiers of Mathematics in China 7, 273–303.
  * Leonard (1997) Leonard, N.E., 1997.  Stability of a bottom-heavy underwater vehicle.  Automatica 33, 331–346.
  * Leonard and Marsden (1997) Leonard, N.E., Marsden, J.E., 1997\.  Stability and drift of underwater vehicle dynamics: mechanical systems with rigid motion symmetry.  Physica D: Nonlinear Phenomena 105, 130–162.
  * Marsden and Ratiu (2013) Marsden, J., Ratiu, T., 2013\.  Introduction to Mechanics and Symmetry: a basic exposition of classical mechanical systems. volume 17.  Springer Science & Business Media.
  * Marsden et al. (1984) Marsden, J., Ratiu, T., Weinstein, A., 1984.  Semidirect products and reduction in mechanics.  Trans. AMS 281, 147–177.
  * Marsden and Weinstein (1983) Marsden, J., Weinstein, A., 1983\.  Coadjoint orbits, vortices, and Clebsch variables for incompressible fluids.  Phys. D 7, 305–323.
  * Marsden et al. (1999) Marsden, J.E., Pekarsky, S., Shkoller, S., 1999.  Discrete Euler–Poincaré and Lie–Poisson equations.  Nonlinearity 12, 1647\.
  * Marsden and Weinstein (1982) Marsden, J.E., Weinstein, A., 1982\.  The Hamiltonian structure of the Maxwell-Vlasov equations.  Physica D: nonlinear phenomena 4, 394–406.
  * Marsden and West (2001) Marsden, J.E., West, M., 2001\.  Discrete mechanics and variational integrators.  Acta numerica 10, 357–514.
  * McLachlan (1993) McLachlan, R.I., 1993.  Explicit Lie–Poisson integration and the Euler equations.  Physical review letters 71, 3043\.
  * McLachlan and Quispel (2002) McLachlan, R.I., Quispel, G.R.W., 2002\.  Splitting methods.  Acta Numerica 11, 341–434.
  * Morrison et al. (2006) Morrison, P., Francoise, J., Naber, G., Tsou, S., 2006\.  Hamiltonian fluid dynamics.  Encyclopedia of Mathematical Physics 2, 593–600.
  * Morrison (1980) Morrison, P.J., 1980.  The Maxwell-Vlasov equations as a continuous hamiltonian system.  Physics Letters A 80, 383–386.
  * Morrison (1998) Morrison, P.J., 1998.  Hamiltonian description of the ideal fluid.  Reviews of modern physics 70, 467\.
  * Ott (2002) Ott, E., 2002.  Chaos in Dynamical Systems.  Cambridge university press.
  * Patel et al. (2022) Patel, R.G., Manickam, I., Trask, N.A., Wood, M.A., Lee, M., Tomas, I., Cyr, E.C., 2022.  Thermodynamically consistent physics-informed neural networks for hyperbolic systems.  Journal of Computational Physics 449, 110754.
  * Pathak et al. (2022) Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z., Azizzadenesheli, K., et al., 2022.  Fourcastnet: A global data-driven high-resolution weather model using adaptive Fourier neural operators.  arXiv preprint arXiv:2202.11214 .
  * Raissi et al. (2019) Raissi, M., Perdikaris, P., Karniadakis, G.E., 2019.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational physics 378, 686–707.
  * Salmon (2004) Salmon, R., 2004.  Poisson-bracket approach to the construction of energy-and potential-enstrophy-conserving algorithms for the shallow-water equations.  Journal of the atmospheric sciences 61, 2016–2036.
  * Šípka et al. (2023) Šípka, M., Pavelka, M., Esen, O., Grmela, M., 2023\.  Direct Poisson neural networks: Learning non-symplectic mechanical systems.  arXiv preprint arXiv:2305.05540 .
  * Tao (2016) Tao, M., 2016.  Explicit symplectic approximation of nonseparable Hamiltonians: Algorithm and long time performance.  Physical Review E 94, 043303\.
  * Tonti (2013) Tonti, E., 2013.  The Mathematical Structure of Classical and Relativistic Physics: A General Classification Diagram.  Springer.  doi:10.1007/978-1-4614-7422-7.
  * Weinstein (1983) Weinstein, A., 1983.  Hamiltonian structure for drift waves and geostrophic flow.  The Physics of Fluids 26, 388–390.
  * Xiong et al. (2020) Xiong, S., Tong, Y., He, X., Yang, S., Yang, C., Zhu, B., 2020.  Nonseparable symplectic neural networks.  arXiv preprint arXiv:2010.12636 .
  * Zhang et al. (2022) Zhang, Z., Shin, Y., Em Karniadakis, G., 2022.  Gfinns: Generic formalism informed neural networks for deterministic and stochastic dynamical systems.  Phil. Trans. of the Royal Soc. A 380, 20210207.

##  Appendix A Derivation of LPNets transformations in coordinate-free form

We quickly review some facts on Lie-Poisson systems using intrinsic notations,
and explain the emergence of the Lie-Poisson bracket as a consequence of the
symmetry of the system.

#### Motion on Lie groups

Suppose $G$ is a Lie group and also the configuration manifold of the
mechanical system. The equations of motion for the system with Lagrangian
$L(g,\dot{g})$ follow as usual from Hamilton’s principle. Assuming that the
linear map $\dot{g}\mapsto p(g,\dot{g})=\frac{\partial
L}{\partial\dot{g}}(g,\dot{g})$ is invertible for all $g\in G$, one can
equivalently formulate the dynamics via the canonical Hamilton equations for
the Hamiltonian $H(g,p)=\left\langle
p,\dot{g}(g,p)\right\rangle-L(g,\dot{g}(g,p))$, exactly as recalled in Sec.
2.1 for vector spaces (except that care must be taken if one wants to
formulate these equations in coordinate free form).

#### Lie group symmetries

Let us assume that the Lagrangian is invariant under the action of the group
on itself by multiplication on the left: $L(hg,h\dot{g})=L(g,\dot{g})$, for
all $h\in G$. One automatically gets that $H$ is also invariant:
$H(hg,hp)=H(g,p)$, for all $h\in G$. It follows from this that $H$ is
completely determined by it’s dependence on momenta at the identity $g=e$,
since we can write $H(g,p)=H(e,g^{-1}p)=:h(g^{-1}p)$. Here $\mu=g^{-1}p$ is
the reduced (or body) momentum, which belongs the dual $\mathfrak{g}^{*}$ to
the Lie algebra $\mathfrak{g}$ of $G$, and
$h:\mathfrak{g}^{*}\rightarrow\mathbb{R}$ is the reduced Hamiltonian. From
this symmetry, it is naturally expected that the canonical Hamilton equations
of motion for $(g(t),p(t))$ can be equivalently formulated by only using the
reduced momenta $\mu(t)=g(t)^{-1}p(t)$, via an equation on $\mathfrak{g}^{*}$.
This is indeed the case, which gives rise to the Lie-Poisson equations for
$\mu(t)$.

#### The Lie-Poisson equations

The Lie-Poisson equations in intrinsic form are written as

$\dot{\mu}=\operatorname{ad}^{*}_{\frac{\partial h}{\partial\mu}}\mu\,,$ (78)

where
$\operatorname{ad}^{*}_{\xi}:\mathfrak{g}^{*}\rightarrow\mathfrak{g}^{*}$ is
the infinitesimal coadjoint action defined by
$\langle\operatorname{ad}^{*}_{\xi}\mu,\eta\rangle=\langle\mu,\operatorname{ad}_{\xi}\eta\rangle$
with $\operatorname{ad}_{\xi}\eta=[\xi,\eta]$, for $\xi,\eta\in\mathfrak{g}$
and $\mu\in\mathfrak{g}^{*}$. This equation is the intrinsic version of (18)
written earlier. The evolution of a function $f(\mu)$ along the solution of
(78) is found as $\dot{f}=\\{f,h\\}$ with

$\left\\{f,h\right\\}=-\left<\mu,\left[\frac{\partial
f}{\partial\mu},\frac{\partial h}{\partial\mu}\right]\right>$ (79)

the Lie-Poisson bracket. As we already commented earlier this Poisson bracket
is noncanonical, however it is related to the canonical Poisson bracket
$\\{\cdot,\cdot\\}_{\rm can}$ governing the dynamics of $(g,p)$ as follows

$\left\\{f\circ\pi,h\circ\pi\right\\}_{\rm can}=\left\\{f,h\right\\}\circ\pi$

with $\pi(g,p)=g^{-1}p=\mu$ the map sending the original momentum, to its
reduced (or body) version. In other words, the map $\pi$ is Poisson with
respect to $\\{\cdot,\cdot\\}_{\rm can}$ and $\\{\cdot,\cdot\\}$, see Def.
3.1. This natural process, a special instance of Poisson reduction, explains
the occurrence of Lie-Poisson brackets in a large class of systems, see Table
1. While some of these systems involve extensions of the process described
above, the resulting noncanonical Poisson structures are each time justified
as arising from a canonical Poisson bracket on a larger space (the initial
phase space) by using the symmetries.

In the case of right-invariant systems, one has $\mu=pg^{-1}$ and a change of
sign arises on the right hand side of (78) and (79).

#### Flow of Lie-Poisson systems

Let us denote by $\operatorname{Ad}_{g}:\mathfrak{g}\rightarrow\mathfrak{g}$
the adjoint action of $G$ on its Lie algebra, defined as
$\operatorname{Ad}_{g}\xi:=\left.\frac{d}{d\varepsilon}\right|_{\varepsilon=0}gc_{\varepsilon}g^{-1}$
with $c_{\varepsilon}\in G$ a curve tangent to $\xi$ at $g=e$, and by
$\operatorname{Ad}^{*}_{g}:\mathfrak{g}^{*}\rightarrow\mathfrak{g}^{*}$ the
coadjoint action defined as
$\langle\operatorname{Ad}^{*}_{g}\mu,\xi\rangle=\langle\mu,\operatorname{Ad}_{g}\xi\rangle$.
For $g(t)\in G$ we have the formula

$\frac{d}{dt}\operatorname{Ad}^{*}_{g(t)}\mu_{0}=\operatorname{ad}^{*}_{g(t)^{-1}\dot{g}(t)}\operatorname{Ad}^{*}_{g(t)}\mu_{0}\,.$

From this it immediately follows that the flow $\phi_{t}(\mu_{0})$ of the Lie-
Poisson equations (78) takes the form

$\phi_{t}(\mu_{0})=\operatorname{Ad}^{*}_{g(t)}\mu_{0}\quad\text{with}\quad
g(t)^{-1}\dot{g}(t)=\frac{\partial h}{\partial\mu}(t)$

and hence preserves the coadjoint orbits
$\mathcal{O}=\\{\operatorname{Ad}^{*}_{g}\mu_{0}\mid g\in
G\\}\subset\mathfrak{g}^{*}$.

For the case of a linear Hamiltonian $h(\mu)=\langle a,\mu\rangle$, where
$\alpha\in\mathfrak{g}$ is a fixed element of the Lie algebra, the Lie-Poisson
equation (78) become

$\dot{\mu}=\operatorname{ad}^{*}_{\alpha}\mu$ (80)

and its flow is found as

$\phi_{t}(\mu_{0})=\operatorname{Ad}^{*}_{g(t)}\mu_{0}\quad\text{with}\quad
g(t)^{-1}\dot{g}(t)=\alpha\,.$

This is the coordinate free version of (23), namely $\mathbb{T}(t,\alpha)$ is
the coordinate version of
$\operatorname{Ad}^{*}_{\operatorname{exp}(t\alpha)}$, with
$\operatorname{exp}$ the Lie group exponential. Then, we can derive LPNets if
we can explicitly exponentiate the elements of the Lie algebra, so that it can
be efficiently used in the minimization procedure (25).

For $SO(3)$ and $SE(3)$ the coadjoint action read

$\operatorname{Ad}_{\Lambda^{-1}}^{*}\boldsymbol{\Pi}=\Lambda\boldsymbol{\Pi}\quad\text{and}\quad\operatorname{Ad}_{(\Lambda,\mathbf{b})^{-1}}^{*}(\boldsymbol{\Pi},\mathbf{p})=(\Lambda\boldsymbol{\Pi}+\mathbf{b}\times\Lambda\mathbf{p},\Lambda\mathbf{p})$

consistently with the formulas for $\mathbb{T}$ appearing in Sec. 5.1&5.4.